system

Abstract

We provide the system. [Solution] By managing a database that holds users' personal information, a means of saving users' profile information is provided. A means for generating appropriate responses using a natural language processing engine for natural language dialogue with users, A means of evaluating the user's health status and emotions by analyzing image information acquired using a camera, Based on the aforementioned evaluation, a means of notifying remote administrators and assisting in the health management of users is provided. A means of reminding users of important appointments and anniversaries based on their set schedule information, A system that includes this.

Description

Technical Field

[0001] The technology of this disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a society with a growing aging population, there is a problem that the reduction of the sense of loneliness and the health management of the elderly are not sufficient. The issue is that such isolation and health anxiety of the elderly lead to a decline in the quality of life. In contrast, there is a demand for providing support that allows the elderly to live with peace of mind and manage their health autonomously.

Means for Solving the Problems

[0005] This invention provides a system for managing a database that holds users' personal information and incorporates a natural language processing engine that enables natural language dialogue with users. Furthermore, it supports health management by evaluating the user's health status and emotions through image information analysis using a camera and notifying a remote administrator based on the evaluation results. It also provides a means to improve a sense of security in daily life by providing a reminder function for important appointments and anniversaries based on the user's schedule information. In addition to these, it aims to reduce social isolation among the elderly by providing a platform that generates feedback from activity data and promotes virtual interaction among multiple users.

[0006] "User" refers to an individual who uses this system.

[0007] "Personal information" refers to all information related to a user, including profile information and activity data.

[0008] A "database" refers to an information system used to store managed information in a categorized and searchable format.

[0009] A "natural language processing engine" refers to a program that understands human language and generates responses.

[0010] A "camera" refers to a device used to acquire and analyze image information.

[0011] "Image information analysis" refers to the process of analyzing acquired image data and evaluating health status and emotions from it.

[0012] "Health status" refers to the physical and mental health condition of the user.

[0013] "Emotions" refer to the psychological state that can be interpreted from the user's facial expressions and words and actions.

[0014] "Administrator" refers to an individual or organization that has the role of supporting the health management of users through the system.

[0015] "Notification" refers to a communication means for notifying users or administrators of information and warnings.

[0016] "Schedule information" refers to information related to the schedules and anniversaries set by users.

[0017] "Reminder function" refers to a function of the system that notifies users of the set schedules and anniversaries and evokes memories.

[0018] "Activity data" refers to data that records information related to the daily activities of users.

[0019] "Feedback" refers to information related to improvement and evaluation provided based on the analysis results of activity data.

[0020] "Virtual communication platform" refers to an online venue where multiple users can communicate via the Internet.

[0021] "Social isolation" refers to the feeling of loneliness and isolation felt due to a situation of having few opportunities to contact others.

Brief Explanation of Drawings

[0022] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0023] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0026] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0027] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0028] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0030] [First Embodiment]

[0031] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0032] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0035] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0038] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0042] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0043] This invention is an AI system for supporting the lives of the elderly, comprehensively providing data management based on personal information, natural language dialogue, image information analysis, health management notifications, schedule reminders, activity data analysis, and a virtual communication platform.

[0044] The entire system operates with a server and terminal working together. The server manages a database that stores user profiles and activity data, and uses a natural language processing engine to analyze user input and generate appropriate responses. For example, if a user says to the terminal, "I want to go for a walk outside today," the server will respond based on weather information, "It's sunny today, so you can comfortably take a walk during the daytime."

[0045] The device is equipped with a camera that periodically captures video of the user. The device analyzes the image information locally to evaluate the user's health and emotions. For example, if the device detects fatigue from the user's facial color, it sends this information to a server, which can then take appropriate action.

[0046] In the reminder function, the server calculates the timing for reminding users of important appointments and anniversaries based on their schedule. The server notifies the device of the reminder, and the device informs the user through voice or screen display. For example, it can notify users of things like, "Tomorrow is your grandchild's birthday."

[0047] Furthermore, the server periodically collects user activity data and generates feedback based on the analysis results. This feedback encourages improvements in daily life and provides positive messages such as, "Your step count has increased recently. Great job!"

[0048] Virtual social platforms allow users to interact with other users who share similar hobbies and interests via the internet. This helps reduce feelings of loneliness and allows users to feel a sense of social connection.

[0049] This invention thus improves the quality of life for the elderly and supports autonomous health management.

[0050] The following describes the processing flow.

[0051] Step 1:

[0052] The server stores personal information submitted by users in a database. This allows for the management of user profile information, which is then used as basic data for response generation and health management.

[0053] Step 2:

[0054] The terminal receives voice input from the user and converts it into text data via a speech recognition system. This converted text data is then sent to the server.

[0055] Step 3:

[0056] The server analyzes the received text data using a natural language processing engine and generates an appropriate response based on its content. This response is sent to the terminal and presented to the user.

[0057] Step 4:

[0058] The device periodically captures the user's face using its built-in camera. The acquired images are analyzed locally to evaluate the user's health and emotions.

[0059] Step 5:

[0060] The device sends evaluated health status and emotional information to the server. The server then sends health notifications to family members and care managers as needed.

[0061] Step 6:

[0062] The server notifies the device of upcoming important events and anniversaries based on a pre-configured schedule. The device then reminds the user visually and audibly.

[0063] Step 7:

[0064] The server collects and analyzes user activity data. The feedback generated from the analysis is provided to the user via their device, helping them to review their daily habits.

[0065] Step 8:

[0066] Users utilize a virtual social networking platform to gain the opportunity to interact online with other users who share similar hobbies and interests. The server provides the infrastructure to support this interaction.

[0067] (Example 1)

[0068] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0069] In the living environment of the elderly, challenges include insufficient health management, lack of social interaction, and a lack of feedback on daily activities. Elderly people living alone, in particular, have limited opportunities for daily health monitoring and appropriate advice. Furthermore, they often struggle to maintain social connections and suffer from feelings of loneliness. This invention aims to comprehensively solve these problems.

[0070] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0071] In this invention, the server includes an information aggregation and management unit that holds the user's personal information, means for storing profile information, means for generating appropriate responses using a natural language processing mechanism for engaging in natural language dialogue with the user, and means for evaluating the user's health status and emotions by analyzing visual information acquired using an imaging device. As a result, the user can live their daily life autonomously and safely while receiving real-time support.

[0072] The "Information Collection and Management Department" is an organization responsible for securely collecting, storing, and managing users' personal information and activity data.

[0073] A "natural language processing mechanism" is a technology that analyzes language data entered by users in the form of speech or text, understands its content, and generates an appropriate response.

[0074] An "imaging device" is a device used to acquire image data of a target object or person, and often includes a camera.

[0075] "Visual information" refers to data such as images and videos acquired by imaging devices, and this data is the subject of analysis.

[0076] A "health status assessment method" is a technology that analyzes visual information and other sensor information to determine the physical and psychological state of a user.

[0077] An "information sharing platform" is a digital platform that allows multiple users to interact with each other via the internet based on their hobbies and interests.

[0078] A "feedback generation method" is a technology that analyzes user activity data and provides users with improvement suggestions and encouraging messages based on the results.

[0079] "Generative machine learning models" refer to a group of algorithms that learn from data and have the function of making predictions or generating new data, and are used in natural language processing, among other applications.

[0080] This invention provides a specific embodiment of an AI system for supporting the daily lives of the elderly. This system operates with a server and terminals working in cooperation to comprehensively support the daily lives of the elderly.

[0081] The server includes an information aggregation and management unit that securely stores and manages users' personal information and activity data. This data includes users' profile information and daily activity details, and the database typically uses an open-source database management system. This allows for the personalization of support needs for each user.

[0082] The device has the functionality to interact directly with the user. When a user speaks to the device, the voice data is sent to a server equipped with a natural language processing mechanism. The server uses a generative AI model to understand the user's request and generate an appropriate response. For example, in response to a question like "What's the weather like today?", the server will answer "It's sunny today, perfect for a walk" based on the latest weather information. An example of a prompt is "Please suggest some simple exercises for seniors."

[0083] The terminal is also equipped with an imaging device that periodically captures images of the user. The terminal uses OpenCV and other image processing software to analyze these images and evaluate the user's health status (e.g., emotions from facial expressions, changes in physical condition from complexion). For example, if signs of fatigue are detected from the user's complexion, the server is notified, and a warning is sent to the maintenance manager.

[0084] Furthermore, the server has the functionality to connect multiple users through a virtual communication platform. This allows users to interact with other users who share common hobbies and interests via the internet, thereby strengthening their social connections. For example, when conducting an "online painting class" as part of a hobby, participants can share their artwork and exchange feedback with each other through the system.

[0085] In this way, the system improves the quality of life for users and supports autonomous and safe living.

[0086] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0087] Step 1:

[0088] The server uses an information aggregation and management unit to store user profile information and activity data in a database. It receives profile and daily activity data updated by users on their terminals as input, formats that data, and saves it to the database. The output is a dataset containing detailed information for each user. Through this process, the server builds a foundation for providing optimal services to users. Specifically, it performs database insertion and update operations in response to user change requests.

[0089] Step 2:

[0090] When a user speaks into the device, the device sends the voice input to the server. The server uses a natural language processing mechanism to convert the voice data into text and analyzes its content. The input is the user's voice data, and the output is an understanding of the intent and a corresponding response. Specifically, it uses a generative AI model to infer unclear words, retrieves information from external APIs as needed, and constructs an appropriate response.

[0091] Step 3:

[0092] An imaging device installed in the terminal periodically captures images of the user. The input is the captured visual information, which the terminal processes with an analysis tool. As output, it generates evaluation results of the user's health status and emotions and sends them to the server. Specifically, it uses an image analysis library to infer fatigue and emotions from changes in skin tone and facial expressions, and generates data useful for health management based on the results.

[0093] Step 4:

[0094] The server uses the collected data to remind users of scheduled events. It retrieves schedule information from a database as input and lists important events and anniversaries. As output, it generates reminder notifications and sends them to the device. Specifically, it uses a notification generation algorithm based on the scheduled date and time to send timely reminders to the user.

[0095] Step 5:

[0096] The server analyzes the user's daily activity data and generates feedback messages. The input is activity data sent from the device, which the server then uses for statistical analysis. The output is a message sent to the device containing improvement suggestions and encouragement. Specifically, it aggregates data such as steps taken and distance traveled, and creates evaluations and feedback based on certain criteria.

[0097] Step 6:

[0098] The server utilizes a virtual communication platform to facilitate interaction among multiple users. It uses information about users' interests and hobbies as input and suggests interaction events and groups to participate in as output. Specifically, it analyzes the interests registered by users and applies an algorithm to match them with other users who have similar interests.

[0099] (Application Example 1)

[0100] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0101] The loneliness and health management challenges faced by the elderly are serious. Many elderly people suffer from isolation and health problems due to a lack of adequate support in their daily lives. Furthermore, a lack of opportunities for interaction among multiple users and a resulting lack of social connection are also problematic. Technological means are needed to address these challenges and improve the quality of life for the elderly.

[0102] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0103] In this invention, the server includes means for storing user attribute information by managing a database that holds the user's personal information, means for generating appropriate responses using a natural language processing system for natural language dialogue with the user, and means for evaluating the user's health condition and emotions by analyzing image information acquired using imaging means. This makes it possible to understand the health condition of elderly people and provide them with emotional support. It also makes it possible to promote interaction with other users through care support devices and reduce feelings of loneliness.

[0104] A "database" is a collection of information used to comprehensively manage users' personal information and store their attribute information.

[0105] A "natural language processing system" is a language analysis device that engages in dialogue with users using natural language and generates appropriate responses.

[0106] An "imaging means" is a device that acquires image information using a camera or similar device and analyzes it.

[0107] "Health status" refers to information indicating the user's physical condition, and is evaluated by imaging techniques.

[0108] "Emotions" refer to information that indicates the user's mental state and are evaluated using imaging techniques and natural language processing systems.

[0109] "Notification" refers to the act of informing a remote administrator or the user themselves of information obtained based on an evaluation.

[0110] "Care support equipment" refers to automated devices used to assist elderly people in their daily lives.

[0111] "Virtual interaction" refers to the act of using information technology to facilitate online interaction between users based on their hobbies and interests.

[0112] "Attribute information" refers to data that shows basic information about a user and is stored in a database.

[0113] "Schedule information" refers to data that records appointments and anniversaries set by the user, and serves as the basis for notifications.

[0114] The system implementing this invention consists of a server and a terminal. The server manages a database that holds users' personal information and stores users' attribute information. By using a natural language processing system, the server analyzes user input and generates appropriate responses. This processing is performed using natural language processing libraries such as Python and NLTK.

[0115] The terminal, for example, uses a Raspberry Pi and a camera module to capture images of the user and analyze that information using OpenCV. The terminal then evaluates the user's health and emotions and sends the results to a server. Based on this information, the server assists with the user's health management and, if necessary, sends notifications to remote administrators.

[0116] The server also manages important appointments based on the schedule information set by the user. It provides schedule notifications to the user via voice or screen display, offering support to help them complete their daily tasks. Furthermore, it regularly collects user activity information, generates feedback from that data, and provides recommendations to help improve daily life.

[0117] Through care support devices, the server facilitates virtual interaction among users, providing online connections based on shared hobbies and interests. This virtual interaction helps users reduce feelings of loneliness and lead richer social lives.

[0118] As a concrete example, the monitoring care robot encourages the user's activity by asking, "Would you like to go for a walk today?" On sunny days, it might say, "The weather is nice today, so let's go for a walk together."

[0119] Example input prompts for a generative AI model:

[0120] "Please explain how AI support systems can be implemented in elderly care. I'd particularly like to focus on health monitoring and reducing feelings of isolation. I'd like detailed information on the necessary hardware and software, as well as their processing capabilities."

[0121] This system enables elderly individuals to manage their own health and receive emotional support. This invention aims to improve the quality of life for the elderly and provide a safe and secure living environment.

[0122] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0123] Step 1:

[0124] The device uses a camera module to acquire image information of the user. The input is real-time image data obtained from the camera, and this data is analyzed using the OpenCV library. Face recognition and skin tone analysis are performed to generate basic data for evaluating health status and emotions. The output is the results of the analyzed health status and emotional evaluation.

[0125] Step 2:

[0126] The terminal sends the evaluation results to the server. The input is the evaluation results of health status and emotions obtained in step 1. The server receives this and stores it in a database, enabling continuous monitoring of health status. The output is the user's health information stored in the database.

[0127] Step 3:

[0128] The user speaks to the server in natural language through a voice input device. The input consists of specific requests and questions from the user. The server uses a natural language processing system to analyze this voice input and generate an appropriate response. The output is the generated response message.

[0129] Step 4:

[0130] The server manages the user's scheduled schedule and creates important appointments and reminders. The input is the user's schedule information, and an algorithm calculates the timing of notifications. The output is a list of events to be notified.

[0131] Step 5:

[0132] The server sends the generated reminder and response messages to the terminal, and the terminal notifies the user via voice or screen display. The input is the data generated in steps 3 and 4. The output is the voice or visual notification to the user.

[0133] Step 6:

[0134] The care support device interacts with a server and provides an interface to facilitate virtual interaction among users. Inputs include data about the user's interests and preferences, as well as interaction requests from other users. Outputs are communication channels between those wishing to participate.

[0135] In this way, the system supports the daily lives of the elderly, reduces feelings of isolation, and manages their health. The use of generative AI models is expected to facilitate smooth natural language dialogue.

[0136] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0137] This invention is an AI system aimed at managing users' health and alleviating social isolation. In particular, by incorporating an emotion engine, it recognizes users' emotions and personalizes the content of conversations and support accordingly.

[0138] The system is configured so that the server and terminals communicate via a network. The server maintains a database for efficiently managing users' personal information and activity data, and is equipped with a natural language processing engine to enable natural conversations with users. Furthermore, by implementing an emotion engine, it is possible to evaluate users' emotions in real time.

[0139] The device is equipped with a camera and microphone to capture the user's daily video and audio. This information is analyzed by an emotion engine, either locally or on a server, to evaluate the user's emotional state. For example, if the device detects changes in the user's facial expressions or tone of voice, and the emotion engine identifies a "sad" state, the server can instruct the device to play relaxing music as an appropriate resource.

[0140] The reminder function uses the server to notify the user of their next appointment based on their schedule information. Furthermore, if the emotion engine determines that the user is in an unstable emotional state, it can also provide specific advice on preparing for important appointments.

[0141] Furthermore, the server analyzes activity data and generates feedback that helps with health management and daily life improvement. This feedback uses positive language to reinforce positive emotions and supports the improvement of daily habits.

[0142] Furthermore, the virtual interaction platform allows multiple users to connect based on shared hobbies and interests. If the emotion engine detects anxiety or loneliness, the server suggests joining a group with similar interests, providing an opportunity to build social connections.

[0143] Thus, the present invention is a system that, by using an emotion engine, enables the provision of more personalized support to users and enhances their mental and physical health.

[0144] The following describes the processing flow.

[0145] Step 1:

[0146] The device captures the user's face and voice using its camera and microphone. This allows for the acquisition of real-time image and audio data.

[0147] Step 2:

[0148] The device sends the acquired image and audio data to an emotion engine to analyze the user's emotions. For example, it analyzes facial muscle movements and voice tone to classify the emotional state into categories such as "joy," "anger," and "sadness."

[0149] Step 3:

[0150] The device receives the results analyzed by the emotion engine and sends them to the server. The server then uses this emotion data to generate the most appropriate response or suggestion.

[0151] Step 4:

[0152] The server uses a natural language processing engine to create responses appropriate to the user's emotional state. For example, if the user is perceived as "sad," it will generate a caring message such as, "Is something wrong today? Is there anything I can do to help?"

[0153] Step 5:

[0154] The server sends the generated response to the terminal, which then notifies the user. This is done through both audio and text, providing the user with reassurance and support.

[0155] Step 6:

[0156] Based on the user's schedule information, the server reminds the user of important appointments on their device. If the emotion engine determines that the user is experiencing stress, the server will notify the user with additional advice, such as "Make sure you allow yourself plenty of time to prepare."

[0157] Step 7:

[0158] The server analyzes user activity data and generates feedback to help with health management. If a user engages in positive activities on a daily basis, feedback praising their efforts is provided from the device.

[0159] Step 8:

[0160] If emotional data indicates feelings of loneliness, the server will encourage the user to participate in a virtual social platform. The user will then have the opportunity to build social connections by connecting with other users who share similar hobbies and interests.

[0161] Through this series of processes, the system can provide users with personalized support that is empathetic to their emotions.

[0162] (Example 2)

[0163] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0164] In modern society, managing people's health and the loss of social connections are serious challenges. The elderly and those with limited social contact due to remote work, in particular, tend to experience emotional isolation and difficulty managing their health. To improve this situation, there is a need to develop systems that provide individualized support that takes emotions into account, thereby improving the quality of daily life for each user.

[0165] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0166] In this invention, the server includes means for managing an information infrastructure that holds the user's personal information, means for generating appropriate responses using a language processing device for natural language dialogue with the user, and means for evaluating the user's health condition and emotions by analyzing video information acquired using an image input device. This enables the provision of personalized support tailored to the user's emotional state, thereby alleviating social isolation and improving health management.

[0167] "Information infrastructure" refers to the data management devices and information processing technologies necessary to securely hold and manage users' personal information and other related data.

[0168] A "language processing device" refers to a part or function of a computer system that analyzes natural language input and generates appropriate responses.

[0169] An "image input device" refers to hardware used to acquire video information of a user using a camera or other video acquisition equipment.

[0170] "Health status" refers to information related to the user's physical and mental health, including data such as heart rate, facial expressions, and voice tone.

[0171] "Evaluating emotions" refers to the process of analyzing and identifying a user's current emotions and psychological state based on facial expressions, voice, actions, etc.

[0172] "Personalized support" refers to an approach that provides optimal responses and services tailored to each user's emotions and health condition.

[0173] "Resolving social isolation" refers to alleviating feelings of loneliness and alienation that users experience in their daily lives, and enabling them to have richer social connections.

[0174] "Improving health management" refers to continuously monitoring the user's health status and providing appropriate feedback and advice as needed, enabling the user to maintain or improve their own health.

[0175] This invention is a comprehensive system for managing users' health and alleviating social isolation. The system consists of a server and terminals, each performing a specific function to provide personalized support to users.

[0176] The server operates as an information management platform, securely managing users' personal information and data related to their daily activities. The database system stores comprehensive information, including users' health history and schedules. Furthermore, by incorporating a natural language processing engine, the server can generate appropriate responses in real time based on text data and voice input received from users. The software used includes a "language processing unit" for natural language processing.

[0177] The device is equipped with a camera and microphone to capture video and audio of the user's daily life. This data is processed on a server or locally, and emotion analysis is performed. Emotion analysis involves analyzing changes in facial expressions and voice obtained from the "image input device" to identify the emotions the user is expressing. For example, if the device determines that the user is "sad," it will play relaxing music according to the server's instructions.

[0178] As a concrete example, a prompt message could look like this: "The next meeting is at 3 PM. If you're feeling nervous, take some time to take a deep breath." In addition, using a virtual social platform facilitates interaction between users with shared interests. For example, the server might suggest to users how to expand their social connections, such as, "Would you like to join an online cooking class?"

[0179] In this way, the server and terminal work together to provide a concrete system for improving the user's health and social connections. The system of the present invention provides users with powerful support to improve their quality of daily life and reduce feelings of isolation.

[0180] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0181] Step 1:

[0182] The device captures the user's daily activities through its camera and microphone. It acquires video and audio data of the user as input. This data is stored locally for a certain period and then compressed for transmission to the server. Specifically, the device uses the camera's activation and recording functions to capture video.

[0183] Step 2:

[0184] The server receives video and audio data sent from the terminal. It takes compressed video and audio files as input, decodes them, and makes them ready for analysis. The server decompresses the data and runs a program to "evaluate emotions" for sentiment analysis. Specifically, at this stage, the server extracts facial expressions from the video and analyzes the tone of voice from the audio data.

[0185] Step 3:

[0186] The server evaluates the user's emotional state from the analyzed data. It uses analyzed facial expression data and voice tone characteristics as input. Based on this information, a generative AI model identifies emotions such as "happy" or "sad." In this step, the server organizes and records the identified emotions in preparation for subsequent processing.

[0187] Step 4:

[0188] The server determines personalized responses and support based on the user's emotional state and sends instructions to the device. It uses the emotional information evaluated in the previous step as input. Based on the data processing, the server sends suggestions to the device, such as "play relaxing music" or "recommended online activities." Specific actions include displaying music provided by the device and links to suggested activities.

[0189] Step 5:

[0190] The device provides feedback to the user based on instructions received from the server. It receives commands from the server as input. Based on this data, the device plays music through its speaker and displays relevant information on its screen. Specifically, a music application is launched and a specified playlist is played, and a video device notifies the user of recommended activities.

[0191] (Application Example 2)

[0192] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0193] In recent years, the importance of individual health management has increased, and prompt and accurate responses to changes in health conditions are particularly required for the elderly and individuals with physical limitations. Furthermore, the problem of social isolation is also prominent, and given the current situation where many people experience feelings of isolation and anxiety, there is a need to provide means of fostering social connections that transcend physical distance. Because conventional technologies struggle to adequately address these individual and dynamic needs, a new system is required that improves health management while alleviating social isolation by providing personalized care tailored to emotional states.

[0194] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0195] In this invention, the server includes means for managing the user's personal information, means for providing corresponding entertainment information based on emotion analysis, and means for suggesting social connections through virtual interaction. This makes it possible to evaluate the user's emotional state in real time and provide care and opportunities for social interaction tailored to individual needs.

[0196] "Personal information" refers to information that can uniquely identify a user, such as their name, contact information, age, and health status.

[0197] A "natural language processing engine" is a technology that enables computers to understand and generate natural language used by humans, thereby supporting dialogue.

[0198] "Emotion analysis" is a technology that determines a user's emotional state by processing data obtained through cameras and microphones.

[0199] "Health management" refers to activities that monitor the physical and mental health status of users and provide appropriate care as needed.

[0200] "Virtual interaction" refers to the exchange of information and dialogue among multiple users who are physically separated, based on shared hobbies and interests, through an online platform.

[0201] "Entertainment information" refers to content such as music and videos that are provided according to the user's emotional state, with the aim of promoting relaxation and a change of pace.

[0202] "Social connection" refers to the sense of intimacy and camaraderie that users feel through interaction with other people.

[0203] This invention realizes a system that provides personalized support by having a server and terminals cooperate via a network to comprehensively manage the user's health status and emotions.

[0204] The server maintains a database to manage users' personal information and utilizes cloud services such as Firebase. It leverages Google Cloud Natural Language API as its natural language processing engine to support user interaction. Furthermore, it uses machine learning libraries such as TENSORFLOW® for sentiment analysis, analyzing video and audio data obtained from cameras and microphones in real time.

[0205] The device collects data on the user's daily life using devices such as smartphones and tablets. Equipped with a camera and microphone, it partially processes the acquired data locally and sends it to a server as needed, allowing for a more detailed assessment of emotional states.

[0206] As a concrete example, if the device detects through emotion analysis that an elderly person is feeling lonely during the day, the server will instruct the device to play relaxing music based on that data. Furthermore, as a suggestion for virtual interaction, it can recommend online events or hobby groups that the user might be interested in. Within this process, a possible prompt using a generative AI model might be: "Analyze the user's emotions from their face; if they appear sad, select and play soothing music. If they feel isolated, suggest a community event and encourage participation."

[0207] This system will enable support that makes users' daily lives safer and more fulfilling.

[0208] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0209] Step 1:

[0210] The device uses a camera and microphone to acquire video and audio data of the user's daily life. This inputs data indicating the user's emotional state, such as facial expressions and tone of voice.

[0211] Step 2:

[0212] The device inputs the acquired video and audio data into an emotion analysis model using TensorFlow, which analyzes the user's emotional state in real time. The emotion analysis model processes the data to identify the user's emotional state, such as "sad," "happy," or "anxious," and outputs this as an emotion label.

[0213] Step 3:

[0214] The server receives the emotion label sent from the terminal and refers to the personal information database. Based on this, if, for example, the emotion "sad" is detected, it selects entertainment information suitable for the user (e.g., relaxing music) and instructs the server to send this information to the terminal.

[0215] Step 4:

[0216] The terminal receives instructions from the server and provides the user with the specified entertainment information. During this process, it uses a music playback application to create an environment where the user can comfortably listen to the content.

[0217] Step 5:

[0218] When a user receives an emotional label indicating feelings of isolation, the server identifies virtual interaction opportunities based on the user's interests and sends information about relevant online events and community groups to the user's device. This allows the user to participate in virtual interactions that match their interests.

[0219] Step 6:

[0220] Finally, the device notifies the user of this information and uses an AI model to generate a message encouraging participation, displaying a prompt such as, "There's a new community event. Would you like to join?" This step helps motivate the user to actively build social connections.

[0221] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0222] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0223] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0224] [Second Embodiment]

[0225] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0226] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0227] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0228] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0229] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0230] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0231] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0232] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0233] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0234] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0235] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0236] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0237] This invention is an AI system for supporting the lives of the elderly, comprehensively providing data management based on personal information, natural language dialogue, image information analysis, health management notifications, schedule reminders, activity data analysis, and a virtual communication platform.

[0238] The entire system operates with a server and terminal working together. The server manages a database that stores user profiles and activity data, and uses a natural language processing engine to analyze user input and generate appropriate responses. For example, if a user says to the terminal, "I want to go for a walk outside today," the server will respond based on weather information, "It's sunny today, so you can comfortably take a walk during the daytime."

[0239] The device is equipped with a camera that periodically captures video of the user. The device analyzes the image information locally to evaluate the user's health and emotions. For example, if the device detects fatigue from the user's facial color, it sends this information to a server, which can then take appropriate action.

[0240] In the reminder function, the server calculates the timing for reminding users of important appointments and anniversaries based on their schedule. The server notifies the device of the reminder, and the device informs the user through voice or screen display. For example, it can notify users of things like, "Tomorrow is your grandchild's birthday."

[0241] Furthermore, the server periodically collects user activity data and generates feedback based on the analysis results. This feedback encourages improvements in daily life and provides positive messages such as, "Your step count has increased recently. Great job!"

[0242] Virtual social platforms allow users to interact with other users who share similar hobbies and interests via the internet. This helps reduce feelings of loneliness and allows users to feel a sense of social connection.

[0243] This invention thus improves the quality of life for the elderly and supports autonomous health management.

[0244] The following describes the processing flow.

[0245] Step 1:

[0246] The server stores personal information submitted by users in a database. This allows for the management of user profile information, which is then used as basic data for response generation and health management.

[0247] Step 2:

[0248] The terminal receives voice input from the user and converts it into text data via a speech recognition system. This converted text data is then sent to the server.

[0249] Step 3:

[0250] The server analyzes the received text data using a natural language processing engine and generates an appropriate response based on its content. This response is sent to the terminal and presented to the user.

[0251] Step 4:

[0252] The device periodically captures the user's face using its built-in camera. The acquired images are analyzed locally to evaluate the user's health and emotions.

[0253] Step 5:

[0254] The device sends evaluated health status and emotional information to the server. The server then sends health notifications to family members and care managers as needed.

[0255] Step 6:

[0256] The server notifies the device of upcoming important events and anniversaries based on a pre-configured schedule. The device then reminds the user visually and audibly.

[0257] Step 7:

[0258] The server collects and analyzes user activity data. The feedback generated from the analysis is provided to the user via their device, helping them to review their daily habits.

[0259] Step 8:

[0260] Users utilize a virtual social networking platform to gain the opportunity to interact online with other users who share similar hobbies and interests. The server provides the infrastructure to support this interaction.

[0261] (Example 1)

[0262] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0263] In the living environment of the elderly, challenges include insufficient health management, lack of social interaction, and a lack of feedback on daily activities. Elderly people living alone, in particular, have limited opportunities for daily health monitoring and appropriate advice. Furthermore, they often struggle to maintain social connections and suffer from feelings of loneliness. This invention aims to comprehensively solve these problems.

[0264] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0265] In this invention, the server includes an information aggregation and management unit that holds the user's personal information, means for storing profile information, means for generating appropriate responses using a natural language processing mechanism for engaging in natural language dialogue with the user, and means for evaluating the user's health status and emotions by analyzing visual information acquired using an imaging device. As a result, the user can live their daily life autonomously and safely while receiving real-time support.

[0266] The "Information Collection and Management Department" is an organization responsible for securely collecting, storing, and managing users' personal information and activity data.

[0267] A "natural language processing mechanism" is a technology that analyzes language data entered by users in the form of speech or text, understands its content, and generates an appropriate response.

[0268] An "imaging device" is a device used to acquire image data of a target object or person, and often includes a camera.

[0269] "Visual information" refers to data such as images and videos acquired by imaging devices, and this data is the subject of analysis.

[0270] A "health status assessment method" is a technology that analyzes visual information and other sensor information to determine the physical and psychological state of a user.

[0271] An "information sharing platform" is a digital platform that allows multiple users to interact with each other via the internet based on their hobbies and interests.

[0272] A "feedback generation method" is a technology that analyzes user activity data and provides users with improvement suggestions and encouraging messages based on the results.

[0273] "Generative machine learning models" refer to a group of algorithms that learn from data and have the function of making predictions or generating new data, and are used in natural language processing, among other applications.

[0274] This invention provides a specific embodiment of an AI system for supporting the daily lives of the elderly. This system operates with a server and terminals working in cooperation to comprehensively support the daily lives of the elderly.

[0275] The server includes an information aggregation and management unit that securely stores and manages users' personal information and activity data. This data includes users' profile information and daily activity details, and the database typically uses an open-source database management system. This allows for the personalization of support needs for each user.

[0276] The device has the functionality to interact directly with the user. When a user speaks to the device, the voice data is sent to a server equipped with a natural language processing mechanism. The server uses a generative AI model to understand the user's request and generate an appropriate response. For example, in response to a question like "What's the weather like today?", the server will answer "It's sunny today, perfect for a walk" based on the latest weather information. An example of a prompt is "Please suggest some simple exercises for seniors."

[0277] The terminal is also equipped with an imaging device that periodically captures images of the user. The terminal uses OpenCV and other image processing software to analyze these images and evaluate the user's health status (e.g., emotions from facial expressions, changes in physical condition from complexion). For example, if signs of fatigue are detected from the user's complexion, the server is notified, and a warning is sent to the maintenance manager.

[0278] Furthermore, the server has the functionality to connect multiple users through a virtual communication platform. This allows users to interact with other users who share common hobbies and interests via the internet, thereby strengthening their social connections. For example, when conducting an "online painting class" as part of a hobby, participants can share their artwork and exchange feedback with each other through the system.

[0279] In this way, the system improves the quality of life for users and supports autonomous and safe living.

[0280] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0281] Step 1:

[0282] The server uses the information integration management department to store the user's profile information and activity data in the database. As input, it receives the profile and daily activity data updated by the user on the terminal, formats the data, and saves it in the database. As output, a dataset storing the detailed information for each user is obtained. Through this process, the server builds a foundation for providing optimal services to users. As specific operations, it performs insertion and update operations on the database according to the user's change requests.

[0283] Step 2:

[0284] When the user speaks to the terminal, the terminal sends the voice input to the server. The server uses the natural language processing mechanism to convert the voice data into text and analyzes its content. The input is the user's voice data, and the output is the understanding of the intention and the corresponding response generated. As specific operations, it uses the generation AI model to infer unclear words, obtains information from external APIs if necessary, and constructs an appropriate response.

[0285] Step 3:

[0286] The imaging device installed on the terminal periodically captures the user's image. The input is the captured visual information, and the terminal processes it with an analysis tool. As output, it generates the evaluation results of the user's health status and emotions and sends them to the server. As specific operations, it uses an image analysis library to infer fatigue and emotions from skin color changes and expressions, and generates data useful for health management based on the results.

[0287] Step 4:

[0288] The server uses the collected data to remind users of scheduled events. It retrieves schedule information from a database as input and lists important events and anniversaries. As output, it generates reminder notifications and sends them to the device. Specifically, it uses a notification generation algorithm based on the scheduled date and time to send timely reminders to the user.

[0289] Step 5:

[0290] The server analyzes the user's daily activity data and generates feedback messages. The input is activity data sent from the device, which the server then uses for statistical analysis. The output is a message sent to the device containing improvement suggestions and encouragement. Specifically, it aggregates data such as steps taken and distance traveled, and creates evaluations and feedback based on certain criteria.

[0291] Step 6:

[0292] The server utilizes a virtual communication platform to facilitate interaction among multiple users. It uses information about users' interests and hobbies as input and suggests interaction events and groups to participate in as output. Specifically, it analyzes the interests registered by users and applies an algorithm to match them with other users who have similar interests.

[0293] (Application Example 1)

[0294] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0295] The loneliness and health management challenges faced by the elderly are serious. Many elderly people suffer from isolation and health problems due to a lack of adequate support in their daily lives. Furthermore, a lack of opportunities for interaction among multiple users and a resulting lack of social connection are also problematic. Technological means are needed to address these challenges and improve the quality of life for the elderly.

[0296] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0297] In this invention, the server includes means for storing user attribute information by managing a database that holds the user's personal information, means for generating appropriate responses using a natural language processing system for natural language dialogue with the user, and means for evaluating the user's health condition and emotions by analyzing image information acquired using imaging means. This makes it possible to understand the health condition of elderly people and provide them with emotional support. It also makes it possible to promote interaction with other users through care support devices and reduce feelings of loneliness.

[0298] A "database" is a collection of information used to comprehensively manage users' personal information and store their attribute information.

[0299] A "natural language processing system" is a language analysis device that engages in dialogue with users using natural language and generates appropriate responses.

[0300] An "imaging means" is a device that acquires image information using a camera or similar device and analyzes it.

[0301] "Health status" refers to information indicating the user's physical condition, and is evaluated by imaging techniques.

[0302] "Emotions" refer to information that indicates the user's mental state and are evaluated using imaging techniques and natural language processing systems.

[0303] "Notification" refers to the act of informing a remote administrator or the user themselves of information obtained based on an evaluation.

[0304] "Care support equipment" refers to automated devices used to assist elderly people in their daily lives.

[0305] "Virtual communication" refers to the act of promoting online communication among users based on their hobbies and interests using information technology.

[0306] "Attribute information" refers to data indicating basic information about a user, which is stored in a database.

[0307] "Schedule information" refers to data recording the schedules and anniversaries set by a user, which serves as a criterion for notifications.

[0308] The system for implementing this invention is composed of a server and a terminal. The server manages a database that holds the personal information of users and is responsible for storing the attribute information of users. By using a natural language processing system, the server analyzes the input from users and generates appropriate responses. This processing is carried out by leveraging natural language processing libraries such as Python and NLTK.

[0309] As an example, a Raspberry Pi and a camera module are used for the terminal to capture the image information of a user and analyze the information using OpenCV. Thereby, the terminal evaluates the health status and emotions of the user and transmits the results to the server. Based on this information, the server supports the health management of the user and sends notifications to administrators in remote locations if necessary.

[0310] In addition, the server manages important schedules based on the schedule information set by the user. It notifies the user of the schedule through voice or screen display and provides support for fulfilling daily schedules. Furthermore, it regularly collects the activity information of the user, generates feedback from the data, and gives recommendations useful for improving daily life.

[0311] Through care support devices, the server promotes virtual communication among users and provides an online connection based on common hobbies and interests. Through this virtual communication, users can reduce their sense of loneliness and lead a richer social life.

[0312] As a concrete example, the monitoring care robot encourages the user's activity by asking, "Would you like to go for a walk today?" On sunny days, it might say, "The weather is nice today, so let's go for a walk together."

[0313] Example input prompts for a generative AI model:

[0314] "Please explain how AI support systems can be implemented in elderly care. I'd particularly like to focus on health monitoring and reducing feelings of isolation. I'd like detailed information on the necessary hardware and software, as well as their processing capabilities."

[0315] This system enables elderly individuals to manage their own health and receive emotional support. This invention aims to improve the quality of life for the elderly and provide a safe and secure living environment.

[0316] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0317] Step 1:

[0318] The device uses a camera module to acquire image information of the user. The input is real-time image data obtained from the camera, and this data is analyzed using the OpenCV library. Face recognition and skin tone analysis are performed to generate basic data for evaluating health status and emotions. The output is the results of the analyzed health status and emotional evaluation.

[0319] Step 2:

[0320] The terminal sends the evaluation results to the server. The input is the evaluation results of health status and emotions obtained in step 1. The server receives this and stores it in a database, enabling continuous monitoring of health status. The output is the user's health information stored in the database.

[0321] Step 3:

[0322] The user speaks to the server in natural language through a voice input device. The input consists of specific requests and questions from the user. The server uses a natural language processing system to analyze this voice input and generate an appropriate response. The output is the generated response message.

[0323] Step 4:

[0324] The server manages the user's scheduled schedule and creates important appointments and reminders. The input is the user's schedule information, and an algorithm calculates the timing of notifications. The output is a list of events to be notified.

[0325] Step 5:

[0326] The server sends the generated reminder and response messages to the terminal, and the terminal notifies the user via voice or screen display. The input is the data generated in steps 3 and 4. The output is the voice or visual notification to the user.

[0327] Step 6:

[0328] The care support device interacts with a server and provides an interface to facilitate virtual interaction among users. Inputs include data about the user's interests and preferences, as well as interaction requests from other users. Outputs are communication channels between those wishing to participate.

[0329] In this way, the system supports the daily lives of the elderly, reduces feelings of isolation, and manages their health. The use of generative AI models is expected to facilitate smooth natural language dialogue.

[0330] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0331] This invention is an AI system aimed at managing users' health and alleviating social isolation. In particular, by incorporating an emotion engine, it recognizes users' emotions and personalizes the content of conversations and support accordingly.

[0332] The system is configured so that the server and terminals communicate via a network. The server maintains a database for efficiently managing users' personal information and activity data, and is equipped with a natural language processing engine to enable natural conversations with users. Furthermore, by implementing an emotion engine, it is possible to evaluate users' emotions in real time.

[0333] The device is equipped with a camera and microphone to capture the user's daily video and audio. This information is analyzed by an emotion engine, either locally or on a server, to evaluate the user's emotional state. For example, if the device detects changes in the user's facial expressions or tone of voice, and the emotion engine identifies a "sad" state, the server can instruct the device to play relaxing music as an appropriate resource.

[0334] The reminder function uses the server to notify the user of their next appointment based on their schedule information. Furthermore, if the emotion engine determines that the user is in an unstable emotional state, it can also provide specific advice on preparing for important appointments.

[0335] Furthermore, the server analyzes activity data and generates feedback that helps with health management and daily life improvement. This feedback uses positive language to reinforce positive emotions and supports the improvement of daily habits.

[0336] Furthermore, the virtual interaction platform allows multiple users to connect based on shared hobbies and interests. If the emotion engine detects anxiety or loneliness, the server suggests joining a group with similar interests, providing an opportunity to build social connections.

[0337] Thus, the present invention is a system that, by using an emotion engine, enables the provision of more personalized support to users and enhances their mental and physical health.

[0338] The following describes the processing flow.

[0339] Step 1:

[0340] The device captures the user's face and voice using its camera and microphone. This allows for the acquisition of real-time image and audio data.

[0341] Step 2:

[0342] The device sends the acquired image and audio data to an emotion engine to analyze the user's emotions. For example, it analyzes facial muscle movements and voice tone to classify the emotional state into categories such as "joy," "anger," and "sadness."

[0343] Step 3:

[0344] The device receives the results analyzed by the emotion engine and sends them to the server. The server then uses this emotion data to generate the most appropriate response or suggestion.

[0345] Step 4:

[0346] The server uses a natural language processing engine to create responses appropriate to the user's emotional state. For example, if the user is perceived as "sad," it will generate a caring message such as, "Is something wrong today? Is there anything I can do to help?"

[0347] Step 5:

[0348] The server sends the generated response to the terminal, which then notifies the user. This is done through both audio and text, providing the user with reassurance and support.

[0349] Step 6:

[0350] Based on the user's schedule information, the server reminds the user of important appointments on their device. If the emotion engine determines that the user is experiencing stress, the server will notify the user with additional advice, such as "Make sure you allow yourself plenty of time to prepare."

[0351] Step 7:

[0352] The server analyzes user activity data and generates feedback to help with health management. If a user engages in positive activities on a daily basis, feedback praising their efforts is provided from the device.

[0353] Step 8:

[0354] If emotional data indicates feelings of loneliness, the server will encourage the user to participate in a virtual social platform. The user will then have the opportunity to build social connections by connecting with other users who share similar hobbies and interests.

[0355] Through this series of processes, the system can provide users with personalized support that is empathetic to their emotions.

[0356] (Example 2)

[0357] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0358] In modern society, managing people's health and the loss of social connections are serious challenges. The elderly and those with limited social contact due to remote work, in particular, tend to experience emotional isolation and difficulty managing their health. To improve this situation, there is a need to develop systems that provide individualized support that takes emotions into account, thereby improving the quality of daily life for each user.

[0359] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0360] In this invention, the server includes means for managing an information infrastructure that holds the user's personal information, means for generating appropriate responses using a language processing device for natural language dialogue with the user, and means for evaluating the user's health condition and emotions by analyzing video information acquired using an image input device. This enables the provision of personalized support tailored to the user's emotional state, thereby alleviating social isolation and improving health management.

[0361] "Information infrastructure" refers to the data management devices and information processing technologies necessary to securely hold and manage users' personal information and other related data.

[0362] A "language processing device" refers to a part or function of a computer system that analyzes natural language input and generates appropriate responses.

[0363] An "image input device" refers to hardware used to acquire video information of a user using a camera or other video acquisition equipment.

[0364] "Health status" refers to information related to the user's physical and mental health, including data such as heart rate, facial expressions, and voice tone.

[0365] "Evaluating emotions" refers to the process of analyzing and identifying a user's current emotions and psychological state based on facial expressions, voice, actions, etc.

[0366] "Personalized support" refers to an approach that provides optimal responses and services tailored to each user's emotions and health condition.

[0367] "Resolving social isolation" refers to alleviating feelings of loneliness and alienation that users experience in their daily lives, and enabling them to have richer social connections.

[0368] "Improving health management" refers to continuously monitoring the user's health status and providing appropriate feedback and advice as needed, enabling the user to maintain or improve their own health.

[0369] This invention is a comprehensive system for managing users' health and alleviating social isolation. The system consists of a server and terminals, each performing a specific function to provide personalized support to users.

[0370] The server operates as an information management platform, securely managing users' personal information and data related to their daily activities. The database system stores comprehensive information, including users' health history and schedules. Furthermore, by incorporating a natural language processing engine, the server can generate appropriate responses in real time based on text data and voice input received from users. The software used includes a "language processing unit" for natural language processing.

[0371] The device is equipped with a camera and microphone to capture video and audio of the user's daily life. This data is processed on a server or locally, and emotion analysis is performed. Emotion analysis involves analyzing changes in facial expressions and voice obtained from the "image input device" to identify the emotions the user is expressing. For example, if the device determines that the user is "sad," it will play relaxing music according to the server's instructions.

[0372] As a concrete example, a prompt message could look like this: "The next meeting is at 3 PM. If you're feeling nervous, take some time to take a deep breath." In addition, using a virtual social platform facilitates interaction between users with shared interests. For example, the server might suggest to users how to expand their social connections, such as, "Would you like to join an online cooking class?"

[0373] In this way, the server and terminal work together to provide a concrete system for improving the user's health and social connections. The system of the present invention provides users with powerful support to improve their quality of daily life and reduce feelings of isolation.

[0374] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0375] Step 1:

[0376] The device captures the user's daily activities through its camera and microphone. It acquires video and audio data of the user as input. This data is stored locally for a certain period and then compressed for transmission to the server. Specifically, the device uses the camera's activation and recording functions to capture video.

[0377] Step 2:

[0378] The server receives video and audio data sent from the terminal. It takes compressed video and audio files as input, decodes them, and makes them ready for analysis. The server decompresses the data and runs a program to "evaluate emotions" for sentiment analysis. Specifically, at this stage, the server extracts facial expressions from the video and analyzes the tone of voice from the audio data.

[0379] Step 3:

[0380] The server evaluates the user's emotional state from the analyzed data. It uses analyzed facial expression data and voice tone characteristics as input. Based on this information, a generative AI model identifies emotions such as "happy" or "sad." In this step, the server organizes and records the identified emotions in preparation for subsequent processing.

[0381] Step 4:

[0382] The server determines personalized responses and support based on the user's emotional state and sends instructions to the device. It uses the emotional information evaluated in the previous step as input. Based on the data processing, the server sends suggestions to the device, such as "play relaxing music" or "recommended online activities." Specific actions include displaying music provided by the device and links to suggested activities.

[0383] Step 5:

[0384] The device provides feedback to the user based on instructions received from the server. It receives commands from the server as input. Based on this data, the device plays music through its speaker and displays relevant information on its screen. Specifically, a music application is launched and a specified playlist is played, and a video device notifies the user of recommended activities.

[0385] (Application Example 2)

[0386] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0387] In recent years, the importance of individual health management has increased, and prompt and accurate responses to changes in health conditions are particularly required for the elderly and individuals with physical limitations. Furthermore, the problem of social isolation is also prominent, and given the current situation where many people experience feelings of isolation and anxiety, there is a need to provide means of fostering social connections that transcend physical distance. Because conventional technologies struggle to adequately address these individual and dynamic needs, a new system is required that improves health management while alleviating social isolation by providing personalized care tailored to emotional states.

[0388] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0389] In this invention, the server includes means for managing the user's personal information, means for providing corresponding entertainment information based on emotion analysis, and means for suggesting social connections through virtual interaction. This makes it possible to evaluate the user's emotional state in real time and provide care and opportunities for social interaction tailored to individual needs.

[0390] "Personal information" refers to information that can uniquely identify a user, such as their name, contact information, age, and health status.

[0391] A "natural language processing engine" is a technology that enables computers to understand and generate natural language used by humans, thereby supporting dialogue.

[0392] "Emotion analysis" is a technology that determines a user's emotional state by processing data obtained through cameras and microphones.

[0393] "Health management" refers to activities that monitor the physical and mental health status of users and provide appropriate care as needed.

[0394] "Virtual interaction" refers to the exchange of information and dialogue among multiple users who are physically separated, based on shared hobbies and interests, through an online platform.

[0395] "Entertainment information" refers to content such as music and videos that are provided according to the user's emotional state, with the aim of promoting relaxation and a change of pace.

[0396] "Social connection" refers to the sense of intimacy and camaraderie that users feel through interaction with other people.

[0397] This invention realizes a system that provides personalized support by having a server and terminals cooperate via a network to comprehensively manage the user's health status and emotions.

[0398] The server maintains a database to manage users' personal information and utilizes cloud services such as Firebase. It leverages the Google Cloud Natural Language API as its natural language processing engine to support user interaction. Furthermore, it uses machine learning libraries such as TensorFlow for sentiment analysis, analyzing video and audio data obtained from cameras and microphones in real time.

[0399] The device collects data on the user's daily life using devices such as smartphones and tablets. Equipped with a camera and microphone, it partially processes the acquired data locally and sends it to a server as needed, allowing for a more detailed assessment of emotional states.

[0400] As a concrete example, if the device detects through emotion analysis that an elderly person is feeling lonely during the day, the server will instruct the device to play relaxing music based on that data. Furthermore, as a suggestion for virtual interaction, it can recommend online events or hobby groups that the user might be interested in. Within this process, a possible prompt using a generative AI model might be: "Analyze the user's emotions from their face; if they appear sad, select and play soothing music. If they feel isolated, suggest a community event and encourage participation."

[0401] This system will enable support that makes users' daily lives safer and more fulfilling.

[0402] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0403] Step 1:

[0404] The device uses a camera and microphone to acquire video and audio data of the user's daily life. This inputs data indicating the user's emotional state, such as facial expressions and tone of voice.

[0405] Step 2:

[0406] The device inputs the acquired video and audio data into an emotion analysis model using TensorFlow, which analyzes the user's emotional state in real time. The emotion analysis model processes the data to identify the user's emotional state, such as "sad," "happy," or "anxious," and outputs this as an emotion label.

[0407] Step 3:

[0408] The server receives the emotion label sent from the terminal and refers to the personal information database. Based on this, if, for example, the emotion "sad" is detected, it selects entertainment information suitable for the user (e.g., relaxing music) and instructs the server to send this information to the terminal.

[0409] Step 4:

[0410] The terminal receives instructions from the server and provides the user with the specified entertainment information. During this process, it uses a music playback application to create an environment where the user can comfortably listen to the content.

[0411] Step 5:

[0412] When a user receives an emotional label indicating feelings of isolation, the server identifies virtual interaction opportunities based on the user's interests and sends information about relevant online events and community groups to the user's device. This allows the user to participate in virtual interactions that match their interests.

[0413] Step 6:

[0414] Finally, the device notifies the user of this information and uses an AI model to generate a message encouraging participation, displaying a prompt such as, "There's a new community event. Would you like to join?" This step helps motivate the user to actively build social connections.

[0415] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0416] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0417] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0418] [Third Embodiment]

[0419] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0420] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0421] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0422] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0423] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0425] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0426] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0427] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0428] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0429] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0430] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0431] This invention is an AI system for supporting the lives of the elderly, comprehensively providing data management based on personal information, natural language dialogue, image information analysis, health management notifications, schedule reminders, activity data analysis, and a virtual communication platform.

[0432] The entire system operates with a server and terminal working together. The server manages a database that stores user profiles and activity data, and uses a natural language processing engine to analyze user input and generate appropriate responses. For example, if a user says to the terminal, "I want to go for a walk outside today," the server will respond based on weather information, "It's sunny today, so you can comfortably take a walk during the daytime."

[0433] The device is equipped with a camera that periodically captures video of the user. The device analyzes the image information locally to evaluate the user's health and emotions. For example, if the device detects fatigue from the user's facial color, it sends this information to a server, which can then take appropriate action.

[0434] In the reminder function, the server calculates the timing for reminding users of important appointments and anniversaries based on their schedule. The server notifies the device of the reminder, and the device informs the user through voice or screen display. For example, it can notify users of things like, "Tomorrow is your grandchild's birthday."

[0435] Furthermore, the server periodically collects user activity data and generates feedback based on the analysis results. This feedback encourages improvements in daily life and provides positive messages such as, "Your step count has increased recently. Great job!"

[0436] Virtual social platforms allow users to interact with other users who share similar hobbies and interests via the internet. This helps reduce feelings of loneliness and allows users to feel a sense of social connection.

[0437] This invention thus improves the quality of life for the elderly and supports autonomous health management.

[0438] The following describes the processing flow.

[0439] Step 1:

[0440] The server stores personal information submitted by users in a database. This allows for the management of user profile information, which is then used as basic data for response generation and health management.

[0441] Step 2:

[0442] The terminal receives voice input from the user and converts it into text data via a speech recognition system. This converted text data is then sent to the server.

[0443] Step 3:

[0444] The server analyzes the received text data using a natural language processing engine and generates an appropriate response based on its content. This response is sent to the terminal and presented to the user.

[0445] Step 4:

[0446] The device periodically captures the user's face using its built-in camera. The acquired images are analyzed locally to evaluate the user's health and emotions.

[0447] Step 5:

[0448] The device sends evaluated health status and emotional information to the server. The server then sends health notifications to family members and care managers as needed.

[0449] Step 6:

[0450] The server notifies the device of upcoming important events and anniversaries based on a pre-configured schedule. The device then reminds the user visually and audibly.

[0451] Step 7:

[0452] The server collects and analyzes user activity data. The feedback generated from the analysis is provided to the user via their device, helping them to review their daily habits.

[0453] Step 8:

[0454] Users utilize a virtual social networking platform to gain the opportunity to interact online with other users who share similar hobbies and interests. The server provides the infrastructure to support this interaction.

[0455] (Example 1)

[0456] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0457] In the living environment of the elderly, challenges include insufficient health management, lack of social interaction, and a lack of feedback on daily activities. Elderly people living alone, in particular, have limited opportunities for daily health monitoring and appropriate advice. Furthermore, they often struggle to maintain social connections and suffer from feelings of loneliness. This invention aims to comprehensively solve these problems.

[0458] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0459] In this invention, the server includes an information aggregation and management unit that holds the user's personal information, means for storing profile information, means for generating appropriate responses using a natural language processing mechanism for engaging in natural language dialogue with the user, and means for evaluating the user's health status and emotions by analyzing visual information acquired using an imaging device. As a result, the user can live their daily life autonomously and safely while receiving real-time support.

[0460] The "Information Collection and Management Department" is an organization responsible for securely collecting, storing, and managing users' personal information and activity data.

[0461] A "natural language processing mechanism" is a technology that analyzes language data entered by users in the form of speech or text, understands its content, and generates an appropriate response.

[0462] An "imaging device" is a device used to acquire image data of a target object or person, and often includes a camera.

[0463] "Visual information" refers to data such as images and videos acquired by imaging devices, and this data is the subject of analysis.

[0464] A "health status assessment method" is a technology that analyzes visual information and other sensor information to determine the physical and psychological state of a user.

[0465] An "information sharing platform" is a digital platform that allows multiple users to interact with each other via the internet based on their hobbies and interests.

[0466] A "feedback generation method" is a technology that analyzes user activity data and provides users with improvement suggestions and encouraging messages based on the results.

[0467] "Generative machine learning models" refer to a group of algorithms that learn from data and have the function of making predictions or generating new data, and are used in natural language processing, among other applications.

[0468] This invention provides a specific embodiment of an AI system for supporting the daily lives of the elderly. This system operates with a server and terminals working in cooperation to comprehensively support the daily lives of the elderly.

[0469] The server includes an information aggregation and management unit that securely stores and manages users' personal information and activity data. This data includes users' profile information and daily activity details, and the database typically uses an open-source database management system. This allows for the personalization of support needs for each user.

[0470] The device has the functionality to interact directly with the user. When a user speaks to the device, the voice data is sent to a server equipped with a natural language processing mechanism. The server uses a generative AI model to understand the user's request and generate an appropriate response. For example, in response to a question like "What's the weather like today?", the server will answer "It's sunny today, perfect for a walk" based on the latest weather information. An example of a prompt is "Please suggest some simple exercises for seniors."

[0471] The terminal is also equipped with an imaging device that periodically captures images of the user. The terminal uses OpenCV and other image processing software to analyze these images and evaluate the user's health status (e.g., emotions from facial expressions, changes in physical condition from complexion). For example, if signs of fatigue are detected from the user's complexion, the server is notified, and a warning is sent to the maintenance manager.

[0472] Furthermore, the server has the functionality to connect multiple users through a virtual communication platform. This allows users to interact with other users who share common hobbies and interests via the internet, thereby strengthening their social connections. For example, when conducting an "online painting class" as part of a hobby, participants can share their artwork and exchange feedback with each other through the system.

[0473] In this way, the system improves the quality of life for users and supports autonomous and safe living.

[0474] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0475] Step 1:

[0476] The server uses an information aggregation and management unit to store user profile information and activity data in a database. It receives profile and daily activity data updated by users on their terminals as input, formats that data, and saves it to the database. The output is a dataset containing detailed information for each user. Through this process, the server builds a foundation for providing optimal services to users. Specifically, it performs database insertion and update operations in response to user change requests.

[0477] Step 2:

[0478] When a user speaks into the device, the device sends the voice input to the server. The server uses a natural language processing mechanism to convert the voice data into text and analyzes its content. The input is the user's voice data, and the output is an understanding of the intent and a corresponding response. Specifically, it uses a generative AI model to infer unclear words, retrieves information from external APIs as needed, and constructs an appropriate response.

[0479] Step 3:

[0480] An imaging device installed in the terminal periodically captures images of the user. The input is the captured visual information, which the terminal processes with an analysis tool. As output, it generates evaluation results of the user's health status and emotions and sends them to the server. Specifically, it uses an image analysis library to infer fatigue and emotions from changes in skin tone and facial expressions, and generates data useful for health management based on the results.

[0481] Step 4:

[0482] The server uses the collected data to remind users of scheduled events. It retrieves schedule information from a database as input and lists important events and anniversaries. As output, it generates reminder notifications and sends them to the device. Specifically, it uses a notification generation algorithm based on the scheduled date and time to send timely reminders to the user.

[0483] Step 5:

[0484] The server analyzes the user's daily activity data and generates feedback messages. The input is activity data sent from the device, which the server then uses for statistical analysis. The output is a message sent to the device containing improvement suggestions and encouragement. Specifically, it aggregates data such as steps taken and distance traveled, and creates evaluations and feedback based on certain criteria.

[0485] Step 6:

[0486] The server utilizes a virtual communication platform to facilitate interaction among multiple users. It uses information about users' interests and hobbies as input and suggests interaction events and groups to participate in as output. Specifically, it analyzes the interests registered by users and applies an algorithm to match them with other users who have similar interests.

[0487] (Application Example 1)

[0488] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0489] The loneliness and health management challenges faced by the elderly are serious. Many elderly people suffer from isolation and health problems due to a lack of adequate support in their daily lives. Furthermore, a lack of opportunities for interaction among multiple users and a resulting lack of social connection are also problematic. Technological means are needed to address these challenges and improve the quality of life for the elderly.

[0490] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0491] In this invention, the server includes means for storing user attribute information by managing a database that holds the user's personal information, means for generating appropriate responses using a natural language processing system for natural language dialogue with the user, and means for evaluating the user's health condition and emotions by analyzing image information acquired using imaging means. This makes it possible to understand the health condition of elderly people and provide them with emotional support. It also makes it possible to promote interaction with other users through care support devices and reduce feelings of loneliness.

[0492] A "database" is a collection of information used to comprehensively manage users' personal information and store their attribute information.

[0493] A "natural language processing system" is a language analysis device that engages in dialogue with users using natural language and generates appropriate responses.

[0494] An "imaging means" is a device that acquires image information using a camera or similar device and analyzes it.

[0495] "Health status" refers to information indicating the user's physical condition, and is evaluated by imaging techniques.

[0496] "Emotions" refer to information that indicates the user's mental state and are evaluated using imaging techniques and natural language processing systems.

[0497] "Notification" refers to the act of informing a remote administrator or the user themselves of information obtained based on an evaluation.

[0498] "Care support equipment" refers to automated devices used to assist elderly people in their daily lives.

[0499] "Virtual interaction" refers to the act of using information technology to facilitate online interaction between users based on their hobbies and interests.

[0500] "Attribute information" refers to data that shows basic information about a user and is stored in a database.

[0501] "Schedule information" refers to data that records appointments and anniversaries set by the user, and serves as the basis for notifications.

[0502] The system implementing this invention consists of a server and a terminal. The server manages a database that holds users' personal information and stores users' attribute information. By using a natural language processing system, the server analyzes user input and generates appropriate responses. This processing is performed using natural language processing libraries such as Python and NLTK.

[0503] The terminal, for example, uses a Raspberry Pi and a camera module to capture images of the user and analyze that information using OpenCV. The terminal then evaluates the user's health and emotions and sends the results to a server. Based on this information, the server assists with the user's health management and, if necessary, sends notifications to remote administrators.

[0504] The server also manages important appointments based on the schedule information set by the user. It provides schedule notifications to the user via voice or screen display, offering support to help them complete their daily tasks. Furthermore, it regularly collects user activity information, generates feedback from that data, and provides recommendations to help improve daily life.

[0505] Through care support devices, the server facilitates virtual interaction among users, providing online connections based on shared hobbies and interests. This virtual interaction helps users reduce feelings of loneliness and lead richer social lives.

[0506] As a concrete example, the monitoring care robot encourages the user's activity by asking, "Would you like to go for a walk today?" On sunny days, it might say, "The weather is nice today, so let's go for a walk together."

[0507] Example input prompts for a generative AI model:

[0508] "Please explain how AI support systems can be implemented in elderly care. I'd particularly like to focus on health monitoring and reducing feelings of isolation. I'd like detailed information on the necessary hardware and software, as well as their processing capabilities."

[0509] This system enables elderly individuals to manage their own health and receive emotional support. This invention aims to improve the quality of life for the elderly and provide a safe and secure living environment.

[0510] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0511] Step 1:

[0512] The device uses a camera module to acquire image information of the user. The input is real-time image data obtained from the camera, and this data is analyzed using the OpenCV library. Face recognition and skin tone analysis are performed to generate basic data for evaluating health status and emotions. The output is the results of the analyzed health status and emotional evaluation.

[0513] Step 2:

[0514] The terminal sends the evaluation results to the server. The input is the evaluation results of health status and emotions obtained in step 1. The server receives this and stores it in a database, enabling continuous monitoring of health status. The output is the user's health information stored in the database.

[0515] Step 3:

[0516] The user speaks to the server in natural language through a voice input device. The input consists of specific requests and questions from the user. The server uses a natural language processing system to analyze this voice input and generate an appropriate response. The output is the generated response message.

[0517] Step 4:

[0518] The server manages the user's scheduled schedule and creates important appointments and reminders. The input is the user's schedule information, and an algorithm calculates the timing of notifications. The output is a list of events to be notified.

[0519] Step 5:

[0520] The server sends the generated reminder and response messages to the terminal, and the terminal notifies the user via voice or screen display. The input is the data generated in steps 3 and 4. The output is the voice or visual notification to the user.

[0521] Step 6:

[0522] The care support device interacts with a server and provides an interface to facilitate virtual interaction among users. Inputs include data about the user's interests and preferences, as well as interaction requests from other users. Outputs are communication channels between those wishing to participate.

[0523] In this way, the system supports the daily lives of the elderly, reduces feelings of isolation, and manages their health. The use of generative AI models is expected to facilitate smooth natural language dialogue.

[0524] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0525] This invention is an AI system aimed at managing users' health and alleviating social isolation. In particular, by incorporating an emotion engine, it recognizes users' emotions and personalizes the content of conversations and support accordingly.

[0526] The system is configured so that the server and terminals communicate via a network. The server maintains a database for efficiently managing users' personal information and activity data, and is equipped with a natural language processing engine to enable natural conversations with users. Furthermore, by implementing an emotion engine, it is possible to evaluate users' emotions in real time.

[0527] The device is equipped with a camera and microphone to capture the user's daily video and audio. This information is analyzed by an emotion engine, either locally or on a server, to evaluate the user's emotional state. For example, if the device detects changes in the user's facial expressions or tone of voice, and the emotion engine identifies a "sad" state, the server can instruct the device to play relaxing music as an appropriate resource.

[0528] The reminder function uses the server to notify the user of their next appointment based on their schedule information. Furthermore, if the emotion engine determines that the user is in an unstable emotional state, it can also provide specific advice on preparing for important appointments.

[0529] Furthermore, the server analyzes activity data and generates feedback that helps with health management and daily life improvement. This feedback uses positive language to reinforce positive emotions and supports the improvement of daily habits.

[0530] Furthermore, the virtual interaction platform allows multiple users to connect based on shared hobbies and interests. If the emotion engine detects anxiety or loneliness, the server suggests joining a group with similar interests, providing an opportunity to build social connections.

[0531] Thus, the present invention is a system that, by using an emotion engine, enables the provision of more personalized support to users and enhances their mental and physical health.

[0532] The following describes the processing flow.

[0533] Step 1:

[0534] The device captures the user's face and voice using its camera and microphone. This allows for the acquisition of real-time image and audio data.

[0535] Step 2:

[0536] The device sends the acquired image and audio data to an emotion engine to analyze the user's emotions. For example, it analyzes facial muscle movements and voice tone to classify the emotional state into categories such as "joy," "anger," and "sadness."

[0537] Step 3:

[0538] The device receives the results analyzed by the emotion engine and sends them to the server. The server then uses this emotion data to generate the most appropriate response or suggestion.

[0539] Step 4:

[0540] The server uses a natural language processing engine to create responses appropriate to the user's emotional state. For example, if the user is perceived as "sad," it will generate a caring message such as, "Is something wrong today? Is there anything I can do to help?"

[0541] Step 5:

[0542] The server sends the generated response to the terminal, which then notifies the user. This is done through both audio and text, providing the user with reassurance and support.

[0543] Step 6:

[0544] Based on the user's schedule information, the server reminds the user of important appointments on their device. If the emotion engine determines that the user is experiencing stress, the server will notify the user with additional advice, such as "Make sure you allow yourself plenty of time to prepare."

[0545] Step 7:

[0546] The server analyzes user activity data and generates feedback to help with health management. If a user engages in positive activities on a daily basis, feedback praising their efforts is provided from the device.

[0547] Step 8:

[0548] If emotional data indicates feelings of loneliness, the server will encourage the user to participate in a virtual social platform. The user will then have the opportunity to build social connections by connecting with other users who share similar hobbies and interests.

[0549] Through this series of processes, the system can provide users with personalized support that is empathetic to their emotions.

[0550] (Example 2)

[0551] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0552] In modern society, managing people's health and the loss of social connections are serious challenges. The elderly and those with limited social contact due to remote work, in particular, tend to experience emotional isolation and difficulty managing their health. To improve this situation, there is a need to develop systems that provide individualized support that takes emotions into account, thereby improving the quality of daily life for each user.

[0553] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0554] In this invention, the server includes means for managing an information infrastructure that holds the user's personal information, means for generating appropriate responses using a language processing device for natural language dialogue with the user, and means for evaluating the user's health condition and emotions by analyzing video information acquired using an image input device. This enables the provision of personalized support tailored to the user's emotional state, thereby alleviating social isolation and improving health management.

[0555] "Information infrastructure" refers to the data management devices and information processing technologies necessary to securely hold and manage users' personal information and other related data.

[0556] A "language processing device" refers to a part or function of a computer system that analyzes natural language input and generates appropriate responses.

[0557] An "image input device" refers to hardware used to acquire video information of a user using a camera or other video acquisition equipment.

[0558] "Health status" refers to information related to the user's physical and mental health, including data such as heart rate, facial expressions, and voice tone.

[0559] "Evaluating emotions" refers to the process of analyzing and identifying a user's current emotions and psychological state based on facial expressions, voice, actions, etc.

[0560] "Personalized support" refers to an approach that provides optimal responses and services tailored to each user's emotions and health condition.

[0561] "Resolving social isolation" refers to alleviating feelings of loneliness and alienation that users experience in their daily lives, and enabling them to have richer social connections.

[0562] "Improving health management" refers to continuously monitoring the user's health status and providing appropriate feedback and advice as needed, enabling the user to maintain or improve their own health.

[0563] This invention is a comprehensive system for managing users' health and alleviating social isolation. The system consists of a server and terminals, each performing a specific function to provide personalized support to users.

[0564] The server operates as an information management platform, securely managing users' personal information and data related to their daily activities. The database system stores comprehensive information, including users' health history and schedules. Furthermore, by incorporating a natural language processing engine, the server can generate appropriate responses in real time based on text data and voice input received from users. The software used includes a "language processing unit" for natural language processing.

[0565] The device is equipped with a camera and microphone to capture video and audio of the user's daily life. This data is processed on a server or locally, and emotion analysis is performed. Emotion analysis involves analyzing changes in facial expressions and voice obtained from the "image input device" to identify the emotions the user is expressing. For example, if the device determines that the user is "sad," it will play relaxing music according to the server's instructions.

[0566] As a concrete example, a prompt message could look like this: "The next meeting is at 3 PM. If you're feeling nervous, take some time to take a deep breath." In addition, using a virtual social platform facilitates interaction between users with shared interests. For example, the server might suggest to users how to expand their social connections, such as, "Would you like to join an online cooking class?"

[0567] In this way, the server and terminal work together to provide a concrete system for improving the user's health and social connections. The system of the present invention provides users with powerful support to improve their quality of daily life and reduce feelings of isolation.

[0568] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0569] Step 1:

[0570] The device captures the user's daily activities through its camera and microphone. It acquires video and audio data of the user as input. This data is stored locally for a certain period and then compressed for transmission to the server. Specifically, the device uses the camera's activation and recording functions to capture video.

[0571] Step 2:

[0572] The server receives video and audio data sent from the terminal. It takes compressed video and audio files as input, decodes them, and makes them ready for analysis. The server decompresses the data and runs a program to "evaluate emotions" for sentiment analysis. Specifically, at this stage, the server extracts facial expressions from the video and analyzes the tone of voice from the audio data.

[0573] Step 3:

[0574] The server evaluates the user's emotional state from the analyzed data. It uses analyzed facial expression data and voice tone characteristics as input. Based on this information, a generative AI model identifies emotions such as "happy" or "sad." In this step, the server organizes and records the identified emotions in preparation for subsequent processing.

[0575] Step 4:

[0576] The server determines personalized responses and support based on the user's emotional state and sends instructions to the device. It uses the emotional information evaluated in the previous step as input. Based on the data processing, the server sends suggestions to the device, such as "play relaxing music" or "recommended online activities." Specific actions include displaying music provided by the device and links to suggested activities.

[0577] Step 5:

[0578] The device provides feedback to the user based on instructions received from the server. It receives commands from the server as input. Based on this data, the device plays music through its speaker and displays relevant information on its screen. Specifically, a music application is launched and a specified playlist is played, and a video device notifies the user of recommended activities.

[0579] (Application Example 2)

[0580] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0581] In recent years, the importance of individual health management has increased, and prompt and accurate responses to changes in health conditions are particularly required for the elderly and individuals with physical limitations. Furthermore, the problem of social isolation is also prominent, and given the current situation where many people experience feelings of isolation and anxiety, there is a need to provide means of fostering social connections that transcend physical distance. Because conventional technologies struggle to adequately address these individual and dynamic needs, a new system is required that improves health management while alleviating social isolation by providing personalized care tailored to emotional states.

[0582] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0583] In this invention, the server includes means for managing the user's personal information, means for providing corresponding entertainment information based on emotion analysis, and means for suggesting social connections through virtual interaction. This makes it possible to evaluate the user's emotional state in real time and provide care and opportunities for social interaction tailored to individual needs.

[0584] "Personal information" refers to information that can uniquely identify a user, such as their name, contact information, age, and health status.

[0585] A "natural language processing engine" is a technology that enables computers to understand and generate natural language used by humans, thereby supporting dialogue.

[0586] "Emotion analysis" is a technology that determines a user's emotional state by processing data obtained through cameras and microphones.

[0587] "Health management" refers to activities that monitor the physical and mental health status of users and provide appropriate care as needed.

[0588] "Virtual interaction" refers to the exchange of information and dialogue among multiple users who are physically separated, based on shared hobbies and interests, through an online platform.

[0589] "Entertainment information" refers to content such as music and videos that are provided according to the user's emotional state, with the aim of promoting relaxation and a change of pace.

[0590] "Social connection" refers to the sense of intimacy and camaraderie that users feel through interaction with other people.

[0591] This invention realizes a system that provides personalized support by having a server and terminals cooperate via a network to comprehensively manage the user's health status and emotions.

[0592] The server maintains a database to manage users' personal information and utilizes cloud services such as Firebase. It leverages the Google Cloud Natural Language API as its natural language processing engine to support user interaction. Furthermore, it uses machine learning libraries such as TensorFlow for sentiment analysis, analyzing video and audio data obtained from cameras and microphones in real time.

[0593] The device collects data on the user's daily life using devices such as smartphones and tablets. Equipped with a camera and microphone, it partially processes the acquired data locally and sends it to a server as needed, allowing for a more detailed assessment of emotional states.

[0594] As a concrete example, if the device detects through emotion analysis that an elderly person is feeling lonely during the day, the server will instruct the device to play relaxing music based on that data. Furthermore, as a suggestion for virtual interaction, it can recommend online events or hobby groups that the user might be interested in. Within this process, a possible prompt using a generative AI model might be: "Analyze the user's emotions from their face; if they appear sad, select and play soothing music. If they feel isolated, suggest a community event and encourage participation."

[0595] This system will enable support that makes users' daily lives safer and more fulfilling.

[0596] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0597] Step 1:

[0598] The device uses a camera and microphone to acquire video and audio data of the user's daily life. This inputs data indicating the user's emotional state, such as facial expressions and tone of voice.

[0599] Step 2:

[0600] The device inputs the acquired video and audio data into an emotion analysis model using TensorFlow, which analyzes the user's emotional state in real time. The emotion analysis model processes the data to identify the user's emotional state, such as "sad," "happy," or "anxious," and outputs this as an emotion label.

[0601] Step 3:

[0602] The server receives the emotion label sent from the terminal and refers to the personal information database. Based on this, if, for example, the emotion "sad" is detected, it selects entertainment information suitable for the user (e.g., relaxing music) and instructs the server to send this information to the terminal.

[0603] Step 4:

[0604] The terminal receives instructions from the server and provides the user with the specified entertainment information. During this process, it uses a music playback application to create an environment where the user can comfortably listen to the content.

[0605] Step 5:

[0606] When a user receives an emotional label indicating feelings of isolation, the server identifies virtual interaction opportunities based on the user's interests and sends information about relevant online events and community groups to the user's device. This allows the user to participate in virtual interactions that match their interests.

[0607] Step 6:

[0608] Finally, the device notifies the user of this information and uses an AI model to generate a message encouraging participation, displaying a prompt such as, "There's a new community event. Would you like to join?" This step helps motivate the user to actively build social connections.

[0609] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0610] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0611] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0612] [Fourth Embodiment]

[0613] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0614] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0615] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0616] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0617] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0618] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0619] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0620] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0621] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0622] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0623] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0624] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0625] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0626] This invention is an AI system for supporting the lives of the elderly, comprehensively providing data management based on personal information, natural language dialogue, image information analysis, health management notifications, schedule reminders, activity data analysis, and a virtual communication platform.

[0627] The entire system operates with a server and terminal working together. The server manages a database that stores user profiles and activity data, and uses a natural language processing engine to analyze user input and generate appropriate responses. For example, if a user says to the terminal, "I want to go for a walk outside today," the server will respond based on weather information, "It's sunny today, so you can comfortably take a walk during the daytime."

[0628] The device is equipped with a camera that periodically captures video of the user. The device analyzes the image information locally to evaluate the user's health and emotions. For example, if the device detects fatigue from the user's facial color, it sends this information to a server, which can then take appropriate action.

[0629] In the reminder function, the server calculates the timing for reminding users of important appointments and anniversaries based on their schedule. The server notifies the device of the reminder, and the device informs the user through voice or screen display. For example, it can notify users of things like, "Tomorrow is your grandchild's birthday."

[0630] Furthermore, the server periodically collects user activity data and generates feedback based on the analysis results. This feedback encourages improvements in daily life and provides positive messages such as, "Your step count has increased recently. Great job!"

[0631] Virtual social platforms allow users to interact with other users who share similar hobbies and interests via the internet. This helps reduce feelings of loneliness and allows users to feel a sense of social connection.

[0632] This invention thus improves the quality of life for the elderly and supports autonomous health management.

[0633] The following describes the processing flow.

[0634] Step 1:

[0635] The server stores personal information submitted by users in a database. This allows for the management of user profile information, which is then used as basic data for response generation and health management.

[0636] Step 2:

[0637] The terminal receives voice input from the user and converts it into text data via a speech recognition system. This converted text data is then sent to the server.

[0638] Step 3:

[0639] The server analyzes the received text data using a natural language processing engine and generates an appropriate response based on its content. This response is sent to the terminal and presented to the user.

[0640] Step 4:

[0641] The device periodically captures the user's face using its built-in camera. The acquired images are analyzed locally to evaluate the user's health and emotions.

[0642] Step 5:

[0643] The device sends evaluated health status and emotional information to the server. The server then sends health notifications to family members and care managers as needed.

[0644] Step 6:

[0645] The server notifies the device of upcoming important events and anniversaries based on a pre-configured schedule. The device then reminds the user visually and audibly.

[0646] Step 7:

[0647] The server collects and analyzes user activity data. The feedback generated from the analysis is provided to the user via their device, helping them to review their daily habits.

[0648] Step 8:

[0649] Users utilize a virtual social networking platform to gain the opportunity to interact online with other users who share similar hobbies and interests. The server provides the infrastructure to support this interaction.

[0650] (Example 1)

[0651] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0652] In the living environment of the elderly, challenges include insufficient health management, lack of social interaction, and a lack of feedback on daily activities. Elderly people living alone, in particular, have limited opportunities for daily health monitoring and appropriate advice. Furthermore, they often struggle to maintain social connections and suffer from feelings of loneliness. This invention aims to comprehensively solve these problems.

[0653] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0654] In this invention, the server includes an information aggregation and management unit that holds the user's personal information, means for storing profile information, means for generating appropriate responses using a natural language processing mechanism for engaging in natural language dialogue with the user, and means for evaluating the user's health status and emotions by analyzing visual information acquired using an imaging device. As a result, the user can live their daily life autonomously and safely while receiving real-time support.

[0655] The "Information Collection and Management Department" is an organization responsible for securely collecting, storing, and managing users' personal information and activity data.

[0656] A "natural language processing mechanism" is a technology that analyzes language data entered by users in the form of speech or text, understands its content, and generates an appropriate response.

[0657] An "imaging device" is a device used to acquire image data of a target object or person, and often includes a camera.

[0658] "Visual information" refers to data such as images and videos acquired by imaging devices, and this data is the subject of analysis.

[0659] A "health status assessment method" is a technology that analyzes visual information and other sensor information to determine the physical and psychological state of a user.

[0660] An "information sharing platform" is a digital platform that allows multiple users to interact with each other via the internet based on their hobbies and interests.

[0661] A "feedback generation method" is a technology that analyzes user activity data and provides users with improvement suggestions and encouraging messages based on the results.

[0662] "Generative machine learning models" refer to a group of algorithms that learn from data and have the function of making predictions or generating new data, and are used in natural language processing, among other applications.

[0663] This invention provides a specific embodiment of an AI system for supporting the daily lives of the elderly. This system operates with a server and terminals working in cooperation to comprehensively support the daily lives of the elderly.

[0664] The server includes an information aggregation and management unit that securely stores and manages users' personal information and activity data. This data includes users' profile information and daily activity details, and the database typically uses an open-source database management system. This allows for the personalization of support needs for each user.

[0665] The device has the functionality to interact directly with the user. When a user speaks to the device, the voice data is sent to a server equipped with a natural language processing mechanism. The server uses a generative AI model to understand the user's request and generate an appropriate response. For example, in response to a question like "What's the weather like today?", the server will answer "It's sunny today, perfect for a walk" based on the latest weather information. An example of a prompt is "Please suggest some simple exercises for seniors."

[0666] The terminal is also equipped with an imaging device that periodically captures images of the user. The terminal uses OpenCV and other image processing software to analyze these images and evaluate the user's health status (e.g., emotions from facial expressions, changes in physical condition from complexion). For example, if signs of fatigue are detected from the user's complexion, the server is notified, and a warning is sent to the maintenance manager.

[0667] Furthermore, the server has the functionality to connect multiple users through a virtual communication platform. This allows users to interact with other users who share common hobbies and interests via the internet, thereby strengthening their social connections. For example, when conducting an "online painting class" as part of a hobby, participants can share their artwork and exchange feedback with each other through the system.

[0668] In this way, the system improves the quality of life for users and supports autonomous and safe living.

[0669] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0670] Step 1:

[0671] The server uses an information aggregation and management unit to store user profile information and activity data in a database. It receives profile and daily activity data updated by users on their terminals as input, formats that data, and saves it to the database. The output is a dataset containing detailed information for each user. Through this process, the server builds a foundation for providing optimal services to users. Specifically, it performs database insertion and update operations in response to user change requests.

[0672] Step 2:

[0673] When a user speaks into the device, the device sends the voice input to the server. The server uses a natural language processing mechanism to convert the voice data into text and analyzes its content. The input is the user's voice data, and the output is an understanding of the intent and a corresponding response. Specifically, it uses a generative AI model to infer unclear words, retrieves information from external APIs as needed, and constructs an appropriate response.

[0674] Step 3:

[0675] An imaging device installed in the terminal periodically captures images of the user. The input is the captured visual information, which the terminal processes with an analysis tool. As output, it generates evaluation results of the user's health status and emotions and sends them to the server. Specifically, it uses an image analysis library to infer fatigue and emotions from changes in skin tone and facial expressions, and generates data useful for health management based on the results.

[0676] Step 4:

[0677] The server uses the collected data to remind users of scheduled events. It retrieves schedule information from a database as input and lists important events and anniversaries. As output, it generates reminder notifications and sends them to the device. Specifically, it uses a notification generation algorithm based on the scheduled date and time to send timely reminders to the user.

[0678] Step 5:

[0679] The server analyzes the user's daily activity data and generates feedback messages. The input is activity data sent from the device, which the server then uses for statistical analysis. The output is a message sent to the device containing improvement suggestions and encouragement. Specifically, it aggregates data such as steps taken and distance traveled, and creates evaluations and feedback based on certain criteria.

[0680] Step 6:

[0681] The server utilizes a virtual communication platform to facilitate interaction among multiple users. It uses information about users' interests and hobbies as input and suggests interaction events and groups to participate in as output. Specifically, it analyzes the interests registered by users and applies an algorithm to match them with other users who have similar interests.

[0682] (Application Example 1)

[0683] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0684] The loneliness and health management challenges faced by the elderly are serious. Many elderly people suffer from isolation and health problems due to a lack of adequate support in their daily lives. Furthermore, a lack of opportunities for interaction among multiple users and a resulting lack of social connection are also problematic. Technological means are needed to address these challenges and improve the quality of life for the elderly.

[0685] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0686] In this invention, the server includes means for storing user attribute information by managing a database that holds the user's personal information, means for generating appropriate responses using a natural language processing system for natural language dialogue with the user, and means for evaluating the user's health condition and emotions by analyzing image information acquired using imaging means. This makes it possible to understand the health condition of elderly people and provide them with emotional support. It also makes it possible to promote interaction with other users through care support devices and reduce feelings of loneliness.

[0687] A "database" is a collection of information used to comprehensively manage users' personal information and store their attribute information.

[0688] A "natural language processing system" is a language analysis device that engages in dialogue with users using natural language and generates appropriate responses.

[0689] An "imaging means" is a device that acquires image information using a camera or similar device and analyzes it.

[0690] "Health status" refers to information indicating the user's physical condition, and is evaluated by imaging techniques.

[0691] "Emotions" refer to information that indicates the user's mental state and are evaluated using imaging techniques and natural language processing systems.

[0692] "Notification" refers to the act of informing a remote administrator or the user themselves of information obtained based on an evaluation.

[0693] "Care support equipment" refers to automated devices used to assist elderly people in their daily lives.

[0694] "Virtual interaction" refers to the act of using information technology to facilitate online interaction between users based on their hobbies and interests.

[0695] "Attribute information" refers to data that shows basic information about a user and is stored in a database.

[0696] "Schedule information" refers to data that records appointments and anniversaries set by the user, and serves as the basis for notifications.

[0697] The system implementing this invention consists of a server and a terminal. The server manages a database that holds users' personal information and stores users' attribute information. By using a natural language processing system, the server analyzes user input and generates appropriate responses. This processing is performed using natural language processing libraries such as Python and NLTK.

[0698] The terminal, for example, uses a Raspberry Pi and a camera module to capture images of the user and analyze that information using OpenCV. The terminal then evaluates the user's health and emotions and sends the results to a server. Based on this information, the server assists with the user's health management and, if necessary, sends notifications to remote administrators.

[0699] The server also manages important appointments based on the schedule information set by the user. It provides schedule notifications to the user via voice or screen display, offering support to help them complete their daily tasks. Furthermore, it regularly collects user activity information, generates feedback from that data, and provides recommendations to help improve daily life.

[0700] Through care support devices, the server facilitates virtual interaction among users, providing online connections based on shared hobbies and interests. This virtual interaction helps users reduce feelings of loneliness and lead richer social lives.

[0701] As a concrete example, the monitoring care robot encourages the user's activity by asking, "Would you like to go for a walk today?" On sunny days, it might say, "The weather is nice today, so let's go for a walk together."

[0702] Example input prompts for a generative AI model:

[0703] "Please explain how AI support systems can be implemented in elderly care. I'd particularly like to focus on health monitoring and reducing feelings of isolation. I'd like detailed information on the necessary hardware and software, as well as their processing capabilities."

[0704] This system enables elderly individuals to manage their own health and receive emotional support. This invention aims to improve the quality of life for the elderly and provide a safe and secure living environment.

[0705] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0706] Step 1:

[0707] The device uses a camera module to acquire image information of the user. The input is real-time image data obtained from the camera, and this data is analyzed using the OpenCV library. Face recognition and skin tone analysis are performed to generate basic data for evaluating health status and emotions. The output is the results of the analyzed health status and emotional evaluation.

[0708] Step 2:

[0709] The terminal sends the evaluation results to the server. The input is the evaluation results of health status and emotions obtained in step 1. The server receives this and stores it in a database, enabling continuous monitoring of health status. The output is the user's health information stored in the database.

[0710] Step 3:

[0711] The user speaks to the server in natural language through a voice input device. The input consists of specific requests and questions from the user. The server uses a natural language processing system to analyze this voice input and generate an appropriate response. The output is the generated response message.

[0712] Step 4:

[0713] The server manages the user's scheduled schedule and creates important appointments and reminders. The input is the user's schedule information, and an algorithm calculates the timing of notifications. The output is a list of events to be notified.

[0714] Step 5:

[0715] The server sends the generated reminder and response messages to the terminal, and the terminal notifies the user via voice or screen display. The input is the data generated in steps 3 and 4. The output is the voice or visual notification to the user.

[0716] Step 6:

[0717] The care support device interacts with a server and provides an interface to facilitate virtual interaction among users. Inputs include data about the user's interests and preferences, as well as interaction requests from other users. Outputs are communication channels between those wishing to participate.

[0718] In this way, the system supports the daily lives of the elderly, reduces feelings of isolation, and manages their health. The use of generative AI models is expected to facilitate smooth natural language dialogue.

[0719] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0720] This invention is an AI system aimed at managing users' health and alleviating social isolation. In particular, by incorporating an emotion engine, it recognizes users' emotions and personalizes the content of conversations and support accordingly.

[0721] The system is configured so that the server and terminals communicate via a network. The server maintains a database for efficiently managing users' personal information and activity data, and is equipped with a natural language processing engine to enable natural conversations with users. Furthermore, by implementing an emotion engine, it is possible to evaluate users' emotions in real time.

[0722] The device is equipped with a camera and microphone to capture the user's daily video and audio. This information is analyzed by an emotion engine, either locally or on a server, to evaluate the user's emotional state. For example, if the device detects changes in the user's facial expressions or tone of voice, and the emotion engine identifies a "sad" state, the server can instruct the device to play relaxing music as an appropriate resource.

[0723] The reminder function uses the server to notify the user of their next appointment based on their schedule information. Furthermore, if the emotion engine determines that the user is in an unstable emotional state, it can also provide specific advice on preparing for important appointments.

[0724] Furthermore, the server analyzes activity data and generates feedback that helps with health management and daily life improvement. This feedback uses positive language to reinforce positive emotions and supports the improvement of daily habits.

[0725] Furthermore, the virtual interaction platform allows multiple users to connect based on shared hobbies and interests. If the emotion engine detects anxiety or loneliness, the server suggests joining a group with similar interests, providing an opportunity to build social connections.

[0726] Thus, the present invention is a system that, by using an emotion engine, enables the provision of more personalized support to users and enhances their mental and physical health.

[0727] The following describes the processing flow.

[0728] Step 1:

[0729] The device captures the user's face and voice using its camera and microphone. This allows for the acquisition of real-time image and audio data.

[0730] Step 2:

[0731] The device sends the acquired image and audio data to an emotion engine to analyze the user's emotions. For example, it analyzes facial muscle movements and voice tone to classify the emotional state into categories such as "joy," "anger," and "sadness."

[0732] Step 3:

[0733] The device receives the results analyzed by the emotion engine and sends them to the server. The server then uses this emotion data to generate the most appropriate response or suggestion.

[0734] Step 4:

[0735] The server uses a natural language processing engine to create responses appropriate to the user's emotional state. For example, if the user is perceived as "sad," it will generate a caring message such as, "Is something wrong today? Is there anything I can do to help?"

[0736] Step 5:

[0737] The server sends the generated response to the terminal, which then notifies the user. This is done through both audio and text, providing the user with reassurance and support.

[0738] Step 6:

[0739] Based on the user's schedule information, the server reminds the user of important appointments on their device. If the emotion engine determines that the user is experiencing stress, the server will notify the user with additional advice, such as "Make sure you allow yourself plenty of time to prepare."

[0740] Step 7:

[0741] The server analyzes user activity data and generates feedback to help with health management. If a user engages in positive activities on a daily basis, feedback praising their efforts is provided from the device.

[0742] Step 8:

[0743] If emotional data indicates feelings of loneliness, the server will encourage the user to participate in a virtual social platform. The user will then have the opportunity to build social connections by connecting with other users who share similar hobbies and interests.

[0744] Through this series of processes, the system can provide users with personalized support that is empathetic to their emotions.

[0745] (Example 2)

[0746] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0747] In modern society, managing people's health and the loss of social connections are serious challenges. The elderly and those with limited social contact due to remote work, in particular, tend to experience emotional isolation and difficulty managing their health. To improve this situation, there is a need to develop systems that provide individualized support that takes emotions into account, thereby improving the quality of daily life for each user.

[0748] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0749] In this invention, the server includes means for managing an information infrastructure that holds the user's personal information, means for generating appropriate responses using a language processing device for natural language dialogue with the user, and means for evaluating the user's health condition and emotions by analyzing video information acquired using an image input device. This enables the provision of personalized support tailored to the user's emotional state, thereby alleviating social isolation and improving health management.

[0750] "Information infrastructure" refers to the data management devices and information processing technologies necessary to securely hold and manage users' personal information and other related data.

[0751] A "language processing device" refers to a part or function of a computer system that analyzes natural language input and generates appropriate responses.

[0752] An "image input device" refers to hardware used to acquire video information of a user using a camera or other video acquisition equipment.

[0753] "Health status" refers to information related to the user's physical and mental health, including data such as heart rate, facial expressions, and voice tone.

[0754] "Evaluating emotions" refers to the process of analyzing and identifying a user's current emotions and psychological state based on facial expressions, voice, actions, etc.

[0755] "Personalized support" refers to an approach that provides optimal responses and services tailored to each user's emotions and health condition.

[0756] "Resolving social isolation" refers to alleviating feelings of loneliness and alienation that users experience in their daily lives, and enabling them to have richer social connections.

[0757] "Improving health management" refers to continuously monitoring the user's health status and providing appropriate feedback and advice as needed, enabling the user to maintain or improve their own health.

[0758] This invention is a comprehensive system for managing users' health and alleviating social isolation. The system consists of a server and terminals, each performing a specific function to provide personalized support to users.

[0759] The server operates as an information management platform, securely managing users' personal information and data related to their daily activities. The database system stores comprehensive information, including users' health history and schedules. Furthermore, by incorporating a natural language processing engine, the server can generate appropriate responses in real time based on text data and voice input received from users. The software used includes a "language processing unit" for natural language processing.

[0760] The device is equipped with a camera and microphone to capture video and audio of the user's daily life. This data is processed on a server or locally, and emotion analysis is performed. Emotion analysis involves analyzing changes in facial expressions and voice obtained from the "image input device" to identify the emotions the user is expressing. For example, if the device determines that the user is "sad," it will play relaxing music according to the server's instructions.

[0761] As a concrete example, a prompt message could look like this: "The next meeting is at 3 PM. If you're feeling nervous, take some time to take a deep breath." In addition, using a virtual social platform facilitates interaction between users with shared interests. For example, the server might suggest to users how to expand their social connections, such as, "Would you like to join an online cooking class?"

[0762] In this way, the server and terminal work together to provide a concrete system for improving the user's health and social connections. The system of the present invention provides users with powerful support to improve their quality of daily life and reduce feelings of isolation.

[0763] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0764] Step 1:

[0765] The device captures the user's daily activities through its camera and microphone. It acquires video and audio data of the user as input. This data is stored locally for a certain period and then compressed for transmission to the server. Specifically, the device uses the camera's activation and recording functions to capture video.

[0766] Step 2:

[0767] The server receives video and audio data sent from the terminal. It takes compressed video and audio files as input, decodes them, and makes them ready for analysis. The server decompresses the data and runs a program to "evaluate emotions" for sentiment analysis. Specifically, at this stage, the server extracts facial expressions from the video and analyzes the tone of voice from the audio data.

[0768] Step 3:

[0769] The server evaluates the user's emotional state from the analyzed data. It uses analyzed facial expression data and voice tone characteristics as input. Based on this information, a generative AI model identifies emotions such as "happy" or "sad." In this step, the server organizes and records the identified emotions in preparation for subsequent processing.

[0770] Step 4:

[0771] The server determines personalized responses and support based on the user's emotional state and sends instructions to the device. It uses the emotional information evaluated in the previous step as input. Based on the data processing, the server sends suggestions to the device, such as "play relaxing music" or "recommended online activities." Specific actions include displaying music provided by the device and links to suggested activities.

[0772] Step 5:

[0773] The device provides feedback to the user based on instructions received from the server. It receives commands from the server as input. Based on this data, the device plays music through its speaker and displays relevant information on its screen. Specifically, a music application is launched and a specified playlist is played, and a video device notifies the user of recommended activities.

[0774] (Application Example 2)

[0775] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0776] In recent years, the importance of individual health management has increased, and prompt and accurate responses to changes in health conditions are particularly required for the elderly and individuals with physical limitations. Furthermore, the problem of social isolation is also prominent, and given the current situation where many people experience feelings of isolation and anxiety, there is a need to provide means of fostering social connections that transcend physical distance. Because conventional technologies struggle to adequately address these individual and dynamic needs, a new system is required that improves health management while alleviating social isolation by providing personalized care tailored to emotional states.

[0777] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0778] In this invention, the server includes means for managing the user's personal information, means for providing corresponding entertainment information based on emotion analysis, and means for suggesting social connections through virtual interaction. This makes it possible to evaluate the user's emotional state in real time and provide care and opportunities for social interaction tailored to individual needs.

[0779] "Personal information" refers to information that can uniquely identify a user, such as their name, contact information, age, and health status.

[0780] A "natural language processing engine" is a technology that enables computers to understand and generate natural language used by humans, thereby supporting dialogue.

[0781] "Emotion analysis" is a technology that determines a user's emotional state by processing data obtained through cameras and microphones.

[0782] "Health management" refers to activities that monitor the physical and mental health status of users and provide appropriate care as needed.

[0783] "Virtual interaction" refers to the exchange of information and dialogue among multiple users who are physically separated, based on shared hobbies and interests, through an online platform.

[0784] "Entertainment information" refers to content such as music and videos that are provided according to the user's emotional state, with the aim of promoting relaxation and a change of pace.

[0785] "Social connection" refers to the sense of intimacy and camaraderie that users feel through interaction with other people.

[0786] This invention realizes a system that provides personalized support by having a server and terminals cooperate via a network to comprehensively manage the user's health status and emotions.

[0787] The server maintains a database to manage users' personal information and utilizes cloud services such as Firebase. It leverages the Google Cloud Natural Language API as its natural language processing engine to support user interaction. Furthermore, it uses machine learning libraries such as TensorFlow for sentiment analysis, analyzing video and audio data obtained from cameras and microphones in real time.

[0788] The device collects data on the user's daily life using devices such as smartphones and tablets. Equipped with a camera and microphone, it partially processes the acquired data locally and sends it to a server as needed, allowing for a more detailed assessment of emotional states.

[0789] As a concrete example, if the device detects through emotion analysis that an elderly person is feeling lonely during the day, the server will instruct the device to play relaxing music based on that data. Furthermore, as a suggestion for virtual interaction, it can recommend online events or hobby groups that the user might be interested in. Within this process, a possible prompt using a generative AI model might be: "Analyze the user's emotions from their face; if they appear sad, select and play soothing music. If they feel isolated, suggest a community event and encourage participation."

[0790] This system will enable support that makes users' daily lives safer and more fulfilling.

[0791] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0792] Step 1:

[0793] The device uses a camera and microphone to acquire video and audio data of the user's daily life. This inputs data indicating the user's emotional state, such as facial expressions and tone of voice.

[0794] Step 2:

[0795] The device inputs the acquired video and audio data into an emotion analysis model using TensorFlow, which analyzes the user's emotional state in real time. The emotion analysis model processes the data to identify the user's emotional state, such as "sad," "happy," or "anxious," and outputs this as an emotion label.

[0796] Step 3:

[0797] The server receives the emotion label sent from the terminal and refers to the personal information database. Based on this, if, for example, the emotion "sad" is detected, it selects entertainment information suitable for the user (e.g., relaxing music) and instructs the server to send this information to the terminal.

[0798] Step 4:

[0799] The terminal receives instructions from the server and provides the user with the specified entertainment information. During this process, it uses a music playback application to create an environment where the user can comfortably listen to the content.

[0800] Step 5:

[0801] When a user receives an emotional label indicating feelings of isolation, the server identifies virtual interaction opportunities based on the user's interests and sends information about relevant online events and community groups to the user's device. This allows the user to participate in virtual interactions that match their interests.

[0802] Step 6:

[0803] Finally, the device notifies the user of this information and uses an AI model to generate a message encouraging participation, displaying a prompt such as, "There's a new community event. Would you like to join?" This step helps motivate the user to actively build social connections.

[0804] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0805] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0806] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0807] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0808] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0809] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0810] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0811] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0812] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0813] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0814] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0815] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0816] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0817] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0818] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0819] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0820] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0821] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0822] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0823] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0824] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0825] The following is further disclosed regarding the embodiments described above.

[0826] (Claim 1)

[0827] By managing a database that holds users' personal information, a means of saving users' profile information is provided.

[0828] A means for generating appropriate responses using a natural language processing engine for natural language dialogue with users,

[0829] A means of evaluating the user's health status and emotions by analyzing image information acquired using a camera,

[0830] Based on the aforementioned evaluation, a means of notifying remote administrators and assisting in the health management of users is provided.

[0831] A means of reminding users of important appointments and anniversaries based on their set schedule information,

[0832] A system that includes this.

[0833] (Claim 2)

[0834] The system according to claim 1, which generates feedback from user activity data and makes suggestions for improving daily life.

[0835] (Claim 3)

[0836] The system according to claim 1, which provides a platform for promoting virtual interaction among multiple users based on shared hobbies and interests.

[0837] "Example 1"

[0838] (Claim 1)

[0839] It includes an information aggregation and management unit that holds users' personal information, and means for storing users' profile information,

[0840] A means for generating an appropriate response using a natural language processing mechanism for natural language dialogue with users,

[0841] A means of evaluating the user's health status and emotions by analyzing visual information acquired using an imaging device,

[0842] Based on the aforementioned evaluation, a means of notifying remote administrators and assisting in the health management of users is provided.

[0843] A means of reminding users of important appointments and anniversaries based on their scheduled information,

[0844] A means of regularly analyzing users' daily activity data and generating feedback,

[0845] A means of providing an information sharing platform to promote virtual interaction among multiple users based on their hobbies and interests,

[0846] A system that includes this.

[0847] (Claim 2)

[0848] The system according to claim 1, which monitors changes in the user's health status and notifies a warning when an abnormality is detected.

[0849] (Claim 3)

[0850] The system according to claim 1, which analyzes acquired audio data using a generative machine learning model and proposes the optimal action.

[0851] "Application Example 1"

[0852] (Claim 1)

[0853] By managing a database that holds users' personal information, a means of storing users' attribute information is provided.

[0854] A means for generating appropriate responses using a natural language processing system for natural language dialogue with users,

[0855] A means for evaluating the user's health condition and emotions by analyzing image information acquired using imaging means,

[0856] Based on the aforementioned evaluation, a means of notifying remote administrators and supporting the health management of users is provided.

[0857] A means of notifying users of important appointments and anniversaries based on the schedule information they have set,

[0858] It is incorporated into care support equipment and is a means of supporting the user's daily life.

[0859] A means of providing an environment that facilitates virtual interaction among multiple users based on shared hobbies and interests via robots,

[0860] A system that includes this.

[0861] (Claim 2)

[0862] The system according to claim 1, which generates feedback from user activity information and makes recommendations for improving daily life.

[0863] (Claim 3)

[0864] The system according to claim 1, which uses a natural language processing system and communication equipment to reduce feelings of loneliness through care support equipment.

[0865] "Example 2 of combining an emotion engine"

[0866] (Claim 1)

[0867] By managing the information infrastructure that holds users' personal information, a means of storing users' characteristic information is provided.

[0868] A means for generating an appropriate response using a language processing device for natural language dialogue with a user,

[0869] A means of evaluating the user's health condition and emotions by analyzing video information acquired using an image input device,

[0870] Based on the aforementioned evaluation, a means of assisting with the health management of users will be provided by notifying the supervisor in a remote location.

[0871] A means of reminding users of important matters and special days based on the schedule information they have set,

[0872] A means of providing personalized responses to users based on emotion analysis,

[0873] A means of providing music and videos that correspond to the user's emotional state based on the results of emotion analysis,

[0874] A system that includes this.

[0875] (Claim 2)

[0876] The system according to claim 1, which generates information from user behavior data and makes suggestions for improving daily life.

[0877] (Claim 3)

[0878] The system according to claim 1, which provides a foundation for facilitating virtual interaction based on common interests among multiple users.

[0879] "Application example 2 when combining with an emotional engine"

[0880] (Claim 1)

[0881] By managing a database that holds users' personal information, a means of saving users' profile information is provided.

[0882] A means for generating appropriate responses using a natural language processing engine for natural language dialogue with users,

[0883] A means of evaluating the user's health status and emotions by analyzing image information acquired using a camera,

[0884] Based on the aforementioned evaluation, a means of notifying remote administrators and assisting in the health management of users is provided.

[0885] A means of reminding users of important appointments and anniversaries based on their set schedule information,

[0886] A means of providing entertainment information that corresponds to an emotional state,

[0887] A means of proposing virtual interaction to alleviate feelings of isolation and anxiety,

[0888] A system that includes this.

[0889] (Claim 2)

[0890] The system according to claim 1, which generates feedback from user activity data and makes suggestions for improving daily life.

[0891] (Claim 3)

[0892] The system according to claim 1, which proposes to promote virtual interaction among multiple users based on their hobbies and interests, and to build social connections. [Explanation of symbols]

[0893] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. By managing a database that holds users' personal information, a means of saving users' profile information is provided. A means for generating appropriate responses using a natural language processing engine for natural language dialogue with users, A means of evaluating the user's health status and emotions by analyzing image information acquired using a camera, Based on the aforementioned evaluation, a means of notifying remote administrators and assisting in the health management of users is provided. A means of reminding users of important appointments and anniversaries based on their set schedule information, A system that includes this.

2. The system according to claim 1, which generates feedback from user activity data and makes suggestions for improving daily life.

3. The system according to claim 1, which provides a platform for promoting virtual interaction among multiple users based on their hobbies and interests.

Images