System

Abstract

A system is provided.SOLUTION: A system including a terminal used by an elderly person, a server that executes a generative model, and a database that manages dialogue data with a user, wherein the terminal includes means for presenting a natural dialogue sentence to the user, the generative model includes means for generating a dialogue with the user, the server includes means for analyzing the dialogue data and evaluating a health condition, and the server includes means for notifying in a case where an abnormality in the health condition is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to solve multiple issues facing Japan's rapidly aging society. Elderly people face a wide range of problems, including feelings of loneliness and a lack of purpose in life, the difficulty of family members living far away from them to monitor their daily lives and monitor their health, rising medical costs, and a shortened healthy life expectancy. In particular, there are concerns that a lack of daily communication among elderly people living alone may have a negative impact on their mental health. [Means for solving the problem]

[0005] The present invention provides a system including a terminal used by an elderly person, a server that executes a generative model, and a database that manages dialogue data with the user. In this system, the terminal includes a means for presenting natural dialogue to the user, and the generative model includes a means for generating dialogue with the user. The server also includes a means for analyzing the dialogue data to evaluate the health status and a means for notifying the elderly person if an abnormality in the health status is detected. The generative model detects specific keywords through everyday conversations and evaluates the health status based on the keywords. The server also periodically generates health status reports and provides them to designated family members or medical institutions, thereby supporting the elderly person's health management.

[0006] "Devices" refer to electronic devices such as tablets and smartphones used by seniors.

[0007] A "generative model" is an artificial intelligence technique used to generate natural-sounding dialogue with a user.

[0008] A "server" is a computer system for executing generative models and managing and analyzing user interaction data.

[0009] A "database" is a collection of data that stores user interaction data and makes it accessible as needed.

[0010] "Dialogue data" is the content of the conversation between the user and the generative model, and is data saved in audio or text format.

[0011] "Health status" refers to the physical and mental state or condition of a user.

[0012] "Abnormal" refers to an unusual condition or problem in health.

[0013] "Notification" refers to information sent to family members or medical institutions when an abnormality in health status is detected.

[0014] A "Report" is a report summarizing a user's health status over a period of time and provided to a designated recipient.

[0015] "Keywords" are important words and phrases that the generative model extracts from the dialogue data and serve as clues for assessing health status. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] MODE FOR CARRYING OUT THE INVENTION

[0038] The present invention provides an interactive medical AI service system for elderly people. Specific embodiments of the system are described below.

[0039] Components

[0040] 1. Devices: Electronic devices such as tablets and smartphones used by the elderly. They provide an interface that allows users to converse with panda characters on a daily basis.

[0041] 2. Server: A computer system that executes the generative model, manages and analyzes the dialogue data with the user, and transmits the generated dialogue and health assessment data to the terminal.

[0042] 3. Generative Model: This is an artificial intelligence technique used to generate natural dialogue with the user, detect specific keywords, and assess health status based on them.

[0043] 4. Database: A collection of data that stores user interaction data and can be accessed as needed.

[0044] System Operation

[0045] 1. Initial Setup: When a user uses the service for the first time, they install the app on their device and enter their personal information (name, age, health information, etc.). This information is sent to the server and stored in a database.

[0046] 2. Start conversation: The user presses the "Start conversation" button on the device. The device sends this request to the server, which uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[0047] 3. Collection and analysis of everyday conversations: As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The device presents the generated new conversation data to the user by displaying and playing it back on the panda character.

[0048] 4. Health monitoring: The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities. Based on this, the generative model generates appropriate responses and questions. For example, if the user says, "My lower back hurts," the server instructs the generative model to ask more detailed questions about the extent and location of the pain.

[0049] 5. Abnormality detection and notification: If the server detects an abnormality in health status, the notification function will send the information to family members and medical institutions in real time. For example, if someone says, "I have chest pain," a notification will be sent immediately to family members and emergency medical institutions, and if necessary, an emergency call will be made.

[0050] 6. Report Generation: The server periodically generates a report summarizing the user's health status and emails it to designated family members or healthcare providers. The report includes a summary of the conversation, any abnormalities detected, and recommended next steps.

[0051] Specific examples

[0052] Scenario: Conversation with an elderly person complaining of lower back pain

[0053] 1. Start a conversation

[0054] Terminal: "Hello! How are you today?" asks the panda character to the user.

[0055] User: "I've been having back pain lately..."

[0056] Terminal: Converts the user's speech into text and sends it to the server.

[0057] 2. Conversation Analysis and Anomaly Detection

[0058] Server: Analyzes the keyword "my lower back hurts," and based on this, the generative model generates a new dialogue: "That's worrying. Where does it hurt?"

[0059] Server: Sends the generated dialogue to the terminal.

[0060] Device: Panda will say and listen, "That's worrying. Where does it hurt?"

[0061] 3. Notifications and Reporting

[0062] Server: Detects abnormalities based on data related to lower back pain and sends a notification to the family member stating, "You have recently been complaining of lower back pain. A more detailed diagnosis may be necessary."

[0063] Server: Generates periodic health status reports and sends them to family members.

[0064] This invention allows elderly people to enjoy daily conversations with panda characters while simultaneously managing their health.Furthermore, family members and medical institutions can monitor their health status in real time, contributing to improving the quality of life for elderly people.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] Device: Display the app installation screen.

[0068] User: Installs and launches the app.

[0069] Step 2:

[0070] Device: Display the initial setup screen (name, age, health information, etc.).

[0071] User: Enter the required information and press the send button.

[0072] Terminal: Sends the entered information to the server.

[0073] Server: Stores the received information in a database and sends a response to the terminal indicating that initial setup is complete.

[0074] Terminal: Notify the user that the initial setup is complete.

[0075] Step 3:

[0076] Device: Display a "Start Conversation" button on the home screen.

[0077] User: Press the start conversation button.

[0078] Terminal: Sends a request to start a conversation to the server.

[0079] Step 4:

[0080] Server: Initiates the conversation session and generates the initial greeting message.

[0081] Server: Sends a greeting message to the terminal.

[0082] On your device: A panda character will display and / or play a greeting message (e.g., "Hello! How was your day today?").

[0083] Step 5:

[0084] User: Answer the panda character's questions (e.g., "It was nice weather today so I went for a walk").

[0085] Terminal: Converts the user's speech into text and sends it to the server.

[0086] Step 6:

[0087] Server: Analyzes the received text data and generates the next question or response.

[0088] Server: Sends the generated conversation data to the terminal.

[0089] On your device: The panda character will display and play your new message (e.g., "That's lovely! Which way did you walk?").

[0090] Step 7:

[0091] Server: Continuously analyzes conversation data and detects keywords and anomalies.

[0092] Server: Generates new responses and questions based on the detected information.

[0093] Device: View and play new questions and responses from the panda character.

[0094] Step 8:

[0095] User: Provides health information as the conversation continues (e.g., "I've been having a lot of back pain lately...").

[0096] Terminal: Converts the user's speech into text and sends it to the server.

[0097] Step 9:

[0098] Server: Analyzes keywords that indicate abnormal health conditions, such as "lower back pain," and detects abnormalities.

[0099] Server: If an abnormality is detected, it generates a notification to notify family members or medical institutions.

[0100] Server: Send notifications to family members or medical institutions (e.g., "You've recently been complaining of back pain. You may need further evaluation.").

[0101] Step 10:

[0102] Server: Periodically aggregates conversation data and generates a health status report for the elderly.

[0103] Server: Sends the generated report to the family or medical institution.

[0104] Step 11:

[0105] Device: Continue everyday conversations with panda characters.

[0106] User: Continue to provide information about health status through interactions with the panda character.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] It is difficult for elderly people to manage their own health status in their daily lives and detect abnormalities early. It is also difficult for family members and medical institutions to grasp the health status of elderly people in real time. This increases health risks, especially for elderly people living alone. Furthermore, there is currently a lack of systems that assess the health status of elderly people through natural dialogue.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes a means for analyzing the dialogue data to evaluate the health condition, a means for notifying when an abnormality in the health condition is detected, and a means for continuously monitoring the health condition based on the dialogue data. This allows the user to manage their own health condition through natural dialogue, and enables family members and medical institutions to understand the health condition in real time.

[0112] "Elderly" refers to users who are aging and who are particularly in need of medical and nursing care support.

[0113] An "information terminal" is an electronic device used by a user, and refers to a device that has the shape of a tablet, smartphone, etc.

[0114] A "natural language generation model" is an artificial intelligence technology used to generate natural dialogue with users, and specifically refers to a computer program that performs language processing.

[0115] "Computer system" refers generally to a computer system for executing generative models and analyzing and managing interaction data.

[0116] "Data Collection" refers to a database system that stores user interaction data and makes it accessible as needed.

[0117] "Natural dialogue" refers to text that is generated naturally in interaction with a user, just like human language.

[0118] "Interaction Data" refers to data including voice, text, and other related information relating to a conversation between a user and a system.

[0119] "Health status" refers to the state of the user's physical and psychological health, and specifically includes physical condition, presence or absence of pain, mental state, etc.

[0120] "Abnormal" refers to any symptom or sign that deviates from normal health, especially one that requires medical attention.

[0121] "Notification" refers to the process of communicating information about an abnormality in a user's health condition to family members or medical institutions.

[0122] "Monitoring" refers to the process of continuously observing a user's health status and assessing it based on specific conditions.

[0123] MODE FOR CARRYING OUT THE INVENTION

[0124] The present invention provides an interactive medical AI service system for elderly people. Specific embodiments of the system are described below.

[0125] Hardware and Software Configuration

[0126] Device:

[0127] This refers to tablets and smartphones that are used daily by seniors, including iPads, Android tablets, iPhones, and Android smartphones. These devices have dedicated applications installed.

[0128] server:

[0129] The server is a computing system equipped with a high-performance CPU, memory, and an NVIDIA GPU specialized for running generative models. This is hardware for efficiently running generative models on the server side.

[0130] Generative model:

[0131] Generative models are natural language processing technologies, including OpenAI's GPT series and Google's BERT model, that are used to generate natural dialogue with users.

[0132] Database:

[0133] This is a database system for managing dialogue data, and uses database engines such as MySQL or MongoDB.

[0134] Program processing

[0135] 1. Initial Setup:

[0136] When a user uses the system for the first time, they launch the app installed on their device and enter personal information such as their name, age, and health information. This information is sent from the device to the server, which then stores the received information in a database.

[0137] 2. Start a conversation:

[0138] The user presses the "Start Conversation" button on the device app. This request is sent from the device to the server, and the server uses the generative model to generate an initial greeting message. The generated greeting message is then sent to the device, where the panda character displays and plays the greeting message.

[0139] 3. Collection and analysis of everyday conversations:

[0140] As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server (using the Google Cloud Speech-to-Text API). The server analyzes the received text data and generates the next question or response. The generated new conversation data is sent to the device and displayed and played back by the panda character.

[0141] 4. Health monitoring:

[0142] The server continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. Based on this information, the generative model generates appropriate responses and questions. For example, if a user says, "My lower back hurts," the generative model creates a question like, "That's worrying. Where does it hurt?"

[0143] 5. Anomaly detection and notification:

[0144] If the server detects any abnormalities in the user's health, it will use Twilio's SMS API to send a notification to family members or medical institutions. For example, if the user says something like "I have chest pain," it will call an emergency number.

[0145] 6. Report Generation:

[0146] The server periodically generates a report on the user's health status and emails it to designated family members or healthcare providers using SendGrid, including a summary of the conversation, any anomalies detected, and recommended next steps.

[0147] Specific examples

[0148] Scenario: An elderly person complains of back pain.

[0149] 1. Start the conversation:

[0150] Terminal: "Hello! How are you today?" asks the panda character to the user.

[0151] User: "I've been having back pain lately..."

[0152] Terminal: Converts speech into text and sends it to the server.

[0153] 2. Conversation analysis and anomaly detection:

[0154] Server: Analyzes the keyword "my lower back hurts" and uses a generative model (GPT-3) to generate a new dialogue: "That's worrying. Where does it hurt?"

[0155] Server: Sends the generated dialogue to the terminal.

[0156] Device: The new dialogue will be displayed and played by the panda character.

[0157] 3. Notifications and Reporting:

[0158] Server: Analyzes data related to lower back pain and detects abnormalities.

[0159] Server: Send a notification to the family member via Twilio's SMS API saying, "You've been complaining of back pain recently. A more detailed diagnosis may be needed."

[0160] Server: Generates periodic health status reports and sends them to family members using SendGrid.

[0161] This system allows elderly people to enjoy daily interactions with panda characters while simultaneously managing their own health conditions, allowing their families and medical institutions to monitor their health conditions in real time and contributing to improving the quality of life of the elderly.

[0162] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0163] Step 1: Initial Setup

[0164] Input: A user installs the app on their tablet or smartphone and enters personal information such as their name, age, and health information.

[0165] Specific behavior:

[0166] The user launches the app and enters their personal information on the "New Registration" screen.

[0167] The terminal sends the entered information to the server in JSON format.

[0168] Data processing:

[0169] The server parses the received JSON data, converts it into the required format, and saves it in the database.

[0170] Output: Personal information is stored on the server and recorded in a database.

[0171] Step 2: Start a conversation

[0172] Input: The user presses the "Start conversation" button in the device app.

[0173] Specific behavior:

[0174] When the user presses the "Start conversation" button, the device sends an HTTP request to the server.

[0175] The server receives the HTTP request and inputs the initial prompt "greet the user" into the generative model.

[0176] Data Calculation:

[0177] The generative model generates an initial greeting message based on the prompt sentence.

[0178] Output: The generated greeting message is sent back to the device, and the panda character displays and plays "Hello! How are you today?"

[0179] Step 3: Collecting and analyzing everyday conversations

[0180] Input: The user continues to talk to the panda character.

[0181] Specific behavior:

[0182] When a user says, "My back has been hurting lately...", the device uses the Google Cloud Speech-to-Text API to convert the speech to text.

[0183] The terminal transmits the converted text data to the server.

[0184] Data Calculation:

[0185] The server analyzes the text data and detects the keyword "my lower back hurts."

[0186] The server inputs the prompt statement "If the user says that their lower back hurts, generate a response" into the generative model.

[0187] Output: The generative model generates a response saying, "That's worrying. Where does it hurt?" and sends it to the device.

[0188] The panda character displays new dialogue and plays it aloud: "That's worrying. Where does it hurt?"

[0189] Step 4: Monitor your health

[0190] Input: Continuous conversation data between the user and the panda character

[0191] Specific behavior:

[0192] The server analyzes all received interaction data in real time.

[0193] To detect specific keywords or signs of anomalies, the server inputs the prompt statement "Generate a new question containing specific keywords" into the generative model.

[0194] Data Calculation:

[0195] The generative model generates appropriate responses and questions to prepare for the next dialogue.

[0196] Output: The appropriate questions and responses are generated and sent to the device.

[0197] A panda character presents the generated questions and responses to the user.

[0198] Step 5: Anomaly detection and notification

[0199] Input: Anomaly information based on conversation data analyzed by the server

[0200] Specific behavior:

[0201] If the server detects any abnormalities in health status, it will use Twilio's SMS API to send notifications to family members or medical institutions.

[0202] For example, if a serious abnormality such as "chest pain" is detected, the server will make an emergency call.

[0203] Data Calculation:

[0204] Generate a notification message containing the anomaly information.

[0205] Output: A notification message is sent to family members and healthcare providers.

[0206] Step 6: Generate reports

[0207] Input: Interaction data periodically collected by the server

[0208] Specific behavior:

[0209] The server aggregates past interaction data and generates periodic health status reports.

[0210] The report includes a summary of the conversation, any anomalies detected, and recommended next steps.

[0211] Data Calculation:

[0212] Generate scheduled reports and convert them into email format.

[0213] Output: The report is sent to the designated family member or healthcare provider using SendGrid.

[0214] (Application example 1)

[0215] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0216] Since there is no system that simultaneously supports the health management and meal selection of the elderly, there is a need for a system that can grasp the health status of the elderly and suggest and order appropriate meals. The lack of such a system has led to issues such as the deterioration of the health status of the elderly and the inconvenience of meal selection.

[0217] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0218] In this invention, the server includes a means for proposing an appropriate meal menu based on the dialogue data and the user's health condition, a means for analyzing the dialogue data and evaluating the user's health condition, and a means for notifying the user and accepting an order if an abnormality in the user's health condition or a meal menu is detected. This enables meal suggestions that take the user's health condition into consideration and easy ordering.

[0219] "Devices used by the elderly" refers to electronic devices operated by the elderly, such as smartphones and tablets.

[0220] A "generative model" is an artificial intelligence technology that generates natural dialogue with users and healthy meal menus.

[0221] The "server" is a computer system that executes the generative model and manages and analyzes user interaction data and health data.

[0222] "User dialogue data" refers to data that records and manages the content of conversations with users.

[0223] The "means for proposing an appropriate meal menu based on health condition" is a means having a function of analyzing the user's health condition and presenting a meal menu suitable for that condition.

[0224] A "health status management database" is a collection of data that stores a user's health information and interaction history and can be accessed as needed.

[0225] The "means for presenting natural dialogue" is a function for displaying natural and easy-to-understand dialogue to the user.

[0226] The "means for proposing healthy meal menus" is a function for proposing healthy meal menus based on the user's health condition and dialogue data.

[0227] The "means for analyzing dialogue data and evaluating health status" is a function for analyzing dialogue data with the user and evaluating the health status.

[0228] "Means for notifying and accepting orders when abnormalities in health status or meal menus are detected" is a function that notifies and accepts orders when abnormalities in the user's health status occur or when selecting or ordering a meal menu.

[0229] This invention is an interactive system that supports elderly people in managing their health and choosing meals simultaneously. The system includes a terminal operated by the elderly, a server that executes generative models, and a database that manages dialogue data and health status.

[0230] 1. System Components

[0231] Devices used by the elderly

[0232] The devices are electronic devices operated by elderly people, such as smartphones and tablets. Through these devices, users can interact with a panda character and receive health assessments and meal menu suggestions.

[0233] Generative Model

[0234] A generative model is an AI technology that generates natural dialogue with users and healthy meal menus. Specifically, it detects keywords from user dialogue data and generates appropriate responses and meal menus based on them.

[0235] server

[0236] The server is a computer system that executes the generative model and manages and analyzes user interaction data and health data. For example, the server analyzes the user's health status and notifies the user if an abnormality is detected.

[0237] Database

[0238] The database is a collection of data that stores user interaction history and health information and can be accessed as needed, enabling continuous health monitoring.

[0239] 2. System Operation

[0240] Start a conversation

[0241] When the user presses the "Start Conversation" button on the device, the device sends this request to the server, which uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[0242] Collection and analysis of everyday conversations

[0243] As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The device then presents the generated new conversation data to the user by displaying and playing it back on the panda character.

[0244] Health monitoring

[0245] The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities. For example, if a user says, "I've been having stomach problems lately," the server uses a generative model to generate detailed questions and present them to the user.

[0246] Anomaly detection and notification

[0247] If the server detects any abnormalities in health, it will send the information to family members or medical institutions in real time via a notification function. For example, if someone says "I have chest pain," a notification will be sent immediately to family members or emergency medical institutions, and an emergency call can even be made.

[0248] Meal menu suggestions and ordering

[0249] Based on the dialogue data generated based on the user's health status, the server proposes a meal menu suitable for the user. The user can select from the proposed menu and place an order through the terminal. The order details are sent to the server, which then notifies the meal service provider.

[0250] 3. Specific Examples

[0251] Scenario: A user complains of stomach upset.

[0252] 1. Start a conversation

[0253] Terminal: "Hello! How are you feeling today?" asks the panda character to the user.

[0254] User: "I've been having stomach problems lately."

[0255] Terminal: Converts the user's speech into text and sends it to the server.

[0256] 2. Conversation analysis and health assessment

[0257] Server: Analyzes the keyword "I've been having stomach problems lately," and based on this, the generative model generates a new dialogue: "That's worrying. What's the best thing to eat to ease your stomach?"

[0258] Server: Sends the generated dialogue to the terminal.

[0259] Device: A panda character will display and play a voice saying, "That's worrying. What is the best food to eat for your stomach?"

[0260] 3. Meal menu suggestions

[0261] Server: Based on the user's health condition, the server suggests meal menus such as "healthy soup with less carbonated drinks" or "nutritiously balanced porridge with plenty of vegetables."

[0262] Terminal: A panda character displays the suggested menu.

[0263] 4. Order Confirmation

[0264] User: Selects from the proposed menu and places an order through the terminal.

[0265] Server: Notifies the food service provider of the order.

[0266] Server: Sends a confirmation message to the terminal saying, "Your order has been confirmed. Delivery will be in 30 minutes."

[0267] Prompt Sentence Examples

[0268] You: I've been having stomach problems lately.

[0269] Panda: That's worrying. What's the easiest thing to eat for your stomach?

[0270] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0271] Step 1:

[0272] The user presses the "Start conversation" button on the device.

[0273] Input: User presses a button

[0274] Output: Sending a conversation initiation request to the server

[0275] Specific operation: The device sends a conversation start request to the server. The server receives this request, uses the generative model to generate an initial greeting message, and sends it to the device.

[0276] Step 2:

[0277] The device will display and play a voice message to the user through the panda character as a greeting message.

[0278] Input: Initial greeting message from the server

[0279] Output: Present natural dialogue to the user

[0280] Specific operation: The device receives a greeting message from the server and displays it as a panda character, playing it aloud.

[0281] Step 3:

[0282] The user responds to the panda character.

[0283] Input: User's voice response

[0284] Output: Audio data

[0285] Specific operation: The device receives the user's voice response and converts the voice into text.

[0286] Step 4:

[0287] The terminal transmits the user's text data to the server.

[0288] Input: Text data converted from audio

[0289] Output: Sending text data to the server

[0290] Specific operation: The device sends the text data converted from the voice to the server.

[0291] Step 5:

[0292] The server analyzes the text data and generates the next question or response.

[0293] Input: User's text data

[0294] Output: The next question or response generated

[0295] Specific operation: The server analyzes the received text data using a generative model and generates the next appropriate question or response.

[0296] Step 6:

[0297] The server transmits the generated dialogue to the terminal.

[0298] Input: Generated dialogue

[0299] Output: Send dialogue to terminal

[0300] Specific operation: The server sends the generated dialogue to the terminal.

[0301] Step 7:

[0302] The device displays and plays aloud the dialogue generated through the panda character to the user.

[0303] Input: Generated dialogue from the server

[0304] Output: Presents dialogue to the user

[0305] Specific operation: The terminal displays and plays aloud the dialogue received from the server using a panda character.

[0306] Step 8:

[0307] The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities.

[0308] Input: Interaction data

[0309] Output: Health status assessment results, abnormality detection results

[0310] Specific operation: The server uses the generative model to analyze the dialogue data and detect specific keywords and signs of anomalies.

[0311] Step 9:

[0312] The server will suggest an appropriate meal menu based on the health assessment results.

[0313] Input: Health status assessment results

[0314] Output: Suggested meal menu

[0315] Specific operation: The server uses the health status assessment results to suggest health-conscious meal menus through a generative model.

[0316] Step 10:

[0317] The terminal displays the suggested meal menu to the user, who places an order.

[0318] Input: Suggested meal menu, user selection

[0319] Output: Order data

[0320] Specific operation: The terminal displays the proposed meal menu and the user selects. The user's order is sent from the terminal to the server.

[0321] Step 11:

[0322] The server transmits the order data to the food service provider.

[0323] Input: Order data

[0324] Output: Notification to food service provider

[0325] Specific operation: The server notifies the meal service provider of the user's order data, confirms the order, and sends a confirmation message to the user's terminal.

[0326] Through the above processing steps, the user can order an appropriate meal menu while monitoring their health condition.

[0327] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0328] MODE FOR CARRYING OUT THE INVENTION

[0329] The present invention combines an emotion engine with an interactive medical AI service system for the elderly. Specific embodiments of the system are described below.

[0330] Components

[0331] 1. Devices: Electronic devices such as tablets and smartphones used by the elderly that provide an interface that allows users to converse with panda characters on a daily basis.

[0332] 2. Server: A computer system that executes the generative model, manages and analyzes the dialogue data with the user, and transmits the generated dialogue and health assessment data to the terminal.

[0333] 3. Generative Model: This is an artificial intelligence technique used to generate natural dialogue with the user, detect specific keywords, and assess health status based on them.

[0334] 4. Database: A collection of data that stores user interaction data and can be accessed as needed.

[0335] 5. Emotion engine: This engine recognizes emotions from the user's voice and text data and adjusts the dialogue with the user based on the recognized emotional data.

[0336] System Operation

[0337] 1. Initial Setup: When a user uses the service for the first time, they install the app on their device and enter their personal information (name, age, health information, etc.). This information is sent to the server and stored in a database.

[0338] 2. Start conversation: The user presses the "Start conversation" button on the device. The device sends this request to the server, which uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[0339] 3. Collection and analysis of everyday conversations: As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The device presents the generated new conversation data to the user by displaying and playing it back on the panda character.

[0340] 4. Health monitoring: The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities. Based on this, the generative model generates appropriate responses and questions. For example, if the user says, "My lower back hurts," the server instructs the generative model to ask more detailed questions about the extent and location of the pain.

[0341] 5. Emotion recognition and adjustment: The device sends the user's voice and facial expression data to the emotion engine, which then recognizes the user's emotions. For example, if the user has an anxious expression or voice, the emotion engine sends that information to the server.

[0342] 6. Emotion-based dialogue adjustment: The server analyzes the emotion data received from the emotion engine and generates appropriate dialogue based on the emotion. For example, if the user is anxious, the panda character will generate a dialogue such as, "Is there anything you're worried about? Please tell me."

[0343] 7. Abnormality detection and notification: If the server detects an abnormality in health status, the notification function will send the information to family members and medical institutions in real time. For example, if someone says, "I have chest pain," a notification will be sent immediately to family members and emergency medical institutions, and if necessary, an emergency call will be made.

[0344] 8. Report Generation: The server periodically generates a report summarizing the user's health status and emails it to designated family members or healthcare providers. The report includes a summary of the conversation, any abnormalities detected, and recommended next steps.

[0345] Specific examples

[0346] Scenario: An elderly person complains of back pain and becomes emotionally unstable.

[0347] 1. Start a conversation

[0348] Terminal: "Hello! How are you today?" asks the panda character to the user.

[0349] User: "I've been having back pain lately..."

[0350] Terminal: Converts the user's speech into text and sends it to the server.

[0351] 2. Conversation Analysis and Anomaly Detection

[0352] Server: Analyzes the keyword "my lower back hurts," and based on this, the generative model generates a new dialogue: "That's worrying. Where does it hurt?"

[0353] Server: Sends the generated dialogue to the terminal.

[0354] Device: Panda will say and listen, "That's worrying. Where does it hurt?"

[0355] 3. Emotional Recognition

[0356] Device: The user answers in an anxious voice, "My whole lower back hurts."

[0357] Device: Sends voice data to the emotion engine, which detects anxiety.

[0358] Server: Receives emotion data from the emotion engine and generates dialogue based on it.

[0359] 4. Adjusting dialogue based on emotions

[0360] Server: Based on the information from the emotion engine, it generates the dialogue, "You're okay. Take a rest and don't push yourself too hard. Is there anything I can help you with?"

[0361] Server: Sends the generated dialogue to the terminal.

[0362] Device: Panda will say, "It's okay. Take care and rest. Is there anything I can help you with?"

[0363] This invention allows elderly people to enjoy daily conversations with panda characters while simultaneously managing their health. It also responds sensitively to the user's emotions and provides appropriate dialogue, providing a greater sense of security. Family members and medical institutions can monitor the elderly's health status in real time, contributing to improving the quality of life for the elderly.

[0364] The processing flow will be explained below.

[0365] Step 1:

[0366] Device: Display the app installation screen.

[0367] User: Installs and launches the app.

[0368] Step 2:

[0369] Device: Display the initial setup screen (name, age, health information, etc.).

[0370] User: Enter the required information and press the send button.

[0371] Terminal: Sends the entered information to the server.

[0372] Server: Stores the received information in a database and sends a response to the terminal indicating that initial setup is complete.

[0373] Terminal: Notify the user that the initial setup is complete.

[0374] Step 3:

[0375] Device: Display a "Start Conversation" button on the home screen.

[0376] User: Press the start conversation button.

[0377] Terminal: Sends a request to start a conversation to the server.

[0378] Step 4:

[0379] Server: Initiates the conversation session and generates the initial greeting message.

[0380] Server: Sends a greeting message to the terminal.

[0381] On your device: A panda character will display and / or play a greeting message (e.g., "Hello! How was your day today?").

[0382] Step 5:

[0383] User: Answer the panda character's questions (e.g., "It was nice weather today so I went for a walk").

[0384] Terminal: Converts the user's speech into text and sends it to the server.

[0385] Step 6:

[0386] Server: Analyzes the received text data and generates the next question or response.

[0387] Server: Sends the generated conversation data to the terminal.

[0388] On your device: The panda character will display and play your new message (e.g., "That's lovely! Which way did you walk?").

[0389] Step 7:

[0390] Server: Continuously analyzes conversation data and detects keywords and anomalies.

[0391] Server: Generates new responses and questions based on the detected information.

[0392] Device: View and play new questions and responses from the panda character.

[0393] Step 8:

[0394] User: Provides health information as the conversation continues (e.g., "I've been having a lot of back pain lately...").

[0395] Terminal: Converts the user's speech into text and sends it to the server.

[0396] Step 9:

[0397] Server: Analyzes keywords that indicate abnormal health conditions, such as "lower back pain," and detects abnormalities.

[0398] Server: If an abnormality is detected, it generates a notification to notify family members or medical institutions.

[0399] Server: Send notifications to family members or medical institutions (e.g., "You've recently been complaining of back pain. You may need further evaluation.").

[0400] Step 10:

[0401] Strategic terminal: Sends the user's voice and facial expression data to the emotion engine.

[0402] Emotion engine: Recognizes user emotions and detects emotions such as anxiety.

[0403] Emotion Engine: Sends detected emotion data to the server.

[0404] Step 11:

[0405] Server: Analyzes the received emotion data and generates appropriate dialogue based on the emotion.

[0406] Server: Sends the generated dialogue to the terminal.

[0407] On your device: The panda character will display and / or play a sensitive message (e.g., "It's okay. Take care and rest. Is there anything I can help you with?").

[0408] Step 12:

[0409] Server: Periodically aggregates conversation data and generates a health status report for the elderly.

[0410] Server: Sends the generated report to the family or medical institution.

[0411] Step 13:

[0412] Device: Continue everyday conversations with panda characters.

[0413] User: Continue to provide information about health status through interactions with the panda character.

[0414] Example 2

[0415] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0416] Elderly people need to be sensitive to changes in their health and emotions in their daily lives. However, it is often difficult for them to visit medical institutions or communicate with specialists on a daily basis. For this reason, there is a need for a system that allows elderly people to easily manage their health at home and receive appropriate support in response to emotional changes.

[0417] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0418] In this invention, the server includes means for analyzing the dialogue data to evaluate the health condition, means for notifying when an abnormality in the health condition is detected, and means for adjusting the dialogue based on the emotion data, thereby enabling the user to manage their health condition while at home and receive appropriate dialogue in response to changes in their emotions.

[0419] "Communication devices" are electronic devices such as tablets and smartphones used by elderly people, and are devices that provide an interface for users to interact with each other on a daily basis.

[0420] An "information processing device" is a computer system that executes generative models and manages and analyzes user interaction data.

[0421] A "storage device" is a collection of data that stores user interaction data and can be accessed as needed.

[0422] A "generative model" is an artificial intelligence technique used to generate natural-sounding dialogue with a user.

[0423] An "emotion recognition engine" is an engine that recognizes emotions from the user's voice and text data and adjusts the dialogue based on that information.

[0424] "Dialogue data" refers to data such as messages and voice recordings exchanged between a user and a communication device.

[0425] "Health status assessment" is the process of analyzing dialogue data and determining the user's health status.

[0426] "Notification means" is a function that sends information to family members or medical institutions when an abnormality in health condition is detected.

[0427] "Emotion data" is data that indicates the emotional state of the user analyzed by the emotion recognition engine.

[0428] "Dialogue adjustment" is the process of modifying the dialogue content and providing an appropriate response to the user based on emotion data.

[0429] This invention relates to an interactive medical support system for the elderly, and in particular to a system that utilizes a generative AI model with emotion recognition capabilities. The specific operation and usage of the system are described below.

[0430] System Overview

[0431] The system includes a communication device (e.g., a tablet or smartphone) used by the elderly, an information processing device that executes a generative model, and a storage device that manages interaction data with the user. The system includes the following means:

[0432] A means for communication devices to present natural dialogue to users

[0433] The means by which the generative model generates interactions with the user

[0434] A means for an information processing device to analyze the dialogue data and evaluate the health condition

[0435] A means for notifying when an information processing device detects an abnormality in health status

[0436] A means for communication devices to send user voice and facial expression data to an emotion recognition engine

[0437] The emotion recognition engine analyzes the user's emotions and transmits the results to the information processing device.

[0438] A means for an information processing device to adjust a dialogue based on emotion data

[0439] Initial Setup

[0440] When a user uses the system for the first time, they install a dedicated app on their communication device and enter personal information such as their name, age, and health information, which is then sent to the server and stored in a database.

[0441] Start a conversation

[0442] When the user presses the "Start Conversation" button on the communication device, the communication device sends this request to the server. The server uses the generative model to generate an initial greeting message and sends it to the communication device. The communication device then displays and plays the greeting message aloud using a panda character.

[0443] Collection and analysis of everyday conversations

[0444] As the user engages in everyday conversation with the panda character, the communication device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The generated dialogue is sent to the communication device, where it is displayed and played aloud by the panda character.

[0445] Health monitoring

[0446] The server continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. For example, if a user says, "My lower back hurts," the server instructs the generative model to ask more detailed questions about the extent and location of the pain.

[0447] Emotional awareness and regulation

[0448] The communication device sends the user's voice and facial expression data to an emotion recognition engine, which then analyzes the user's emotions. For example, if the user has an anxious expression or voice, that information is sent to the server. The server analyzes the emotional data, generates appropriate dialogue based on the emotion, and sends it to the communication device.

[0449] Health Assessment and Notification

[0450] If the server detects any abnormalities in a person's health, it will notify family members or medical institutions in real time. For example, if someone says, "I have chest pain," a notification will be sent immediately, and an emergency call will be made if necessary.

[0451] Report Generation

[0452] The server periodically generates a report summarizing the user's health status and emails it to designated family members or healthcare providers, including a summary of the conversation, any abnormalities detected, and recommended next steps.

[0453] Specific examples

[0454] Scenario: An elderly person complains of back pain and becomes emotionally unstable.

[0455] 1. Start a conversation

[0456] Terminal: "Hello! How are you today?" asks the panda character to the user.

[0457] User: "I've been having back pain lately..."

[0458] Terminal: Converts the user's speech into text and sends it to the server.

[0459] 2. Conversation Analysis and Anomaly Detection

[0460] Server: Analyzes the keyword "my lower back hurts," and based on this, the generative model generates a new dialogue: "That's worrying. Where does it hurt?"

[0461] Server: Sends the generated dialogue to the terminal.

[0462] Device: A panda character will display and play a voice message saying, "That's worrying. Where does it hurt?"

[0463] 3. Emotional Recognition

[0464] User: In an anxious voice, replies, "My lower back hurts all over."

[0465] Device: Sends voice data to the emotion engine to detect anxiety.

[0466] Server: Receives the emotion data and generates a dialogue such as, "You're okay. Take care and rest. Is there anything I can help you with?"

[0467] 4. Coordinating dialogue

[0468] Server: Sends the generated dialogue to the terminal.

[0469] Device: A panda character will display and play a voice saying, "It's okay. Please take it easy and rest. Is there anything I can help you with?"

[0470] Prompt Sentence Examples

[0471] "The user complains of back pain and appears anxious. Based on this, please generate a dialogue that will reassure the user."

[0472] This invention allows elderly people to manage their health through the panda character in their daily lives and provides appropriate dialogue, allowing them to spend their days with peace of mind.Furthermore, family members and medical institutions can monitor their health status in real time, contributing to improving the quality of life for elderly people.

[0473] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0474] Step 1: Initial Setup

[0475] User: Installs the app on the device and enters personal information (name, age, health information, etc.) through the app's input form.

[0476] Terminal: Receives input personal information and converts it into JSON format to send to the server. The input is personal information, and the output is the converted JSON data.

[0477] Server: Analyzes the received JSON data and saves the user information in a database. This allows basic user information to be managed on the server side.

[0478] Step 2: Start a conversation

[0479] User: Presses the "Start conversation" button on the device. Input is the action of clicking the "Start conversation" button.

[0480] Terminal: Generates a conversation initiation request and sends it to the server. The output is the generated request data.

[0481] Server: Receives the request and generates the initial greeting message using the generative model. The input is the request and the output is the greeting message.

[0482] Server: Sends the generated greeting message to the terminal. The output is the generated message data.

[0483] Terminal: The panda character displays and plays audibly the received greeting message. The input is the greeting message, and the output is the display and audio playback for the user.

[0484] Step 3: Collecting and analyzing everyday conversations

[0485] User: Have everyday conversations with a panda character. Input is via user voice and text.

[0486] Terminal: Converts user speech from voice to text. This conversion is done by speech recognition software. The input is the user's voice, and the output is text data.

[0487] Terminal: Transmits converted text data to the server. The output is text data.

[0488] Server: Analyzes the received text data and generates the next question or response. This analysis is performed using a natural language processing model. The input is text data, and the output is a new dialogue.

[0489] Server: Sends the generated dialogue to the terminal. The output is dialogue data.

[0490] Terminal: The panda character displays and plays audibly the received dialogue. The input is the dialogue, and the output is a display and audible playback for the user.

[0491] Step 4: Monitor your health

[0492] Server: Continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. A keyword extraction algorithm is used for analysis. The input is dialogue data, and the output is health assessment data.

[0493] Server: When an anomaly is detected, it uses a generative model to generate appropriate questions and responses. The input is the anomaly detection result, and the output is the new questions and responses.

[0494] Server: Sends the generated dialogue to the terminal. The output is dialogue data.

[0495] Terminal: The panda character displays and plays audibly the received dialogue. The input is the dialogue, and the output is a display and audible playback for the user.

[0496] Step 5: Recognize and regulate your emotions

[0497] Terminal: Sends the user's voice and facial expression data to the emotion recognition engine. This transmission is done in real time. The input is voice and facial expression data, and the output is the transmitted data.

[0498] Emotion recognition engine: Analyzes the user's emotions and sends the results to the server. The input is voice and facial expression data, and the output is emotional data.

[0499] Server: Receives emotional data and adjusts the dialogue based on the analysis results. The input is emotional data, and the output is the adjusted dialogue.

[0500] Server: Sends the adjusted dialogue to the terminal. The output is dialogue data.

[0501] Terminal: The panda character displays and plays audibly the received dialogue. The input is the dialogue, and the output is a display and audible playback for the user.

[0502] Step 6: Health Assessment and Notification

[0503] Server: When an abnormality in health status is detected, the notification function is activated. This function includes push notifications and email notifications. The input is the abnormality detection result, and the output is the notification data.

[0504] Server: Notifies family and medical institutions of health status in real time. Makes emergency calls if necessary. Input is notification data, output is notification and alert.

[0505] Step 7: Generate reports

[0506] Server: Periodically generates a report summarizing the user's health status. This report includes a summary of the conversation, any abnormalities detected, and recommended next steps. The input is the conversation data and health assessment data, and the output is the report.

[0507] Server: Sends the generated report to the designated family or medical institution by email. The input is the report data, and the output is the sent report.

[0508] (Application example 2)

[0509] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0510] Health management, especially for the elderly, is an important issue in modern society. Elderly people require daily health monitoring, and prompt action is required when abnormalities occur. However, conventional systems do not provide natural dialogue and are difficult to respond to, taking into account the user's emotions. Therefore, there is a need for a system that combines natural dialogue and emotion recognition that can be used comfortably by the elderly.

[0511] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for analyzing dialogue data and evaluating health status; means for the server to notify when it detects an abnormality in health status; means for the terminal and the server to recognize emotions and adjust the dialogue based on the emotions; means for the terminal to check the health status of residents of the welfare facility through everyday conversation; and means for the server to periodically generate health status reports and provide them to a specified contact. This enables elderly people to manage their health with a sense of security through more natural dialogue.

[0512] The term "elderly" refers to people over a certain age, and in healthcare and welfare in particular, generally refers to people aged 65 and over.

[0513] A "terminal" is an electronic device used by a user, which inputs and outputs data via an interface.

[0514] A "generative model" is an algorithm or program that uses artificial intelligence technology to generate natural-sounding dialogue.

[0515] A "server" is a computer system that manages data and programs and provides services to terminals via a network.

[0516] A "database" is a collection of data that stores user interaction data and health information and can be accessed as needed.

[0517] "Emotions" refer to psychological states such as joy, sadness, and anxiety that humans experience, and can be read from voice and facial expressions.

[0518] A "dialogue" is a written communication between a user and a terminal.

[0519] "Abnormal" refers to a state that deviates from normal health and requires immediate action.

[0520] "Notification" is the act of communicating specific information to designated recipients in real time.

[0521] A "welfare facility" refers to a dedicated facility where elderly people, people with disabilities, etc. can live while receiving support and care.

[0522] MODE FOR CARRYING OUT THE INVENTION

[0523] This invention describes an embodiment of an interactive medical AI service system for elderly people specifically implemented in a welfare facility. The components and operation of the system are described in detail below.

[0524] System Configuration

[0525] 1. Devices: Electronic devices such as tablets and smartphones used by the elderly. Devices provide an interface with users and have the ability to display and play natural dialogue.

[0526] 2. Generative model: An artificial intelligence algorithm that uses natural language processing to generate dialogue with the user. Specifically, it incorporates Spacy and BERT.

[0527] 3. Server: A computer system that manages the database and generative model and sends dialogues and notifications to devices via a network. The server is built using Python or Flask and is responsible for data analysis and dialogue generation.

[0528] 4. Emotion Engine: An algorithm that recognizes emotions from the user's voice and text data and adjusts the dialogue accordingly. The EmotionEngine class is an example of this.

[0529] 5. Database: A system for storing conversation data and health information. MySQL, PostgreSQL, etc. are used.

[0530] Operation overview

[0531] 1. Initial setup: The server stores the user's basic information (name, age, health information, etc.) in a database. During the initial setup, the user installs the app on their device and enters the required information.

[0532] 2. Start of conversation: When the user presses the "Start conversation" button on the device, the server uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[0533] 3. Collection and analysis of everyday conversations: As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. For example, if the user says, "I've been having a pain in my lower back lately...", the generative model will generate a response such as, "That's worrying. Where does it hurt?"

[0534] 4. Health status monitoring: The server continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. Based on this, the generative model generates appropriate responses and questions. For example, if the keyword "chest pain" is detected, this information will be promptly notified to family members or medical institutions.

[0535] 5. Emotion recognition and adjustment: The device sends the user's voice and facial expression data to the emotion engine, which then recognizes the user's emotions. For example, if the user's facial expression or voice looks anxious, the emotion engine detects the anxiety and sends this information to the server. The server analyzes the emotion data and generates a dialogue based on the emotion.

[0536] 6. Periodic Report Generation: The server periodically generates a report summarizing the user's health status and provides it to designated contacts (family members or healthcare providers), including a summary of the conversation, any abnormalities detected, and recommended next steps.

[0537] Specific examples

[0538] Scenario: An elderly person complains of lower back pain

[0539] 1. Start a conversation

[0540] User: "I've been having back pain lately..."

[0541] System: "That's concerning. Where does it hurt?"

[0542] 2. Emotional Recognition

[0543] User: "My lower back hurts all over."

[0544] System: The emotion engine detects the user's anxiety and generates a response such as, "You're OK. Take care and get some rest. Is there anything I can help you with?"

[0545] With these actions, elderly people can manage their health with peace of mind through everyday conversation. In addition, if any abnormalities occur, family members and medical institutions will be notified promptly, allowing them to live their lives with peace of mind.

[0546] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0547] Step 1: Initial Setup

[0548] Users install a medical app for the elderly on their device and enter their personal information (name, age, health information, etc.).

[0549] Input: User personal information

[0550] Processing: The device sends this information to the server, which stores it in a database.

[0551] Output: User information stored in the database

[0552] Step 2: Start a conversation

[0553] The user presses the "Start conversation" button on the device.

[0554] Input: User operation (pressing the "Start conversation" button)

[0555] Processing: The server uses the generative model to generate an initial greeting message and sends it to the terminal.

[0556] Output: A greeting message displayed on the terminal (natural dialogue)

[0557] Step 3: Collecting and analyzing everyday conversations

[0558] Users can converse with a panda character.

[0559] Input: User's voice data

[0560] Processing: The device converts the voice data into text and sends it to the server. The server analyzes the received text, and the generative model generates the next question or response. The server then sends the generated dialogue back to the device.

[0561] Output: The new dialogue displayed on the terminal.

[0562] Step 4: Monitor your health

[0563] The server analyzes the conversation data and detects specific keywords and signs of abnormalities.

[0564] Input: User's text data

[0565] Processing: The server detects specific keywords (e.g., "my lower back hurts") and evaluates the health status. If an abnormality is detected, it instructs the generative model on the next course of action.

[0566] Output: The next dialogue generated, and the anomaly detection results.

[0567] Step 5: Recognize and regulate your emotions

[0568] The device sends the user's voice and facial expression data to the emotion engine.

[0569] Input: User's voice and facial expression data

[0570] Processing: The emotion engine analyzes these data to recognize emotions and sends the results to the server. The server analyzes the emotion data, and the generative model generates appropriate dialogue based on the emotion.

[0571] Output: Generated dialogue (content corresponding to emotions)

[0572] Step 6: Anomaly detection and notification

[0573] If the server detects any abnormalities in health, it will use the notification function to send information to family members or medical institutions in real time.

[0574] Input: Anomaly detection results

[0575] Processing: The server generates a notification and sends it to the specified contacts, including calling emergency services if necessary.

[0576] Output: Notification sent to family and healthcare providers

[0577] Step 7: Generate reports

[0578] The server periodically generates a report summarizing the user's health status and emails it to designated contacts.

[0579] Input: Dialogue data and emotion data

[0580] Processing: The server compiles and analyzes the data, generates a report, and sends the report to the designated contact.

[0581] Output: Health report sent to family and healthcare providers

[0582] Through the above processing steps, elderly people can manage their health through natural dialogue and emotion recognition, and can live with peace of mind even in welfare facilities.

[0583] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0584] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0585] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0586] [Second embodiment]

[0587] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0588] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0589] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0590] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0591] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0592] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0593] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0594] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0595] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0596] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0597] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0598] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0599] MODE FOR CARRYING OUT THE INVENTION

[0600] The present invention provides an interactive medical AI service system for elderly people. Specific embodiments of the system are described below.

[0601] Components

[0602] 1. Devices: Electronic devices such as tablets and smartphones used by the elderly. They provide an interface that allows users to converse with panda characters on a daily basis.

[0603] 2. Server: A computer system that executes the generative model, manages and analyzes the dialogue data with the user, and transmits the generated dialogue and health assessment data to the terminal.

[0604] 3. Generative Model: This is an artificial intelligence technique used to generate natural dialogue with the user, detect specific keywords, and assess health status based on them.

[0605] 4. Database: A collection of data that stores user interaction data and can be accessed as needed.

[0606] System Operation

[0607] 1. Initial Setup: When a user uses the service for the first time, they install the app on their device and enter their personal information (name, age, health information, etc.). This information is sent to the server and stored in a database.

[0608] 2. Start conversation: The user presses the "Start conversation" button on the device. The device sends this request to the server, which uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[0609] 3. Collection and analysis of everyday conversations: As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The device presents the generated new conversation data to the user by displaying and playing it back on the panda character.

[0610] 4. Health monitoring: The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities. Based on this, the generative model generates appropriate responses and questions. For example, if the user says, "My lower back hurts," the server instructs the generative model to ask more detailed questions about the extent and location of the pain.

[0611] 5. Abnormality detection and notification: If the server detects an abnormality in health status, the notification function will send the information to family members and medical institutions in real time. For example, if someone says, "I have chest pain," a notification will be sent immediately to family members and emergency medical institutions, and if necessary, an emergency call will be made.

[0612] 6. Report Generation: The server periodically generates a report summarizing the user's health status and emails it to designated family members or healthcare providers. The report includes a summary of the conversation, any abnormalities detected, and recommended next steps.

[0613] Specific examples

[0614] Scenario: Conversation with an elderly person complaining of lower back pain

[0615] 1. Start a conversation

[0616] Terminal: "Hello! How are you today?" asks the panda character to the user.

[0617] User: "I've been having back pain lately..."

[0618] Terminal: Converts the user's speech into text and sends it to the server.

[0619] 2. Conversation Analysis and Anomaly Detection

[0620] Server: Analyzes the keyword "my lower back hurts," and based on this, the generative model generates a new dialogue: "That's worrying. Where does it hurt?"

[0621] Server: Sends the generated dialogue to the terminal.

[0622] Device: Panda will say and listen, "That's worrying. Where does it hurt?"

[0623] 3. Notifications and Reporting

[0624] Server: Detects abnormalities based on data related to lower back pain and sends a notification to the family member stating, "You have recently been complaining of lower back pain. A more detailed diagnosis may be necessary."

[0625] Server: Generates periodic health status reports and sends them to family members.

[0626] This invention allows elderly people to enjoy daily conversations with panda characters while simultaneously managing their health.Furthermore, family members and medical institutions can monitor their health status in real time, contributing to improving the quality of life for elderly people.

[0627] The processing flow will be explained below.

[0628] Step 1:

[0629] Device: Display the app installation screen.

[0630] User: Installs and launches the app.

[0631] Step 2:

[0632] Device: Display the initial setup screen (name, age, health information, etc.).

[0633] User: Enter the required information and press the send button.

[0634] Terminal: Sends the entered information to the server.

[0635] Server: Stores the received information in a database and sends a response to the terminal indicating that initial setup is complete.

[0636] Terminal: Notify the user that the initial setup is complete.

[0637] Step 3:

[0638] Device: Display a "Start Conversation" button on the home screen.

[0639] User: Press the start conversation button.

[0640] Terminal: Sends a request to start a conversation to the server.

[0641] Step 4:

[0642] Server: Initiates the conversation session and generates the initial greeting message.

[0643] Server: Sends a greeting message to the terminal.

[0644] On your device: A panda character will display and / or play a greeting message (e.g., "Hello! How was your day today?").

[0645] Step 5:

[0646] User: Answer the panda character's questions (e.g., "It was nice weather today so I went for a walk").

[0647] Terminal: Converts the user's speech into text and sends it to the server.

[0648] Step 6:

[0649] Server: Analyzes the received text data and generates the next question or response.

[0650] Server: Sends the generated conversation data to the terminal.

[0651] On your device: The panda character will display and play your new message (e.g., "That's lovely! Which way did you walk?").

[0652] Step 7:

[0653] Server: Continuously analyzes conversation data and detects keywords and anomalies.

[0654] Server: Generates new responses and questions based on the detected information.

[0655] Device: View and play new questions and responses from the panda character.

[0656] Step 8:

[0657] User: Provides health information as the conversation continues (e.g., "I've been having a lot of back pain lately...").

[0658] Terminal: Converts the user's speech into text and sends it to the server.

[0659] Step 9:

[0660] Server: Analyzes keywords that indicate abnormal health conditions, such as "lower back pain," and detects abnormalities.

[0661] Server: If an abnormality is detected, it generates a notification to notify family members or medical institutions.

[0662] Server: Send notifications to family members or medical institutions (e.g., "You've recently been complaining of back pain. You may need further evaluation.").

[0663] Step 10:

[0664] Server: Periodically aggregates conversation data and generates a health status report for the elderly.

[0665] Server: Sends the generated report to the family or medical institution.

[0666] Step 11:

[0667] Device: Continue everyday conversations with panda characters.

[0668] User: Continue to provide information about health status through interactions with the panda character.

[0669] Example 1

[0670] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0671] It is difficult for elderly people to manage their own health status in their daily lives and detect abnormalities early. It is also difficult for family members and medical institutions to grasp the health status of elderly people in real time. This increases health risks, especially for elderly people living alone. Furthermore, there is currently a lack of systems that assess the health status of elderly people through natural dialogue.

[0672] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0673] In this invention, the server includes a means for analyzing the dialogue data to evaluate the health condition, a means for notifying when an abnormality in the health condition is detected, and a means for continuously monitoring the health condition based on the dialogue data. This allows the user to manage their own health condition through natural dialogue, and enables family members and medical institutions to understand the health condition in real time.

[0674] "Elderly" refers to users who are aging and who are particularly in need of medical and nursing care support.

[0675] An "information terminal" is an electronic device used by a user, and refers to a device that has the shape of a tablet, smartphone, etc.

[0676] A "natural language generation model" is an artificial intelligence technology used to generate natural dialogue with users, and specifically refers to a computer program that performs language processing.

[0677] "Computer system" refers generally to a computer system for executing generative models and analyzing and managing interaction data.

[0678] "Data Collection" refers to a database system that stores user interaction data and makes it accessible as needed.

[0679] "Natural dialogue" refers to text that is generated naturally in interaction with a user, just like human language.

[0680] "Interaction Data" refers to data including voice, text, and other related information relating to a conversation between a user and a system.

[0681] "Health status" refers to the state of the user's physical and psychological health, and specifically includes physical condition, presence or absence of pain, mental state, etc.

[0682] "Abnormal" refers to any symptom or sign that deviates from normal health, especially one that requires medical attention.

[0683] "Notification" refers to the process of communicating information about an abnormality in a user's health condition to family members or medical institutions.

[0684] "Monitoring" refers to the process of continuously observing a user's health status and assessing it based on specific conditions.

[0685] MODE FOR CARRYING OUT THE INVENTION

[0686] The present invention provides an interactive medical AI service system for elderly people. Specific embodiments of the system are described below.

[0687] Hardware and Software Configuration

[0688] Device:

[0689] This refers to tablets and smartphones that are used daily by seniors, including iPads, Android tablets, iPhones, and Android smartphones. These devices have dedicated applications installed.

[0690] server:

[0691] The server is a computing system equipped with a high-performance CPU, memory, and an NVIDIA GPU specialized for running generative models. This is hardware for efficiently running generative models on the server side.

[0692] Generative model:

[0693] Generative models are natural language processing technologies, including OpenAI's GPT series and Google's BERT model, that are used to generate natural dialogue with users.

[0694] Database:

[0695] This is a database system for managing dialogue data, and uses database engines such as MySQL or MongoDB.

[0696] Program processing

[0697] 1. Initial Setup:

[0698] When a user uses the system for the first time, they launch the app installed on their device and enter personal information such as their name, age, and health information. This information is sent from the device to the server, which then stores the received information in a database.

[0699] 2. Start a conversation:

[0700] The user presses the "Start Conversation" button on the device app. This request is sent from the device to the server, and the server uses the generative model to generate an initial greeting message. The generated greeting message is then sent to the device, where the panda character displays and plays the greeting message.

[0701] 3. Collection and analysis of everyday conversations:

[0702] As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server (using the Google Cloud Speech-to-Text API). The server analyzes the received text data and generates the next question or response. The generated new conversation data is sent to the device and displayed and played back by the panda character.

[0703] 4. Health monitoring:

[0704] The server continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. Based on this information, the generative model generates appropriate responses and questions. For example, if a user says, "My lower back hurts," the generative model creates a question like, "That's worrying. Where does it hurt?"

[0705] 5. Anomaly detection and notification:

[0706] If the server detects any abnormalities in the user's health, it will use Twilio's SMS API to send a notification to family members or medical institutions. For example, if the user says something like "I have chest pain," it will call an emergency number.

[0707] 6. Report Generation:

[0708] The server periodically generates a report on the user's health status and emails it to designated family members or healthcare providers using SendGrid, including a summary of the conversation, any anomalies detected, and recommended next steps.

[0709] Specific examples

[0710] Scenario: An elderly person complains of back pain.

[0711] 1. Start the conversation:

[0712] Terminal: "Hello! How are you today?" asks the panda character to the user.

[0713] User: "I've been having back pain lately..."

[0714] Terminal: Converts speech into text and sends it to the server.

[0715] 2. Conversation analysis and anomaly detection:

[0716] Server: Analyzes the keyword "my lower back hurts" and uses a generative model (GPT-3) to generate a new dialogue: "That's worrying. Where does it hurt?"

[0717] Server: Sends the generated dialogue to the terminal.

[0718] Device: The new dialogue will be displayed and played by the panda character.

[0719] 3. Notifications and Reporting:

[0720] Server: Analyzes data related to lower back pain and detects abnormalities.

[0721] Server: Send a notification to the family member via Twilio's SMS API saying, "You've been complaining of back pain recently. A more detailed diagnosis may be needed."

[0722] Server: Generates periodic health status reports and sends them to family members using SendGrid.

[0723] This system allows elderly people to enjoy daily interactions with panda characters while simultaneously managing their own health conditions, allowing their families and medical institutions to monitor their health conditions in real time and contributing to improving the quality of life of the elderly.

[0724] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0725] Step 1: Initial Setup

[0726] Input: A user installs the app on their tablet or smartphone and enters personal information such as their name, age, and health information.

[0727] Specific behavior:

[0728] The user launches the app and enters their personal information on the "New Registration" screen.

[0729] The terminal sends the entered information to the server in JSON format.

[0730] Data processing:

[0731] The server parses the received JSON data, converts it into the required format, and saves it in the database.

[0732] Output: Personal information is stored on the server and recorded in a database.

[0733] Step 2: Start a conversation

[0734] Input: The user presses the "Start conversation" button in the device app.

[0735] Specific behavior:

[0736] When the user presses the "Start conversation" button, the device sends an HTTP request to the server.

[0737] The server receives the HTTP request and inputs the initial prompt "greet the user" into the generative model.

[0738] Data Calculation:

[0739] The generative model generates an initial greeting message based on the prompt sentence.

[0740] Output: The generated greeting message is sent back to the device, and the panda character displays and plays "Hello! How are you today?"

[0741] Step 3: Collecting and analyzing everyday conversations

[0742] Input: The user continues to talk to the panda character.

[0743] Specific behavior:

[0744] When a user says, "My back has been hurting lately...", the device uses the Google Cloud Speech-to-Text API to convert the speech to text.

[0745] The terminal transmits the converted text data to the server.

[0746] Data Calculation:

[0747] The server analyzes the text data and detects the keyword "my lower back hurts."

[0748] The server inputs the prompt statement "If the user says that their lower back hurts, generate a response" into the generative model.

[0749] Output: The generative model generates a response saying, "That's worrying. Where does it hurt?" and sends it to the device.

[0750] The panda character displays new dialogue and plays it aloud: "That's worrying. Where does it hurt?"

[0751] Step 4: Monitor your health

[0752] Input: Continuous conversation data between the user and the panda character

[0753] Specific behavior:

[0754] The server analyzes all received interaction data in real time.

[0755] To detect specific keywords or signs of anomalies, the server inputs the prompt statement "Generate a new question containing specific keywords" into the generative model.

[0756] Data Calculation:

[0757] The generative model generates appropriate responses and questions to prepare for the next dialogue.

[0758] Output: The appropriate questions and responses are generated and sent to the device.

[0759] A panda character presents the generated questions and responses to the user.

[0760] Step 5: Anomaly detection and notification

[0761] Input: Anomaly information based on conversation data analyzed by the server

[0762] Specific behavior:

[0763] If the server detects any abnormalities in health status, it will use Twilio's SMS API to send notifications to family members or medical institutions.

[0764] For example, if a serious abnormality such as "chest pain" is detected, the server will make an emergency call.

[0765] Data Calculation:

[0766] Generate a notification message containing the anomaly information.

[0767] Output: A notification message is sent to family members and healthcare providers.

[0768] Step 6: Generate reports

[0769] Input: Interaction data periodically collected by the server

[0770] Specific behavior:

[0771] The server aggregates past interaction data and generates periodic health status reports.

[0772] The report includes a summary of the conversation, any anomalies detected, and recommended next steps.

[0773] Data Calculation:

[0774] Generate scheduled reports and convert them into email format.

[0775] Output: The report is sent to the designated family member or healthcare provider using SendGrid.

[0776] (Application example 1)

[0777] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0778] Since there is no system that simultaneously supports the health management and meal selection of the elderly, there is a need for a system that can grasp the health status of the elderly and suggest and order appropriate meals. The lack of such a system has led to issues such as the deterioration of the health status of the elderly and the inconvenience of meal selection.

[0779] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0780] In this invention, the server includes a means for proposing an appropriate meal menu based on the dialogue data and the user's health condition, a means for analyzing the dialogue data and evaluating the user's health condition, and a means for notifying the user and accepting an order if an abnormality in the user's health condition or a meal menu is detected. This enables meal suggestions that take the user's health condition into consideration and easy ordering.

[0781] "Devices used by the elderly" refers to electronic devices operated by the elderly, such as smartphones and tablets.

[0782] A "generative model" is an artificial intelligence technology that generates natural dialogue with users and healthy meal menus.

[0783] The "server" is a computer system that executes the generative model and manages and analyzes user interaction data and health data.

[0784] "User dialogue data" refers to data that records and manages the content of conversations with users.

[0785] The "means for proposing an appropriate meal menu based on health condition" is a means having a function of analyzing the user's health condition and presenting a meal menu suitable for that condition.

[0786] A "health status management database" is a collection of data that stores a user's health information and interaction history and can be accessed as needed.

[0787] The "means for presenting natural dialogue" is a function for displaying natural and easy-to-understand dialogue to the user.

[0788] The "means for proposing healthy meal menus" is a function for proposing healthy meal menus based on the user's health condition and dialogue data.

[0789] The "means for analyzing dialogue data and evaluating health status" is a function for analyzing dialogue data with the user and evaluating the health status.

[0790] "Means for notifying and accepting orders when abnormalities in health status or meal menus are detected" is a function that notifies and accepts orders when abnormalities in the user's health status occur or when selecting or ordering a meal menu.

[0791] This invention is an interactive system that supports elderly people in managing their health and choosing meals simultaneously. The system includes a terminal operated by the elderly, a server that executes generative models, and a database that manages dialogue data and health status.

[0792] 1. System Components

[0793] Devices used by the elderly

[0794] The devices are electronic devices operated by elderly people, such as smartphones and tablets. Through these devices, users can interact with a panda character and receive health assessments and meal menu suggestions.

[0795] Generative Model

[0796] A generative model is an AI technology that generates natural dialogue with users and healthy meal menus. Specifically, it detects keywords from user dialogue data and generates appropriate responses and meal menus based on them.

[0797] server

[0798] The server is a computer system that executes the generative model and manages and analyzes user interaction data and health data. For example, the server analyzes the user's health status and notifies the user if an abnormality is detected.

[0799] Database

[0800] The database is a collection of data that stores user interaction history and health information and can be accessed as needed, enabling continuous health monitoring.

[0801] 2. System Operation

[0802] Start a conversation

[0803] When the user presses the "Start Conversation" button on the device, the device sends this request to the server, which uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[0804] Collection and analysis of everyday conversations

[0805] As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The device then presents the generated new conversation data to the user by displaying and playing it back on the panda character.

[0806] Health monitoring

[0807] The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities. For example, if a user says, "I've been having stomach problems lately," the server uses a generative model to generate detailed questions and present them to the user.

[0808] Anomaly detection and notification

[0809] If the server detects any abnormalities in health, it will send the information to family members or medical institutions in real time via a notification function. For example, if someone says "I have chest pain," a notification will be sent immediately to family members or emergency medical institutions, and an emergency call can even be made.

[0810] Meal menu suggestions and ordering

[0811] Based on the dialogue data generated based on the user's health status, the server proposes a meal menu suitable for the user. The user can select from the proposed menu and place an order through the terminal. The order details are sent to the server, which then notifies the meal service provider.

[0812] 3. Specific Examples

[0813] Scenario: A user complains of stomach upset.

[0814] 1. Start a conversation

[0815] Terminal: "Hello! How are you feeling today?" asks the panda character to the user.

[0816] User: "I've been having stomach problems lately."

[0817] Terminal: Converts the user's speech into text and sends it to the server.

[0818] 2. Conversation analysis and health assessment

[0819] Server: Analyzes the keyword "I've been having stomach problems lately," and based on this, the generative model generates a new dialogue: "That's worrying. What's the best thing to eat to ease your stomach?"

[0820] Server: Sends the generated dialogue to the terminal.

[0821] Device: A panda character will display and play a voice saying, "That's worrying. What is the best food to eat for your stomach?"

[0822] 3. Meal menu suggestions

[0823] Server: Based on the user's health condition, the server suggests meal menus such as "healthy soup with less carbonated drinks" or "nutritiously balanced porridge with plenty of vegetables."

[0824] Terminal: A panda character displays the suggested menu.

[0825] 4. Order Confirmation

[0826] User: Selects from the proposed menu and places an order through the terminal.

[0827] Server: Notifies the food service provider of the order.

[0828] Server: Sends a confirmation message to the terminal saying, "Your order has been confirmed. Delivery will be in 30 minutes."

[0829] Prompt Sentence Examples

[0830] You: I've been having stomach problems lately.

[0831] Panda: That's worrying. What's the easiest thing to eat for your stomach?

[0832] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0833] Step 1:

[0834] The user presses the "Start conversation" button on the device.

[0835] Input: User presses a button

[0836] Output: Sending a conversation initiation request to the server

[0837] Specific operation: The device sends a conversation start request to the server. The server receives this request, uses the generative model to generate an initial greeting message, and sends it to the device.

[0838] Step 2:

[0839] The device will display and play a voice message to the user through the panda character as a greeting message.

[0840] Input: Initial greeting message from the server

[0841] Output: Present natural dialogue to the user

[0842] Specific operation: The device receives a greeting message from the server and displays it as a panda character, playing it aloud.

[0843] Step 3:

[0844] The user responds to the panda character.

[0845] Input: User's voice response

[0846] Output: Audio data

[0847] Specific operation: The device receives the user's voice response and converts the voice into text.

[0848] Step 4:

[0849] The terminal transmits the user's text data to the server.

[0850] Input: Text data converted from audio

[0851] Output: Sending text data to the server

[0852] Specific operation: The device sends the text data converted from the voice to the server.

[0853] Step 5:

[0854] The server analyzes the text data and generates the next question or response.

[0855] Input: User's text data

[0856] Output: The next question or response generated

[0857] Specific operation: The server analyzes the received text data using a generative model and generates the next appropriate question or response.

[0858] Step 6:

[0859] The server transmits the generated dialogue to the terminal.

[0860] Input: Generated dialogue

[0861] Output: Send dialogue to terminal

[0862] Specific operation: The server sends the generated dialogue to the terminal.

[0863] Step 7:

[0864] The device displays and plays aloud the dialogue generated through the panda character to the user.

[0865] Input: Generated dialogue from the server

[0866] Output: Presents dialogue to the user

[0867] Specific operation: The terminal displays and plays aloud the dialogue received from the server using a panda character.

[0868] Step 8:

[0869] The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities.

[0870] Input: Interaction data

[0871] Output: Health status assessment results, abnormality detection results

[0872] Specific operation: The server uses the generative model to analyze the dialogue data and detect specific keywords and signs of anomalies.

[0873] Step 9:

[0874] The server will suggest an appropriate meal menu based on the health assessment results.

[0875] Input: Health status assessment results

[0876] Output: Suggested meal menu

[0877] Specific operation: The server uses the health status assessment results to suggest health-conscious meal menus through a generative model.

[0878] Step 10:

[0879] The terminal displays the suggested meal menu to the user, who places an order.

[0880] Input: Suggested meal menu, user selection

[0881] Output: Order data

[0882] Specific operation: The terminal displays the proposed meal menu and the user selects. The user's order is sent from the terminal to the server.

[0883] Step 11:

[0884] The server transmits the order data to the food service provider.

[0885] Input: Order data

[0886] Output: Notification to food service provider

[0887] Specific operation: The server notifies the meal service provider of the user's order data, confirms the order, and sends a confirmation message to the user's terminal.

[0888] Through the above processing steps, the user can order an appropriate meal menu while monitoring their health condition.

[0889] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0890] MODE FOR CARRYING OUT THE INVENTION

[0891] The present invention combines an emotion engine with an interactive medical AI service system for the elderly. Specific embodiments of the system are described below.

[0892] Components

[0893] 1. Devices: Electronic devices such as tablets and smartphones used by the elderly that provide an interface that allows users to converse with panda characters on a daily basis.

[0894] 2. Server: A computer system that executes the generative model, manages and analyzes the dialogue data with the user, and transmits the generated dialogue and health assessment data to the terminal.

[0895] 3. Generative Model: This is an artificial intelligence technique used to generate natural dialogue with the user, detect specific keywords, and assess health status based on them.

[0896] 4. Database: A collection of data that stores user interaction data and can be accessed as needed.

[0897] 5. Emotion engine: This engine recognizes emotions from the user's voice and text data and adjusts the dialogue with the user based on the recognized emotional data.

[0898] System Operation

[0899] 1. Initial Setup: When a user uses the service for the first time, they install the app on their device and enter their personal information (name, age, health information, etc.). This information is sent to the server and stored in a database.

[0900] 2. Start conversation: The user presses the "Start conversation" button on the device. The device sends this request to the server, which uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[0901] 3. Collection and analysis of everyday conversations: As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The device presents the generated new conversation data to the user by displaying and playing it back on the panda character.

[0902] 4. Health monitoring: The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities. Based on this, the generative model generates appropriate responses and questions. For example, if the user says, "My lower back hurts," the server instructs the generative model to ask more detailed questions about the extent and location of the pain.

[0903] 5. Emotion recognition and adjustment: The device sends the user's voice and facial expression data to the emotion engine, which then recognizes the user's emotions. For example, if the user has an anxious expression or voice, the emotion engine sends that information to the server.

[0904] 6. Emotion-based dialogue adjustment: The server analyzes the emotion data received from the emotion engine and generates appropriate dialogue based on the emotion. For example, if the user is anxious, the panda character will generate a dialogue such as, "Is there anything you're worried about? Please tell me."

[0905] 7. Abnormality detection and notification: If the server detects an abnormality in health status, the notification function will send the information to family members and medical institutions in real time. For example, if someone says, "I have chest pain," a notification will be sent immediately to family members and emergency medical institutions, and if necessary, an emergency call will be made.

[0906] 8. Report Generation: The server periodically generates a report summarizing the user's health status and emails it to designated family members or healthcare providers. The report includes a summary of the conversation, any abnormalities detected, and recommended next steps.

[0907] Specific examples

[0908] Scenario: An elderly person complains of back pain and becomes emotionally unstable.

[0909] 1. Start a conversation

[0910] Terminal: "Hello! How are you today?" asks the panda character to the user.

[0911] User: "I've been having back pain lately..."

[0912] Terminal: Converts the user's speech into text and sends it to the server.

[0913] 2. Conversation Analysis and Anomaly Detection

[0914] Server: Analyzes the keyword "my lower back hurts," and based on this, the generative model generates a new dialogue: "That's worrying. Where does it hurt?"

[0915] Server: Sends the generated dialogue to the terminal.

[0916] Device: Panda will say and listen, "That's worrying. Where does it hurt?"

[0917] 3. Emotional Recognition

[0918] Device: The user answers in an anxious voice, "My whole lower back hurts."

[0919] Device: Sends voice data to the emotion engine, which detects anxiety.

[0920] Server: Receives emotion data from the emotion engine and generates dialogue based on it.

[0921] 4. Adjusting dialogue based on emotions

[0922] Server: Based on the information from the emotion engine, it generates the dialogue, "You're okay. Take a rest and don't push yourself too hard. Is there anything I can help you with?"

[0923] Server: Sends the generated dialogue to the terminal.

[0924] Device: Panda will say, "It's okay. Take care and rest. Is there anything I can help you with?"

[0925] This invention allows elderly people to enjoy daily conversations with panda characters while simultaneously managing their health. It also responds sensitively to the user's emotions and provides appropriate dialogue, providing a greater sense of security. Family members and medical institutions can monitor the elderly's health status in real time, contributing to improving the quality of life for the elderly.

[0926] The processing flow will be explained below.

[0927] Step 1:

[0928] Device: Display the app installation screen.

[0929] User: Installs and launches the app.

[0930] Step 2:

[0931] Device: Display the initial setup screen (name, age, health information, etc.).

[0932] User: Enter the required information and press the send button.

[0933] Terminal: Sends the entered information to the server.

[0934] Server: Stores the received information in a database and sends a response to the terminal indicating that initial setup is complete.

[0935] Terminal: Notify the user that the initial setup is complete.

[0936] Step 3:

[0937] Device: Display a "Start Conversation" button on the home screen.

[0938] User: Press the start conversation button.

[0939] Terminal: Sends a request to start a conversation to the server.

[0940] Step 4:

[0941] Server: Initiates the conversation session and generates the initial greeting message.

[0942] Server: Sends a greeting message to the terminal.

[0943] On your device: A panda character will display and / or play a greeting message (e.g., "Hello! How was your day today?").

[0944] Step 5:

[0945] User: Answer the panda character's questions (e.g., "It was nice weather today so I went for a walk").

[0946] Terminal: Converts the user's speech into text and sends it to the server.

[0947] Step 6:

[0948] Server: Analyzes the received text data and generates the next question or response.

[0949] Server: Sends the generated conversation data to the terminal.

[0950] On your device: The panda character will display and play your new message (e.g., "That's lovely! Which way did you walk?").

[0951] Step 7:

[0952] Server: Continuously analyzes conversation data and detects keywords and anomalies.

[0953] Server: Generates new responses and questions based on the detected information.

[0954] Device: View and play new questions and responses from the panda character.

[0955] Step 8:

[0956] User: Provides health information as the conversation continues (e.g., "I've been having a lot of back pain lately...").

[0957] Terminal: Converts the user's speech into text and sends it to the server.

[0958] Step 9:

[0959] Server: Analyzes keywords that indicate abnormal health conditions, such as "lower back pain," and detects abnormalities.

[0960] Server: If an abnormality is detected, it generates a notification to notify family members or medical institutions.

[0961] Server: Send notifications to family members or medical institutions (e.g., "You've recently been complaining of back pain. You may need further evaluation.").

[0962] Step 10:

[0963] Strategic terminal: Sends the user's voice and facial expression data to the emotion engine.

[0964] Emotion engine: Recognizes user emotions and detects emotions such as anxiety.

[0965] Emotion Engine: Sends detected emotion data to the server.

[0966] Step 11:

[0967] Server: Analyzes the received emotion data and generates appropriate dialogue based on the emotion.

[0968] Server: Sends the generated dialogue to the terminal.

[0969] On your device: The panda character will display and / or play a sensitive message (e.g., "It's okay. Take care and rest. Is there anything I can help you with?").

[0970] Step 12:

[0971] Server: Periodically aggregates conversation data and generates a health status report for the elderly.

[0972] Server: Sends the generated report to the family or medical institution.

[0973] Step 13:

[0974] Device: Continue everyday conversations with panda characters.

[0975] User: Continue to provide information about health status through interactions with the panda character.

[0976] Example 2

[0977] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0978] Elderly people need to be sensitive to changes in their health and emotions in their daily lives. However, it is often difficult for them to visit medical institutions or communicate with specialists on a daily basis. For this reason, there is a need for a system that allows elderly people to easily manage their health at home and receive appropriate support in response to emotional changes.

[0979] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0980] In this invention, the server includes means for analyzing the dialogue data to evaluate the health condition, means for notifying when an abnormality in the health condition is detected, and means for adjusting the dialogue based on the emotion data, thereby enabling the user to manage their health condition while at home and receive appropriate dialogue in response to changes in their emotions.

[0981] "Communication devices" are electronic devices such as tablets and smartphones used by elderly people, and are devices that provide an interface for users to interact with each other on a daily basis.

[0982] An "information processing device" is a computer system that executes generative models and manages and analyzes user interaction data.

[0983] A "storage device" is a collection of data that stores user interaction data and can be accessed as needed.

[0984] A "generative model" is an artificial intelligence technique used to generate natural-sounding dialogue with a user.

[0985] An "emotion recognition engine" is an engine that recognizes emotions from the user's voice and text data and adjusts the dialogue based on that information.

[0986] "Dialogue data" refers to data such as messages and voice recordings exchanged between a user and a communication device.

[0987] "Health status assessment" is the process of analyzing dialogue data and determining the user's health status.

[0988] "Notification means" is a function that sends information to family members or medical institutions when an abnormality in health condition is detected.

[0989] "Emotion data" is data that indicates the emotional state of the user analyzed by the emotion recognition engine.

[0990] "Dialogue adjustment" is the process of modifying the dialogue content and providing an appropriate response to the user based on emotion data.

[0991] This invention relates to an interactive medical support system for the elderly, and in particular to a system that utilizes a generative AI model with emotion recognition capabilities. The specific operation and usage of the system are described below.

[0992] System Overview

[0993] The system includes a communication device (e.g., a tablet or smartphone) used by the elderly, an information processing device that executes a generative model, and a storage device that manages interaction data with the user. The system includes the following means:

[0994] A means for communication devices to present natural dialogue to users

[0995] The means by which the generative model generates interactions with the user

[0996] A means for an information processing device to analyze the dialogue data and evaluate the health condition

[0997] A means for notifying when an information processing device detects an abnormality in health status

[0998] A means for communication devices to send user voice and facial expression data to an emotion recognition engine

[0999] The emotion recognition engine analyzes the user's emotions and transmits the results to the information processing device.

[1000] A means for an information processing device to adjust a dialogue based on emotion data

[1001] Initial Setup

[1002] When a user uses the system for the first time, they install a dedicated app on their communication device and enter personal information such as their name, age, and health information, which is then sent to the server and stored in a database.

[1003] Start a conversation

[1004] When the user presses the "Start Conversation" button on the communication device, the communication device sends this request to the server. The server uses the generative model to generate an initial greeting message and sends it to the communication device. The communication device then displays and plays the greeting message aloud using a panda character.

[1005] Collection and analysis of everyday conversations

[1006] As the user engages in everyday conversation with the panda character, the communication device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The generated dialogue is sent to the communication device, where it is displayed and played aloud by the panda character.

[1007] Health monitoring

[1008] The server continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. For example, if a user says, "My lower back hurts," the server instructs the generative model to ask more detailed questions about the extent and location of the pain.

[1009] Emotional awareness and regulation

[1010] The communication device sends the user's voice and facial expression data to an emotion recognition engine, which then analyzes the user's emotions. For example, if the user has an anxious expression or voice, that information is sent to the server. The server analyzes the emotional data, generates appropriate dialogue based on the emotion, and sends it to the communication device.

[1011] Health Assessment and Notification

[1012] If the server detects any abnormalities in a person's health, it will notify family members or medical institutions in real time. For example, if someone says, "I have chest pain," a notification will be sent immediately, and an emergency call will be made if necessary.

[1013] Report Generation

[1014] The server periodically generates a report summarizing the user's health status and emails it to designated family members or healthcare providers, including a summary of the conversation, any abnormalities detected, and recommended next steps.

[1015] Specific examples

[1016] Scenario: An elderly person complains of back pain and becomes emotionally unstable.

[1017] 1. Start a conversation

[1018] Terminal: "Hello! How are you today?" asks the panda character to the user.

[1019] User: "I've been having back pain lately..."

[1020] Terminal: Converts the user's speech into text and sends it to the server.

[1021] 2. Conversation Analysis and Anomaly Detection

[1022] Server: Analyzes the keyword "my lower back hurts," and based on this, the generative model generates a new dialogue: "That's worrying. Where does it hurt?"

[1023] Server: Sends the generated dialogue to the terminal.

[1024] Device: A panda character will display and play a voice message saying, "That's worrying. Where does it hurt?"

[1025] 3. Emotional Recognition

[1026] User: In an anxious voice, replies, "My lower back hurts all over."

[1027] Device: Sends voice data to the emotion engine to detect anxiety.

[1028] Server: Receives the emotion data and generates a dialogue such as, "You're okay. Take care and rest. Is there anything I can help you with?"

[1029] 4. Coordinating dialogue

[1030] Server: Sends the generated dialogue to the terminal.

[1031] Device: A panda character will display and play a voice saying, "It's okay. Please take it easy and rest. Is there anything I can help you with?"

[1032] Prompt Sentence Examples

[1033] "The user complains of back pain and appears anxious. Based on this, please generate a dialogue that will reassure the user."

[1034] This invention allows elderly people to manage their health through the panda character in their daily lives and provides appropriate dialogue, allowing them to spend their days with peace of mind.Furthermore, family members and medical institutions can monitor their health status in real time, contributing to improving the quality of life for elderly people.

[1035] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1036] Step 1: Initial Setup

[1037] User: Installs the app on the device and enters personal information (name, age, health information, etc.) through the app's input form.

[1038] Terminal: Receives input personal information and converts it into JSON format to send to the server. The input is personal information, and the output is the converted JSON data.

[1039] Server: Analyzes the received JSON data and saves the user information in a database. This allows basic user information to be managed on the server side.

[1040] Step 2: Start a conversation

[1041] User: Presses the "Start conversation" button on the device. Input is the action of clicking the "Start conversation" button.

[1042] Terminal: Generates a conversation initiation request and sends it to the server. The output is the generated request data.

[1043] Server: Receives the request and generates the initial greeting message using the generative model. The input is the request and the output is the greeting message.

[1044] Server: Sends the generated greeting message to the terminal. The output is the generated message data.

[1045] Terminal: The panda character displays and plays audibly the received greeting message. The input is the greeting message, and the output is the display and audio playback for the user.

[1046] Step 3: Collecting and analyzing everyday conversations

[1047] User: Have everyday conversations with a panda character. Input is via user voice and text.

[1048] Terminal: Converts user speech from voice to text. This conversion is done by speech recognition software. The input is the user's voice, and the output is text data.

[1049] Terminal: Transmits converted text data to the server. The output is text data.

[1050] Server: Analyzes the received text data and generates the next question or response. This analysis is performed using a natural language processing model. The input is text data, and the output is a new dialogue.

[1051] Server: Sends the generated dialogue to the terminal. The output is dialogue data.

[1052] Terminal: The panda character displays and plays audibly the received dialogue. The input is the dialogue, and the output is a display and audible playback for the user.

[1053] Step 4: Monitor your health

[1054] Server: Continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. A keyword extraction algorithm is used for analysis. The input is dialogue data, and the output is health assessment data.

[1055] Server: When an anomaly is detected, it uses a generative model to generate appropriate questions and responses. The input is the anomaly detection result, and the output is the new questions and responses.

[1056] Server: Sends the generated dialogue to the terminal. The output is dialogue data.

[1057] Terminal: The panda character displays and plays audibly the received dialogue. The input is the dialogue, and the output is a display and audible playback for the user.

[1058] Step 5: Recognize and regulate your emotions

[1059] Terminal: Sends the user's voice and facial expression data to the emotion recognition engine. This transmission is done in real time. The input is voice and facial expression data, and the output is the transmitted data.

[1060] Emotion recognition engine: Analyzes the user's emotions and sends the results to the server. The input is voice and facial expression data, and the output is emotional data.

[1061] Server: Receives emotional data and adjusts the dialogue based on the analysis results. The input is emotional data, and the output is the adjusted dialogue.

[1062] Server: Sends the adjusted dialogue to the terminal. The output is dialogue data.

[1063] Terminal: The panda character displays and plays audibly the received dialogue. The input is the dialogue, and the output is a display and audible playback for the user.

[1064] Step 6: Health Assessment and Notification

[1065] Server: When an abnormality in health status is detected, the notification function is activated. This function includes push notifications and email notifications. The input is the abnormality detection result, and the output is the notification data.

[1066] Server: Notifies family and medical institutions of health status in real time. Makes emergency calls if necessary. Input is notification data, output is notification and alert.

[1067] Step 7: Generate reports

[1068] Server: Periodically generates a report summarizing the user's health status. This report includes a summary of the conversation, any abnormalities detected, and recommended next steps. The input is the conversation data and health assessment data, and the output is the report.

[1069] Server: Sends the generated report to the designated family or medical institution by email. The input is the report data, and the output is the sent report.

[1070] (Application example 2)

[1071] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1072] Health management, especially for the elderly, is an important issue in modern society. Elderly people require daily health monitoring, and prompt action is required when abnormalities occur. However, conventional systems do not provide natural dialogue and are difficult to respond to, taking into account the user's emotions. Therefore, there is a need for a system that combines natural dialogue and emotion recognition that can be used comfortably by the elderly.

[1073] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for analyzing dialogue data and evaluating health status; means for the server to notify when it detects an abnormality in health status; means for the terminal and the server to recognize emotions and adjust the dialogue based on the emotions; means for the terminal to check the health status of residents of the welfare facility through everyday conversation; and means for the server to periodically generate health status reports and provide them to a specified contact. This enables elderly people to manage their health with a sense of security through more natural dialogue.

[1074] The term "elderly" refers to people over a certain age, and in healthcare and welfare in particular, generally refers to people aged 65 and over.

[1075] A "terminal" is an electronic device used by a user, which inputs and outputs data via an interface.

[1076] A "generative model" is an algorithm or program that uses artificial intelligence technology to generate natural-sounding dialogue.

[1077] A "server" is a computer system that manages data and programs and provides services to terminals via a network.

[1078] A "database" is a collection of data that stores user interaction data and health information and can be accessed as needed.

[1079] "Emotions" refer to psychological states such as joy, sadness, and anxiety that humans experience, and can be read from voice and facial expressions.

[1080] A "dialogue" is a written communication between a user and a terminal.

[1081] "Abnormal" refers to a state that deviates from normal health and requires immediate action.

[1082] "Notification" is the act of communicating specific information to designated recipients in real time.

[1083] A "welfare facility" refers to a dedicated facility where elderly people, people with disabilities, etc. can live while receiving support and care.

[1084] MODE FOR CARRYING OUT THE INVENTION

[1085] This invention describes an embodiment of an interactive medical AI service system for elderly people specifically implemented in a welfare facility. The components and operation of the system are described in detail below.

[1086] System Configuration

[1087] 1. Devices: Electronic devices such as tablets and smartphones used by the elderly. Devices provide an interface with users and have the ability to display and play natural dialogue.

[1088] 2. Generative model: An artificial intelligence algorithm that uses natural language processing to generate dialogue with the user. Specifically, it incorporates Spacy and BERT.

[1089] 3. Server: A computer system that manages the database and generative model and sends dialogues and notifications to devices via a network. The server is built using Python or Flask and is responsible for data analysis and dialogue generation.

[1090] 4. Emotion Engine: An algorithm that recognizes emotions from the user's voice and text data and adjusts the dialogue accordingly. The EmotionEngine class is an example of this.

[1091] 5. Database: A system for storing conversation data and health information. MySQL, PostgreSQL, etc. are used.

[1092] Operation overview

[1093] 1. Initial setup: The server stores the user's basic information (name, age, health information, etc.) in a database. During the initial setup, the user installs the app on their device and enters the required information.

[1094] 2. Start of conversation: When the user presses the "Start conversation" button on the device, the server uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[1095] 3. Collection and analysis of everyday conversations: As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. For example, if the user says, "I've been having a pain in my lower back lately...", the generative model will generate a response such as, "That's worrying. Where does it hurt?"

[1096] 4. Health status monitoring: The server continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. Based on this, the generative model generates appropriate responses and questions. For example, if the keyword "chest pain" is detected, this information will be promptly notified to family members or medical institutions.

[1097] 5. Emotion recognition and adjustment: The device sends the user's voice and facial expression data to the emotion engine, which then recognizes the user's emotions. For example, if the user's facial expression or voice looks anxious, the emotion engine detects the anxiety and sends this information to the server. The server analyzes the emotion data and generates a dialogue based on the emotion.

[1098] 6. Periodic Report Generation: The server periodically generates a report summarizing the user's health status and provides it to designated contacts (family members or healthcare providers), including a summary of the conversation, any abnormalities detected, and recommended next steps.

[1099] Specific examples

[1100] Scenario: An elderly person complains of lower back pain

[1101] 1. Start a conversation

[1102] User: "I've been having back pain lately..."

[1103] System: "That's concerning. Where does it hurt?"

[1104] 2. Emotional Recognition

[1105] User: "My lower back hurts all over."

[1106] System: The emotion engine detects the user's anxiety and generates a response such as, "You're OK. Take care and get some rest. Is there anything I can help you with?"

[1107] With these actions, elderly people can manage their health with peace of mind through everyday conversation. In addition, if any abnormalities occur, family members and medical institutions will be notified promptly, allowing them to live their lives with peace of mind.

[1108] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1109] Step 1: Initial Setup

[1110] Users install a medical app for the elderly on their device and enter their personal information (name, age, health information, etc.).

[1111] Input: User personal information

[1112] Processing: The device sends this information to the server, which stores it in a database.

[1113] Output: User information stored in the database

[1114] Step 2: Start a conversation

[1115] The user presses the "Start conversation" button on the device.

[1116] Input: User operation (pressing the "Start conversation" button)

[1117] Processing: The server uses the generative model to generate an initial greeting message and sends it to the terminal.

[1118] Output: A greeting message displayed on the terminal (natural dialogue)

[1119] Step 3: Collecting and analyzing everyday conversations

[1120] Users can converse with a panda character.

[1121] Input: User's voice data

[1122] Processing: The device converts the voice data into text and sends it to the server. The server analyzes the received text, and the generative model generates the next question or response. The server then sends the generated dialogue back to the device.

[1123] Output: The new dialogue displayed on the terminal.

[1124] Step 4: Monitor your health

[1125] The server analyzes the conversation data and detects specific keywords and signs of abnormalities.

[1126] Input: User's text data

[1127] Processing: The server detects specific keywords (e.g., "my lower back hurts") and evaluates the health status. If an abnormality is detected, it instructs the generative model on the next course of action.

[1128] Output: The next dialogue generated, and the anomaly detection results.

[1129] Step 5: Recognize and regulate your emotions

[1130] The device sends the user's voice and facial expression data to the emotion engine.

[1131] Input: User's voice and facial expression data

[1132] Processing: The emotion engine analyzes these data to recognize emotions and sends the results to the server. The server analyzes the emotion data, and the generative model generates appropriate dialogue based on the emotion.

[1133] Output: Generated dialogue (content corresponding to emotions)

[1134] Step 6: Anomaly detection and notification

[1135] If the server detects any abnormalities in health, it will use the notification function to send information to family members or medical institutions in real time.

[1136] Input: Anomaly detection results

[1137] Processing: The server generates a notification and sends it to the specified contacts, including calling emergency services if necessary.

[1138] Output: Notification sent to family and healthcare providers

[1139] Step 7: Generate reports

[1140] The server periodically generates a report summarizing the user's health status and emails it to designated contacts.

[1141] Input: Dialogue data and emotion data

[1142] Processing: The server compiles and analyzes the data, generates a report, and sends the report to the designated contact.

[1143] Output: Health report sent to family and healthcare providers

[1144] Through the above processing steps, elderly people can manage their health through natural dialogue and emotion recognition, and can live with peace of mind even in welfare facilities.

[1145] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1146] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1147] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1148] [Third embodiment]

[1149] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1150] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1151] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1152] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1153] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1154] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1155] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1156] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1157] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1158] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1159] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1160] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1161] MODE FOR CARRYING OUT THE INVENTION

[1162] The present invention provides an interactive medical AI service system for elderly people. Specific embodiments of the system are described below.

[1163] Components

[1164] 1. Devices: Electronic devices such as tablets and smartphones used by the elderly. They provide an interface that allows users to converse with panda characters on a daily basis.

[1165] 2. Server: A computer system that executes the generative model, manages and analyzes the dialogue data with the user, and transmits the generated dialogue and health assessment data to the terminal.

[1166] 3. Generative Model: This is an artificial intelligence technique used to generate natural dialogue with the user, detect specific keywords, and assess health status based on them.

[1167] 4. Database: A collection of data that stores user interaction data and can be accessed as needed.

[1168] System Operation

[1169] 1. Initial Setup: When a user uses the service for the first time, they install the app on their device and enter their personal information (name, age, health information, etc.). This information is sent to the server and stored in a database.

[1170] 2. Start conversation: The user presses the "Start conversation" button on the device. The device sends this request to the server, which uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[1171] 3. Collection and analysis of everyday conversations: As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The device presents the generated new conversation data to the user by displaying and playing it back on the panda character.

[1172] 4. Health monitoring: The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities. Based on this, the generative model generates appropriate responses and questions. For example, if the user says, "My lower back hurts," the server instructs the generative model to ask more detailed questions about the extent and location of the pain.

[1173] 5. Abnormality detection and notification: If the server detects an abnormality in health status, the notification function will send the information to family members and medical institutions in real time. For example, if someone says, "I have chest pain," a notification will be sent immediately to family members and emergency medical institutions, and if necessary, an emergency call will be made.

[1174] 6. Report Generation: The server periodically generates a report summarizing the user's health status and emails it to designated family members or healthcare providers. The report includes a summary of the conversation, any abnormalities detected, and recommended next steps.

[1175] Specific examples

[1176] Scenario: Conversation with an elderly person complaining of lower back pain

[1177] 1. Start a conversation

[1178] Terminal: "Hello! How are you today?" asks the panda character to the user.

[1179] User: "I've been having back pain lately..."

[1180] Terminal: Converts the user's speech into text and sends it to the server.

[1181] 2. Conversation Analysis and Anomaly Detection

[1182] Server: Analyzes the keyword "my lower back hurts," and based on this, the generative model generates a new dialogue: "That's worrying. Where does it hurt?"

[1183] Server: Sends the generated dialogue to the terminal.

[1184] Device: Panda will say and listen, "That's worrying. Where does it hurt?"

[1185] 3. Notifications and Reporting

[1186] Server: Detects abnormalities based on data related to lower back pain and sends a notification to the family member stating, "You have recently been complaining of lower back pain. A more detailed diagnosis may be necessary."

[1187] Server: Generates periodic health status reports and sends them to family members.

[1188] This invention allows elderly people to enjoy daily conversations with panda characters while simultaneously managing their health.Furthermore, family members and medical institutions can monitor their health status in real time, contributing to improving the quality of life for elderly people.

[1189] The processing flow will be explained below.

[1190] Step 1:

[1191] Device: Display the app installation screen.

[1192] User: Installs and launches the app.

[1193] Step 2:

[1194] Device: Display the initial setup screen (name, age, health information, etc.).

[1195] User: Enter the required information and press the send button.

[1196] Terminal: Sends the entered information to the server.

[1197] Server: Stores the received information in a database and sends a response to the terminal indicating that initial setup is complete.

[1198] Terminal: Notify the user that the initial setup is complete.

[1199] Step 3:

[1200] Device: Display a "Start Conversation" button on the home screen.

[1201] User: Press the start conversation button.

[1202] Terminal: Sends a request to start a conversation to the server.

[1203] Step 4:

[1204] Server: Initiates the conversation session and generates the initial greeting message.

[1205] Server: Sends a greeting message to the terminal.

[1206] On your device: A panda character will display and / or play a greeting message (e.g., "Hello! How was your day today?").

[1207] Step 5:

[1208] User: Answer the panda character's questions (e.g., "It was nice weather today so I went for a walk").

[1209] Terminal: Converts the user's speech into text and sends it to the server.

[1210] Step 6:

[1211] Server: Analyzes the received text data and generates the next question or response.

[1212] Server: Sends the generated conversation data to the terminal.

[1213] On your device: The panda character will display and play your new message (e.g., "That's lovely! Which way did you walk?").

[1214] Step 7:

[1215] Server: Continuously analyzes conversation data and detects keywords and anomalies.

[1216] Server: Generates new responses and questions based on the detected information.

[1217] Device: View and play new questions and responses from the panda character.

[1218] Step 8:

[1219] User: Provides health information as the conversation continues (e.g., "I've been having a lot of back pain lately...").

[1220] Terminal: Converts the user's speech into text and sends it to the server.

[1221] Step 9:

[1222] Server: Analyzes keywords that indicate abnormal health conditions, such as "lower back pain," and detects abnormalities.

[1223] Server: If an abnormality is detected, it generates a notification to notify family members or medical institutions.

[1224] Server: Send notifications to family members or medical institutions (e.g., "You've recently been complaining of back pain. You may need further evaluation.").

[1225] Step 10:

[1226] Server: Periodically aggregates conversation data and generates a health status report for the elderly.

[1227] Server: Sends the generated report to the family or medical institution.

[1228] Step 11:

[1229] Device: Continue everyday conversations with panda characters.

[1230] User: Continue to provide information about health status through interactions with the panda character.

[1231] Example 1

[1232] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1233] It is difficult for elderly people to manage their own health status in their daily lives and detect abnormalities early. It is also difficult for family members and medical institutions to grasp the health status of elderly people in real time. This increases health risks, especially for elderly people living alone. Furthermore, there is currently a lack of systems that assess the health status of elderly people through natural dialogue.

[1234] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1235] In this invention, the server includes a means for analyzing the dialogue data to evaluate the health condition, a means for notifying when an abnormality in the health condition is detected, and a means for continuously monitoring the health condition based on the dialogue data. This allows the user to manage their own health condition through natural dialogue, and enables family members and medical institutions to understand the health condition in real time.

[1236] "Elderly" refers to users who are aging and who are particularly in need of medical and nursing care support.

[1237] An "information terminal" is an electronic device used by a user, and refers to a device that has the shape of a tablet, smartphone, etc.

[1238] A "natural language generation model" is an artificial intelligence technology used to generate natural dialogue with users, and specifically refers to a computer program that performs language processing.

[1239] "Computer system" refers generally to a computer system for executing generative models and analyzing and managing interaction data.

[1240] "Data Collection" refers to a database system that stores user interaction data and makes it accessible as needed.

[1241] "Natural dialogue" refers to text that is generated naturally in interaction with a user, just like human language.

[1242] "Interaction Data" refers to data including voice, text, and other related information relating to a conversation between a user and a system.

[1243] "Health status" refers to the state of the user's physical and psychological health, and specifically includes physical condition, presence or absence of pain, mental state, etc.

[1244] "Abnormal" refers to any symptom or sign that deviates from normal health, especially one that requires medical attention.

[1245] "Notification" refers to the process of communicating information about an abnormality in a user's health condition to family members or medical institutions.

[1246] "Monitoring" refers to the process of continuously observing a user's health status and assessing it based on specific conditions.

[1247] MODE FOR CARRYING OUT THE INVENTION

[1248] The present invention provides an interactive medical AI service system for elderly people. Specific embodiments of the system are described below.

[1249] Hardware and Software Configuration

[1250] Device:

[1251] This refers to tablets and smartphones that are used daily by seniors, including iPads, Android tablets, iPhones, and Android smartphones. These devices have dedicated applications installed.

[1252] server:

[1253] The server is a computing system equipped with a high-performance CPU, memory, and an NVIDIA GPU specialized for running generative models. This is hardware for efficiently running generative models on the server side.

[1254] Generative model:

[1255] Generative models are natural language processing technologies, including OpenAI's GPT series and Google's BERT model, that are used to generate natural dialogue with users.

[1256] Database:

[1257] This is a database system for managing dialogue data, and uses database engines such as MySQL or MongoDB.

[1258] Program processing

[1259] 1. Initial Setup:

[1260] When a user uses the system for the first time, they launch the app installed on their device and enter personal information such as their name, age, and health information. This information is sent from the device to the server, which then stores the received information in a database.

[1261] 2. Start a conversation:

[1262] The user presses the "Start Conversation" button on the device app. This request is sent from the device to the server, and the server uses the generative model to generate an initial greeting message. The generated greeting message is then sent to the device, where the panda character displays and plays the greeting message.

[1263] 3. Collection and analysis of everyday conversations:

[1264] As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server (using the Google Cloud Speech-to-Text API). The server analyzes the received text data and generates the next question or response. The generated new conversation data is sent to the device and displayed and played back by the panda character.

[1265] 4. Health monitoring:

[1266] The server continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. Based on this information, the generative model generates appropriate responses and questions. For example, if a user says, "My lower back hurts," the generative model creates a question like, "That's worrying. Where does it hurt?"

[1267] 5. Anomaly detection and notification:

[1268] If the server detects any abnormalities in the user's health, it will use Twilio's SMS API to send a notification to family members or medical institutions. For example, if the user says something like "I have chest pain," it will call an emergency number.

[1269] 6. Report Generation:

[1270] The server periodically generates a report on the user's health status and emails it to designated family members or healthcare providers using SendGrid, including a summary of the conversation, any anomalies detected, and recommended next steps.

[1271] Specific examples

[1272] Scenario: An elderly person complains of back pain.

[1273] 1. Start the conversation:

[1274] Terminal: "Hello! How are you today?" asks the panda character to the user.

[1275] User: "I've been having back pain lately..."

[1276] Terminal: Converts speech into text and sends it to the server.

[1277] 2. Conversation analysis and anomaly detection:

[1278] Server: Analyzes the keyword "my lower back hurts" and uses a generative model (GPT-3) to generate a new dialogue: "That's worrying. Where does it hurt?"

[1279] Server: Sends the generated dialogue to the terminal.

[1280] Device: The new dialogue will be displayed and played by the panda character.

[1281] 3. Notifications and Reporting:

[1282] Server: Analyzes data related to lower back pain and detects abnormalities.

[1283] Server: Send a notification to the family member via Twilio's SMS API saying, "You've been complaining of back pain recently. A more detailed diagnosis may be needed."

[1284] Server: Generates periodic health status reports and sends them to family members using SendGrid.

[1285] This system allows elderly people to enjoy daily interactions with panda characters while simultaneously managing their own health conditions, allowing their families and medical institutions to monitor their health conditions in real time and contributing to improving the quality of life of the elderly.

[1286] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1287] Step 1: Initial Setup

[1288] Input: A user installs the app on their tablet or smartphone and enters personal information such as their name, age, and health information.

[1289] Specific behavior:

[1290] The user launches the app and enters their personal information on the "New Registration" screen.

[1291] The terminal sends the entered information to the server in JSON format.

[1292] Data processing:

[1293] The server parses the received JSON data, converts it into the required format, and saves it in the database.

[1294] Output: Personal information is stored on the server and recorded in a database.

[1295] Step 2: Start a conversation

[1296] Input: The user presses the "Start conversation" button in the device app.

[1297] Specific behavior:

[1298] When the user presses the "Start conversation" button, the device sends an HTTP request to the server.

[1299] The server receives the HTTP request and inputs the initial prompt "greet the user" into the generative model.

[1300] Data Calculation:

[1301] The generative model generates an initial greeting message based on the prompt sentence.

[1302] Output: The generated greeting message is sent back to the device, and the panda character displays and plays "Hello! How are you today?"

[1303] Step 3: Collecting and analyzing everyday conversations

[1304] Input: The user continues to talk to the panda character.

[1305] Specific behavior:

[1306] When a user says, "My back has been hurting lately...", the device uses the Google Cloud Speech-to-Text API to convert the speech to text.

[1307] The terminal transmits the converted text data to the server.

[1308] Data Calculation:

[1309] The server analyzes the text data and detects the keyword "my lower back hurts."

[1310] The server inputs the prompt statement "If the user says that their lower back hurts, generate a response" into the generative model.

[1311] Output: The generative model generates a response saying, "That's worrying. Where does it hurt?" and sends it to the device.

[1312] The panda character displays new dialogue and plays it aloud: "That's worrying. Where does it hurt?"

[1313] Step 4: Monitor your health

[1314] Input: Continuous conversation data between the user and the panda character

[1315] Specific behavior:

[1316] The server analyzes all received interaction data in real time.

[1317] To detect specific keywords or signs of anomalies, the server inputs the prompt statement "Generate a new question containing specific keywords" into the generative model.

[1318] Data Calculation:

[1319] The generative model generates appropriate responses and questions to prepare for the next dialogue.

[1320] Output: The appropriate questions and responses are generated and sent to the device.

[1321] A panda character presents the generated questions and responses to the user.

[1322] Step 5: Anomaly detection and notification

[1323] Input: Anomaly information based on conversation data analyzed by the server

[1324] Specific behavior:

[1325] If the server detects any abnormalities in health status, it will use Twilio's SMS API to send notifications to family members or medical institutions.

[1326] For example, if a serious abnormality such as "chest pain" is detected, the server will make an emergency call.

[1327] Data Calculation:

[1328] Generate a notification message containing the anomaly information.

[1329] Output: A notification message is sent to family members and healthcare providers.

[1330] Step 6: Generate reports

[1331] Input: Interaction data periodically collected by the server

[1332] Specific behavior:

[1333] The server aggregates past interaction data and generates periodic health status reports.

[1334] The report includes a summary of the conversation, any anomalies detected, and recommended next steps.

[1335] Data Calculation:

[1336] Generate scheduled reports and convert them into email format.

[1337] Output: The report is sent to the designated family member or healthcare provider using SendGrid.

[1338] (Application example 1)

[1339] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1340] Since there is no system that simultaneously supports the health management and meal selection of the elderly, there is a need for a system that can grasp the health status of the elderly and suggest and order appropriate meals. The lack of such a system has led to issues such as the deterioration of the health status of the elderly and the inconvenience of meal selection.

[1341] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1342] In this invention, the server includes a means for proposing an appropriate meal menu based on the dialogue data and the user's health condition, a means for analyzing the dialogue data and evaluating the user's health condition, and a means for notifying the user and accepting an order if an abnormality in the user's health condition or a meal menu is detected. This enables meal suggestions that take the user's health condition into consideration and easy ordering.

[1343] "Devices used by the elderly" refers to electronic devices operated by the elderly, such as smartphones and tablets.

[1344] A "generative model" is an artificial intelligence technology that generates natural dialogue with users and healthy meal menus.

[1345] The "server" is a computer system that executes the generative model and manages and analyzes user interaction data and health data.

[1346] "User dialogue data" refers to data that records and manages the content of conversations with users.

[1347] The "means for proposing an appropriate meal menu based on health condition" is a means having a function of analyzing the user's health condition and presenting a meal menu suitable for that condition.

[1348] A "health status management database" is a collection of data that stores a user's health information and interaction history and can be accessed as needed.

[1349] The "means for presenting natural dialogue" is a function for displaying natural and easy-to-understand dialogue to the user.

[1350] The "means for proposing healthy meal menus" is a function for proposing healthy meal menus based on the user's health condition and dialogue data.

[1351] The "means for analyzing dialogue data and evaluating health status" is a function for analyzing dialogue data with the user and evaluating the health status.

[1352] "Means for notifying and accepting orders when abnormalities in health status or meal menus are detected" is a function that notifies and accepts orders when abnormalities in the user's health status occur or when selecting or ordering a meal menu.

[1353] This invention is an interactive system that supports elderly people in managing their health and choosing meals simultaneously. The system includes a terminal operated by the elderly, a server that executes generative models, and a database that manages dialogue data and health status.

[1354] 1. System Components

[1355] Devices used by the elderly

[1356] The devices are electronic devices operated by elderly people, such as smartphones and tablets. Through these devices, users can interact with a panda character and receive health assessments and meal menu suggestions.

[1357] Generative Model

[1358] A generative model is an AI technology that generates natural dialogue with users and healthy meal menus. Specifically, it detects keywords from user dialogue data and generates appropriate responses and meal menus based on them.

[1359] server

[1360] The server is a computer system that executes the generative model and manages and analyzes user interaction data and health data. For example, the server analyzes the user's health status and notifies the user if an abnormality is detected.

[1361] Database

[1362] The database is a collection of data that stores user interaction history and health information and can be accessed as needed, enabling continuous health monitoring.

[1363] 2. System Operation

[1364] Start a conversation

[1365] When the user presses the "Start Conversation" button on the device, the device sends this request to the server, which uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[1366] Collection and analysis of everyday conversations

[1367] As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The device then presents the generated new conversation data to the user by displaying and playing it back on the panda character.

[1368] Health monitoring

[1369] The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities. For example, if a user says, "I've been having stomach problems lately," the server uses a generative model to generate detailed questions and present them to the user.

[1370] Anomaly detection and notification

[1371] If the server detects any abnormalities in health, it will send the information to family members or medical institutions in real time via a notification function. For example, if someone says "I have chest pain," a notification will be sent immediately to family members or emergency medical institutions, and an emergency call can even be made.

[1372] Meal menu suggestions and ordering

[1373] Based on the dialogue data generated based on the user's health status, the server proposes a meal menu suitable for the user. The user can select from the proposed menu and place an order through the terminal. The order details are sent to the server, which then notifies the meal service provider.

[1374] 3. Specific Examples

[1375] Scenario: A user complains of stomach upset.

[1376] 1. Start a conversation

[1377] Terminal: "Hello! How are you feeling today?" asks the panda character to the user.

[1378] User: "I've been having stomach problems lately."

[1379] Terminal: Converts the user's speech into text and sends it to the server.

[1380] 2. Conversation analysis and health assessment

[1381] Server: Analyzes the keyword "I've been having stomach problems lately," and based on this, the generative model generates a new dialogue: "That's worrying. What's the best thing to eat to ease your stomach?"

[1382] Server: Sends the generated dialogue to the terminal.

[1383] Device: A panda character will display and play a voice saying, "That's worrying. What is the best food to eat for your stomach?"

[1384] 3. Meal menu suggestions

[1385] Server: Based on the user's health condition, the server suggests meal menus such as "healthy soup with less carbonated drinks" or "nutritiously balanced porridge with plenty of vegetables."

[1386] Terminal: A panda character displays the suggested menu.

[1387] 4. Order Confirmation

[1388] User: Selects from the proposed menu and places an order through the terminal.

[1389] Server: Notifies the food service provider of the order.

[1390] Server: Sends a confirmation message to the terminal saying, "Your order has been confirmed. Delivery will be in 30 minutes."

[1391] Prompt Sentence Examples

[1392] You: I've been having stomach problems lately.

[1393] Panda: That's worrying. What's the easiest thing to eat for your stomach?

[1394] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1395] Step 1:

[1396] The user presses the "Start conversation" button on the device.

[1397] Input: User presses a button

[1398] Output: Sending a conversation initiation request to the server

[1399] Specific operation: The device sends a conversation start request to the server. The server receives this request, uses the generative model to generate an initial greeting message, and sends it to the device.

[1400] Step 2:

[1401] The device will display and play a voice message to the user through the panda character as a greeting message.

[1402] Input: Initial greeting message from the server

[1403] Output: Present natural dialogue to the user

[1404] Specific operation: The device receives a greeting message from the server and displays it as a panda character, playing it aloud.

[1405] Step 3:

[1406] The user responds to the panda character.

[1407] Input: User's voice response

[1408] Output: Audio data

[1409] Specific operation: The device receives the user's voice response and converts the voice into text.

[1410] Step 4:

[1411] The terminal transmits the user's text data to the server.

[1412] Input: Text data converted from audio

[1413] Output: Sending text data to the server

[1414] Specific operation: The device sends the text data converted from the voice to the server.

[1415] Step 5:

[1416] The server analyzes the text data and generates the next question or response.

[1417] Input: User's text data

[1418] Output: The next question or response generated

[1419] Specific operation: The server analyzes the received text data using a generative model and generates the next appropriate question or response.

[1420] Step 6:

[1421] The server transmits the generated dialogue to the terminal.

[1422] Input: Generated dialogue

[1423] Output: Send dialogue to terminal

[1424] Specific operation: The server sends the generated dialogue to the terminal.

[1425] Step 7:

[1426] The device displays and plays aloud the dialogue generated through the panda character to the user.

[1427] Input: Generated dialogue from the server

[1428] Output: Presents dialogue to the user

[1429] Specific operation: The terminal displays and plays aloud the dialogue received from the server using a panda character.

[1430] Step 8:

[1431] The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities.

[1432] Input: Interaction data

[1433] Output: Health status assessment results, abnormality detection results

[1434] Specific operation: The server uses the generative model to analyze the dialogue data and detect specific keywords and signs of anomalies.

[1435] Step 9:

[1436] The server will suggest an appropriate meal menu based on the health assessment results.

[1437] Input: Health status assessment results

[1438] Output: Suggested meal menu

[1439] Specific operation: The server uses the health status assessment results to suggest health-conscious meal menus through a generative model.

[1440] Step 10:

[1441] The terminal displays the suggested meal menu to the user, who places an order.

[1442] Input: Suggested meal menu, user selection

[1443] Output: Order data

[1444] Specific operation: The terminal displays the proposed meal menu and the user selects. The user's order is sent from the terminal to the server.

[1445] Step 11:

[1446] The server transmits the order data to the food service provider.

[1447] Input: Order data

[1448] Output: Notification to food service provider

[1449] Specific operation: The server notifies the meal service provider of the user's order data, confirms the order, and sends a confirmation message to the user's terminal.

[1450] Through the above processing steps, the user can order an appropriate meal menu while monitoring their health condition.

[1451] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1452] MODE FOR CARRYING OUT THE INVENTION

[1453] The present invention combines an emotion engine with an interactive medical AI service system for the elderly. Specific embodiments of the system are described below.

[1454] Components

[1455] 1. Devices: Electronic devices such as tablets and smartphones used by the elderly that provide an interface that allows users to converse with panda characters on a daily basis.

[1456] 2. Server: A computer system that executes the generative model, manages and analyzes the dialogue data with the user, and transmits the generated dialogue and health assessment data to the terminal.

[1457] 3. Generative Model: This is an artificial intelligence technique used to generate natural dialogue with the user, detect specific keywords, and assess health status based on them.

[1458] 4. Database: A collection of data that stores user interaction data and can be accessed as needed.

[1459] 5. Emotion engine: This engine recognizes emotions from the user's voice and text data and adjusts the dialogue with the user based on the recognized emotional data.

[1460] System Operation

[1461] 1. Initial Setup: When a user uses the service for the first time, they install the app on their device and enter their personal information (name, age, health information, etc.). This information is sent to the server and stored in a database.

[1462] 2. Start conversation: The user presses the "Start conversation" button on the device. The device sends this request to the server, which uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[1463] 3. Collection and analysis of everyday conversations: As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The device presents the generated new conversation data to the user by displaying and playing it back on the panda character.

[1464] 4. Health monitoring: The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities. Based on this, the generative model generates appropriate responses and questions. For example, if the user says, "My lower back hurts," the server instructs the generative model to ask more detailed questions about the extent and location of the pain.

[1465] 5. Emotion recognition and adjustment: The device sends the user's voice and facial expression data to the emotion engine, which then recognizes the user's emotions. For example, if the user has an anxious expression or voice, the emotion engine sends that information to the server.

[1466] 6. Emotion-based dialogue adjustment: The server analyzes the emotion data received from the emotion engine and generates appropriate dialogue based on the emotion. For example, if the user is anxious, the panda character will generate a dialogue such as, "Is there anything you're worried about? Please tell me."

[1467] 7. Abnormality detection and notification: If the server detects an abnormality in health status, the notification function will send the information to family members and medical institutions in real time. For example, if someone says, "I have chest pain," a notification will be sent immediately to family members and emergency medical institutions, and if necessary, an emergency call will be made.

[1468] 8. Report Generation: The server periodically generates a report summarizing the user's health status and emails it to designated family members or healthcare providers. The report includes a summary of the conversation, any abnormalities detected, and recommended next steps.

[1469] Specific examples

[1470] Scenario: An elderly person complains of back pain and becomes emotionally unstable.

[1471] 1. Start a conversation

[1472] Terminal: "Hello! How are you today?" asks the panda character to the user.

[1473] User: "I've been having back pain lately..."

[1474] Terminal: Converts the user's speech into text and sends it to the server.

[1475] 2. Conversation Analysis and Anomaly Detection

[1476] Server: Analyzes the keyword "my lower back hurts," and based on this, the generative model generates a new dialogue: "That's worrying. Where does it hurt?"

[1477] Server: Sends the generated dialogue to the terminal.

[1478] Device: Panda will say and listen, "That's worrying. Where does it hurt?"

[1479] 3. Emotional Recognition

[1480] Device: The user answers in an anxious voice, "My whole lower back hurts."

[1481] Device: Sends voice data to the emotion engine, which detects anxiety.

[1482] Server: Receives emotion data from the emotion engine and generates dialogue based on it.

[1483] 4. Adjusting dialogue based on emotions

[1484] Server: Based on the information from the emotion engine, it generates the dialogue, "You're okay. Take a rest and don't push yourself too hard. Is there anything I can help you with?"

[1485] Server: Sends the generated dialogue to the terminal.

[1486] Device: Panda will say, "It's okay. Take care and rest. Is there anything I can help you with?"

[1487] This invention allows elderly people to enjoy daily conversations with panda characters while simultaneously managing their health. It also responds sensitively to the user's emotions and provides appropriate dialogue, providing a greater sense of security. Family members and medical institutions can monitor the elderly's health status in real time, contributing to improving the quality of life for the elderly.

[1488] The processing flow will be explained below.

[1489] Step 1:

[1490] Device: Display the app installation screen.

[1491] User: Installs and launches the app.

[1492] Step 2:

[1493] Device: Display the initial setup screen (name, age, health information, etc.).

[1494] User: Enter the required information and press the send button.

[1495] Terminal: Sends the entered information to the server.

[1496] Server: Stores the received information in a database and sends a response to the terminal indicating that initial setup is complete.

[1497] Terminal: Notify the user that the initial setup is complete.

[1498] Step 3:

[1499] Device: Display a "Start Conversation" button on the home screen.

[1500] User: Press the start conversation button.

[1501] Terminal: Sends a request to start a conversation to the server.

[1502] Step 4:

[1503] Server: Initiates the conversation session and generates the initial greeting message.

[1504] Server: Sends a greeting message to the terminal.

[1505] On your device: A panda character will display and / or play a greeting message (e.g., "Hello! How was your day today?").

[1506] Step 5:

[1507] User: Answer the panda character's questions (e.g., "It was nice weather today so I went for a walk").

[1508] Terminal: Converts the user's speech into text and sends it to the server.

[1509] Step 6:

[1510] Server: Analyzes the received text data and generates the next question or response.

[1511] Server: Sends the generated conversation data to the terminal.

[1512] On your device: The panda character will display and play your new message (e.g., "That's lovely! Which way did you walk?").

[1513] Step 7:

[1514] Server: Continuously analyzes conversation data and detects keywords and anomalies.

[1515] Server: Generates new responses and questions based on the detected information.

[1516] Device: View and play new questions and responses from the panda character.

[1517] Step 8:

[1518] User: Provides health information as the conversation continues (e.g., "I've been having a lot of back pain lately...").

[1519] Terminal: Converts the user's speech into text and sends it to the server.

[1520] Step 9:

[1521] Server: Analyzes keywords that indicate abnormal health conditions, such as "lower back pain," and detects abnormalities.

[1522] Server: If an abnormality is detected, it generates a notification to notify family members or medical institutions.

[1523] Server: Send notifications to family members or medical institutions (e.g., "You've recently been complaining of back pain. You may need further evaluation.").

[1524] Step 10:

[1525] Strategic terminal: Sends the user's voice and facial expression data to the emotion engine.

[1526] Emotion engine: Recognizes user emotions and detects emotions such as anxiety.

[1527] Emotion Engine: Sends detected emotion data to the server.

[1528] Step 11:

[1529] Server: Analyzes the received emotion data and generates appropriate dialogue based on the emotion.

[1530] Server: Sends the generated dialogue to the terminal.

[1531] On your device: The panda character will display and / or play a sensitive message (e.g., "It's okay. Take care and rest. Is there anything I can help you with?").

[1532] Step 12:

[1533] Server: Periodically aggregates conversation data and generates a health status report for the elderly.

[1534] Server: Sends the generated report to the family or medical institution.

[1535] Step 13:

[1536] Device: Continue everyday conversations with panda characters.

[1537] User: Continue to provide information about health status through interactions with the panda character.

[1538] Example 2

[1539] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1540] Elderly people need to be sensitive to changes in their health and emotions in their daily lives. However, it is often difficult for them to visit medical institutions or communicate with specialists on a daily basis. For this reason, there is a need for a system that allows elderly people to easily manage their health at home and receive appropriate support in response to emotional changes.

[1541] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1542] In this invention, the server includes means for analyzing the dialogue data to evaluate the health condition, means for notifying when an abnormality in the health condition is detected, and means for adjusting the dialogue based on the emotion data, thereby enabling the user to manage their health condition while at home and receive appropriate dialogue in response to changes in their emotions.

[1543] "Communication devices" are electronic devices such as tablets and smartphones used by elderly people, and are devices that provide an interface for users to interact with each other on a daily basis.

[1544] An "information processing device" is a computer system that executes generative models and manages and analyzes user interaction data.

[1545] A "storage device" is a collection of data that stores user interaction data and can be accessed as needed.

[1546] A "generative model" is an artificial intelligence technique used to generate natural-sounding dialogue with a user.

[1547] An "emotion recognition engine" is an engine that recognizes emotions from the user's voice and text data and adjusts the dialogue based on that information.

[1548] "Dialogue data" refers to data such as messages and voice recordings exchanged between a user and a communication device.

[1549] "Health status assessment" is the process of analyzing dialogue data and determining the user's health status.

[1550] "Notification means" is a function that sends information to family members or medical institutions when an abnormality in health condition is detected.

[1551] "Emotion data" is data that indicates the emotional state of the user analyzed by the emotion recognition engine.

[1552] "Dialogue adjustment" is the process of modifying the dialogue content and providing an appropriate response to the user based on emotion data.

[1553] This invention relates to an interactive medical support system for the elderly, and in particular to a system that utilizes a generative AI model with emotion recognition capabilities. The specific operation and usage of the system are described below.

[1554] System Overview

[1555] The system includes a communication device (e.g., a tablet or smartphone) used by the elderly, an information processing device that executes a generative model, and a storage device that manages interaction data with the user. The system includes the following means:

[1556] A means for communication devices to present natural dialogue to users

[1557] The means by which the generative model generates interactions with the user

[1558] A means for an information processing device to analyze the dialogue data and evaluate the health condition

[1559] A means for notifying when an information processing device detects an abnormality in health status

[1560] A means for communication devices to send user voice and facial expression data to an emotion recognition engine

[1561] The emotion recognition engine analyzes the user's emotions and transmits the results to the information processing device.

[1562] A means for an information processing device to adjust a dialogue based on emotion data

[1563] Initial Setup

[1564] When a user uses the system for the first time, they install a dedicated app on their communication device and enter personal information such as their name, age, and health information, which is then sent to the server and stored in a database.

[1565] Start a conversation

[1566] When the user presses the "Start Conversation" button on the communication device, the communication device sends this request to the server. The server uses the generative model to generate an initial greeting message and sends it to the communication device. The communication device then displays and plays the greeting message aloud using a panda character.

[1567] Collection and analysis of everyday conversations

[1568] As the user engages in everyday conversation with the panda character, the communication device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The generated dialogue is sent to the communication device, where it is displayed and played aloud by the panda character.

[1569] Health monitoring

[1570] The server continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. For example, if a user says, "My lower back hurts," the server instructs the generative model to ask more detailed questions about the extent and location of the pain.

[1571] Emotional awareness and regulation

[1572] The communication device sends the user's voice and facial expression data to an emotion recognition engine, which then analyzes the user's emotions. For example, if the user has an anxious expression or voice, that information is sent to the server. The server analyzes the emotional data, generates appropriate dialogue based on the emotion, and sends it to the communication device.

[1573] Health Assessment and Notification

[1574] If the server detects any abnormalities in a person's health, it will notify family members or medical institutions in real time. For example, if someone says, "I have chest pain," a notification will be sent immediately, and an emergency call will be made if necessary.

[1575] Report Generation

[1576] The server periodically generates a report summarizing the user's health status and emails it to designated family members or healthcare providers, including a summary of the conversation, any abnormalities detected, and recommended next steps.

[1577] Specific examples

[1578] Scenario: An elderly person complains of back pain and becomes emotionally unstable.

[1579] 1. Start a conversation

[1580] Terminal: "Hello! How are you today?" asks the panda character to the user.

[1581] User: "I've been having back pain lately..."

[1582] Terminal: Converts the user's speech into text and sends it to the server.

[1583] 2. Conversation Analysis and Anomaly Detection

[1584] Server: Analyzes the keyword "my lower back hurts," and based on this, the generative model generates a new dialogue: "That's worrying. Where does it hurt?"

[1585] Server: Sends the generated dialogue to the terminal.

[1586] Device: A panda character will display and play a voice message saying, "That's worrying. Where does it hurt?"

[1587] 3. Emotional Recognition

[1588] User: In an anxious voice, replies, "My lower back hurts all over."

[1589] Device: Sends voice data to the emotion engine to detect anxiety.

[1590] Server: Receives the emotion data and generates a dialogue such as, "You're okay. Take care and rest. Is there anything I can help you with?"

[1591] 4. Coordinating dialogue

[1592] Server: Sends the generated dialogue to the terminal.

[1593] Device: A panda character will display and play a voice saying, "It's okay. Please take it easy and rest. Is there anything I can help you with?"

[1594] Prompt Sentence Examples

[1595] "The user complains of back pain and appears anxious. Based on this, please generate a dialogue that will reassure the user."

[1596] This invention allows elderly people to manage their health through the panda character in their daily lives and provides appropriate dialogue, allowing them to spend their days with peace of mind.Furthermore, family members and medical institutions can monitor their health status in real time, contributing to improving the quality of life for elderly people.

[1597] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1598] Step 1: Initial Setup

[1599] User: Installs the app on the device and enters personal information (name, age, health information, etc.) through the app's input form.

[1600] Terminal: Receives input personal information and converts it into JSON format to send to the server. The input is personal information, and the output is the converted JSON data.

[1601] Server: Analyzes the received JSON data and saves the user information in a database. This allows basic user information to be managed on the server side.

[1602] Step 2: Start a conversation

[1603] User: Presses the "Start conversation" button on the device. Input is the action of clicking the "Start conversation" button.

[1604] Terminal: Generates a conversation initiation request and sends it to the server. The output is the generated request data.

[1605] Server: Receives the request and generates the initial greeting message using the generative model. The input is the request and the output is the greeting message.

[1606] Server: Sends the generated greeting message to the terminal. The output is the generated message data.

[1607] Terminal: The panda character displays and plays audibly the received greeting message. The input is the greeting message, and the output is the display and audio playback for the user.

[1608] Step 3: Collecting and analyzing everyday conversations

[1609] User: Have everyday conversations with a panda character. Input is via user voice and text.

[1610] Terminal: Converts user speech from voice to text. This conversion is done by speech recognition software. The input is the user's voice, and the output is text data.

[1611] Terminal: Transmits converted text data to the server. The output is text data.

[1612] Server: Analyzes the received text data and generates the next question or response. This analysis is performed using a natural language processing model. The input is text data, and the output is a new dialogue.

[1613] Server: Sends the generated dialogue to the terminal. The output is dialogue data.

[1614] Terminal: The panda character displays and plays audibly the received dialogue. The input is the dialogue, and the output is a display and audible playback for the user.

[1615] Step 4: Monitor your health

[1616] Server: Continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. A keyword extraction algorithm is used for analysis. The input is dialogue data, and the output is health assessment data.

[1617] Server: When an anomaly is detected, it uses a generative model to generate appropriate questions and responses. The input is the anomaly detection result, and the output is the new questions and responses.

[1618] Server: Sends the generated dialogue to the terminal. The output is dialogue data.

[1619] Terminal: The panda character displays and plays audibly the received dialogue. The input is the dialogue, and the output is a display and audible playback for the user.

[1620] Step 5: Recognize and regulate your emotions

[1621] Terminal: Sends the user's voice and facial expression data to the emotion recognition engine. This transmission is done in real time. The input is voice and facial expression data, and the output is the transmitted data.

[1622] Emotion recognition engine: Analyzes the user's emotions and sends the results to the server. The input is voice and facial expression data, and the output is emotional data.

[1623] Server: Receives emotional data and adjusts the dialogue based on the analysis results. The input is emotional data, and the output is the adjusted dialogue.

[1624] Server: Sends the adjusted dialogue to the terminal. The output is dialogue data.

[1625] Terminal: The panda character displays and plays audibly the received dialogue. The input is the dialogue, and the output is a display and audible playback for the user.

[1626] Step 6: Health Assessment and Notification

[1627] Server: When an abnormality in health status is detected, the notification function is activated. This function includes push notifications and email notifications. The input is the abnormality detection result, and the output is the notification data.

[1628] Server: Notifies family and medical institutions of health status in real time. Makes emergency calls if necessary. Input is notification data, output is notification and alert.

[1629] Step 7: Generate reports

[1630] Server: Periodically generates a report summarizing the user's health status. This report includes a summary of the conversation, any abnormalities detected, and recommended next steps. The input is the conversation data and health assessment data, and the output is the report.

[1631] Server: Sends the generated report to the designated family or medical institution by email. The input is the report data, and the output is the sent report.

[1632] (Application example 2)

[1633] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1634] Health management, especially for the elderly, is an important issue in modern society. Elderly people require daily health monitoring, and prompt action is required when abnormalities occur. However, conventional systems do not provide natural dialogue and are difficult to respond to, taking into account the user's emotions. Therefore, there is a need for a system that combines natural dialogue and emotion recognition that can be used comfortably by the elderly.

[1635] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for analyzing dialogue data and evaluating health status; means for the server to notify when it detects an abnormality in health status; means for the terminal and the server to recognize emotions and adjust the dialogue based on the emotions; means for the terminal to check the health status of residents of the welfare facility through everyday conversation; and means for the server to periodically generate health status reports and provide them to a specified contact. This enables elderly people to manage their health with a sense of security through more natural dialogue.

[1636] The term "elderly" refers to people over a certain age, and in healthcare and welfare in particular, generally refers to people aged 65 and over.

[1637] A "terminal" is an electronic device used by a user, which inputs and outputs data via an interface.

[1638] A "generative model" is an algorithm or program that uses artificial intelligence technology to generate natural-sounding dialogue.

[1639] A "server" is a computer system that manages data and programs and provides services to terminals via a network.

[1640] A "database" is a collection of data that stores user interaction data and health information and can be accessed as needed.

[1641] "Emotions" refer to psychological states such as joy, sadness, and anxiety that humans experience, and can be read from voice and facial expressions.

[1642] A "dialogue" is a written communication between a user and a terminal.

[1643] "Abnormal" refers to a state that deviates from normal health and requires immediate action.

[1644] "Notification" is the act of communicating specific information to designated recipients in real time.

[1645] A "welfare facility" refers to a dedicated facility where elderly people, people with disabilities, etc. can live while receiving support and care.

[1646] MODE FOR CARRYING OUT THE INVENTION

[1647] This invention describes an embodiment of an interactive medical AI service system for elderly people specifically implemented in a welfare facility. The components and operation of the system are described in detail below.

[1648] System Configuration

[1649] 1. Devices: Electronic devices such as tablets and smartphones used by the elderly. Devices provide an interface with users and have the ability to display and play natural dialogue.

[1650] 2. Generative model: An artificial intelligence algorithm that uses natural language processing to generate dialogue with the user. Specifically, it incorporates Spacy and BERT.

[1651] 3. Server: A computer system that manages the database and generative model and sends dialogues and notifications to devices via a network. The server is built using Python or Flask and is responsible for data analysis and dialogue generation.

[1652] 4. Emotion Engine: An algorithm that recognizes emotions from the user's voice and text data and adjusts the dialogue accordingly. The EmotionEngine class is an example of this.

[1653] 5. Database: A system for storing conversation data and health information. MySQL, PostgreSQL, etc. are used.

[1654] Operation overview

[1655] 1. Initial setup: The server stores the user's basic information (name, age, health information, etc.) in a database. During the initial setup, the user installs the app on their device and enters the required information.

[1656] 2. Start of conversation: When the user presses the "Start conversation" button on the device, the server uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[1657] 3. Collection and analysis of everyday conversations: As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. For example, if the user says, "I've been having a pain in my lower back lately...", the generative model will generate a response such as, "That's worrying. Where does it hurt?"

[1658] 4. Health status monitoring: The server continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. Based on this, the generative model generates appropriate responses and questions. For example, if the keyword "chest pain" is detected, this information will be promptly notified to family members or medical institutions.

[1659] 5. Emotion recognition and adjustment: The device sends the user's voice and facial expression data to the emotion engine, which then recognizes the user's emotions. For example, if the user's facial expression or voice looks anxious, the emotion engine detects the anxiety and sends this information to the server. The server analyzes the emotion data and generates a dialogue based on the emotion.

[1660] 6. Periodic Report Generation: The server periodically generates a report summarizing the user's health status and provides it to designated contacts (family members or healthcare providers), including a summary of the conversation, any abnormalities detected, and recommended next steps.

[1661] Specific examples

[1662] Scenario: An elderly person complains of lower back pain

[1663] 1. Start a conversation

[1664] User: "I've been having back pain lately..."

[1665] System: "That's concerning. Where does it hurt?"

[1666] 2. Emotional Recognition

[1667] User: "My lower back hurts all over."

[1668] System: The emotion engine detects the user's anxiety and generates a response such as, "You're OK. Take care and get some rest. Is there anything I can help you with?"

[1669] With these actions, elderly people can manage their health with peace of mind through everyday conversation. In addition, if any abnormalities occur, family members and medical institutions will be notified promptly, allowing them to live their lives with peace of mind.

[1670] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1671] Step 1: Initial Setup

[1672] Users install a medical app for the elderly on their device and enter their personal information (name, age, health information, etc.).

[1673] Input: User personal information

[1674] Processing: The device sends this information to the server, which stores it in a database.

[1675] Output: User information stored in the database

[1676] Step 2: Start a conversation

[1677] The user presses the "Start conversation" button on the device.

[1678] Input: User operation (pressing the "Start conversation" button)

[1679] Processing: The server uses the generative model to generate an initial greeting message and sends it to the terminal.

[1680] Output: A greeting message displayed on the terminal (natural dialogue)

[1681] Step 3: Collecting and analyzing everyday conversations

[1682] Users can converse with a panda character.

[1683] Input: User's voice data

[1684] Processing: The device converts the voice data into text and sends it to the server. The server analyzes the received text, and the generative model generates the next question or response. The server then sends the generated dialogue back to the device.

[1685] Output: The new dialogue displayed on the terminal.

[1686] Step 4: Monitor your health

[1687] The server analyzes the conversation data and detects specific keywords and signs of abnormalities.

[1688] Input: User's text data

[1689] Processing: The server detects specific keywords (e.g., "my lower back hurts") and evaluates the health status. If an abnormality is detected, it instructs the generative model on the next course of action.

[1690] Output: The next dialogue generated, and the anomaly detection results.

[1691] Step 5: Recognize and regulate your emotions

[1692] The device sends the user's voice and facial expression data to the emotion engine.

[1693] Input: User's voice and facial expression data

[1694] Processing: The emotion engine analyzes these data to recognize emotions and sends the results to the server. The server analyzes the emotion data, and the generative model generates appropriate dialogue based on the emotion.

[1695] Output: Generated dialogue (content corresponding to emotions)

[1696] Step 6: Anomaly detection and notification

[1697] If the server detects any abnormalities in health, it will use the notification function to send information to family members or medical institutions in real time.

[1698] Input: Anomaly detection results

[1699] Processing: The server generates a notification and sends it to the specified contacts, including calling emergency services if necessary.

[1700] Output: Notification sent to family and healthcare providers

[1701] Step 7: Generate reports

[1702] The server periodically generates a report summarizing the user's health status and emails it to designated contacts.

[1703] Input: Dialogue data and emotion data

[1704] Processing: The server compiles and analyzes the data, generates a report, and sends the report to the designated contact.

[1705] Output: Health report sent to family and healthcare providers

[1706] Through the above processing steps, elderly people can manage their health through natural dialogue and emotion recognition, and can live with peace of mind even in welfare facilities.

[1707] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1708] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1709] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1710] [Fourth embodiment]

[1711] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1712] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1713] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1714] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1715] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1716] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1717] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1718] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1719] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1720] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1721] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1722] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1723] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1724] MODE FOR CARRYING OUT THE INVENTION

[1725] The present invention provides an interactive medical AI service system for elderly people. Specific embodiments of the system are described below.

[1726] Components

[1727] 1. Devices: Electronic devices such as tablets and smartphones used by the elderly. They provide an interface that allows users to converse with panda characters on a daily basis.

[1728] 2. Server: A computer system that executes the generative model, manages and analyzes the dialogue data with the user, and transmits the generated dialogue and health assessment data to the terminal.

[1729] 3. Generative Model: This is an artificial intelligence technique used to generate natural dialogue with the user, detect specific keywords, and assess health status based on them.

[1730] 4. Database: A collection of data that stores user interaction data and can be accessed as needed.

[1731] System Operation

[1732] 1. Initial Setup: When a user uses the service for the first time, they install the app on their device and enter their personal information (name, age, health information, etc.). This information is sent to the server and stored in a database.

[1733] 2. Start conversation: The user presses the "Start conversation" button on the device. The device sends this request to the server, which uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[1734] 3. Collection and analysis of everyday conversations: As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The device presents the generated new conversation data to the user by displaying and playing it back on the panda character.

[1735] 4. Health monitoring: The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities. Based on this, the generative model generates appropriate responses and questions. For example, if the user says, "My lower back hurts," the server instructs the generative model to ask more detailed questions about the extent and location of the pain.

[1736] 5. Abnormality detection and notification: If the server detects an abnormality in health status, the notification function will send the information to family members and medical institutions in real time. For example, if someone says, "I have chest pain," a notification will be sent immediately to family members and emergency medical institutions, and if necessary, an emergency call will be made.

[1737] 6. Report Generation: The server periodically generates a report summarizing the user's health status and emails it to designated family members or healthcare providers. The report includes a summary of the conversation, any abnormalities detected, and recommended next steps.

[1738] Specific examples

[1739] Scenario: Conversation with an elderly person complaining of lower back pain

[1740] 1. Start a conversation

[1741] Terminal: "Hello! How are you today?" asks the panda character to the user.

[1742] User: "I've been having back pain lately..."

[1743] Terminal: Converts the user's speech into text and sends it to the server.

[1744] 2. Conversation Analysis and Anomaly Detection

[1745] Server: Analyzes the keyword "my lower back hurts," and based on this, the generative model generates a new dialogue: "That's worrying. Where does it hurt?"

[1746] Server: Sends the generated dialogue to the terminal.

[1747] Device: Panda will say and listen, "That's worrying. Where does it hurt?"

[1748] 3. Notifications and Reporting

[1749] Server: Detects abnormalities based on data related to lower back pain and sends a notification to the family member stating, "You have recently been complaining of lower back pain. A more detailed diagnosis may be necessary."

[1750] Server: Generates periodic health status reports and sends them to family members.

[1751] This invention allows elderly people to enjoy daily conversations with panda characters while simultaneously managing their health.Furthermore, family members and medical institutions can monitor their health status in real time, contributing to improving the quality of life for elderly people.

[1752] The processing flow will be explained below.

[1753] Step 1:

[1754] Device: Display the app installation screen.

[1755] User: Installs and launches the app.

[1756] Step 2:

[1757] Device: Display the initial setup screen (name, age, health information, etc.).

[1758] User: Enter the required information and press the send button.

[1759] Terminal: Sends the entered information to the server.

[1760] Server: Stores the received information in a database and sends a response to the terminal indicating that initial setup is complete.

[1761] Terminal: Notify the user that the initial setup is complete.

[1762] Step 3:

[1763] Device: Display a "Start Conversation" button on the home screen.

[1764] User: Press the start conversation button.

[1765] Terminal: Sends a request to start a conversation to the server.

[1766] Step 4:

[1767] Server: Initiates the conversation session and generates the initial greeting message.

[1768] Server: Sends a greeting message to the terminal.

[1769] On your device: A panda character will display and / or play a greeting message (e.g., "Hello! How was your day today?").

[1770] Step 5:

[1771] User: Answer the panda character's questions (e.g., "It was nice weather today so I went for a walk").

[1772] Terminal: Converts the user's speech into text and sends it to the server.

[1773] Step 6:

[1774] Server: Analyzes the received text data and generates the next question or response.

[1775] Server: Sends the generated conversation data to the terminal.

[1776] On your device: The panda character will display and play your new message (e.g., "That's lovely! Which way did you walk?").

[1777] Step 7:

[1778] Server: Continuously analyzes conversation data and detects keywords and anomalies.

[1779] Server: Generates new responses and questions based on the detected information.

[1780] Device: View and play new questions and responses from the panda character.

[1781] Step 8:

[1782] User: Provides health information as the conversation continues (e.g., "I've been having a lot of back pain lately...").

[1783] Terminal: Converts the user's speech into text and sends it to the server.

[1784] Step 9:

[1785] Server: Analyzes keywords that indicate abnormal health conditions, such as "lower back pain," and detects abnormalities.

[1786] Server: If an abnormality is detected, it generates a notification to notify family members or medical institutions.

[1787] Server: Send notifications to family members or medical institutions (e.g., "You've recently been complaining of back pain. You may need further evaluation.").

[1788] Step 10:

[1789] Server: Periodically aggregates conversation data and generates a health status report for the elderly.

[1790] Server: Sends the generated report to the family or medical institution.

[1791] Step 11:

[1792] Device: Continue everyday conversations with panda characters.

[1793] User: Continue to provide information about health status through interactions with the panda character.

[1794] Example 1

[1795] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1796] It is difficult for elderly people to manage their own health status in their daily lives and detect abnormalities early. It is also difficult for family members and medical institutions to grasp the health status of elderly people in real time. This increases health risks, especially for elderly people living alone. Furthermore, there is currently a lack of systems that assess the health status of elderly people through natural dialogue.

[1797] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1798] In this invention, the server includes a means for analyzing the dialogue data to evaluate the health condition, a means for notifying when an abnormality in the health condition is detected, and a means for continuously monitoring the health condition based on the dialogue data. This allows the user to manage their own health condition through natural dialogue, and enables family members and medical institutions to understand the health condition in real time.

[1799] "Elderly" refers to users who are aging and who are particularly in need of medical and nursing care support.

[1800] An "information terminal" is an electronic device used by a user, and refers to a device that has the shape of a tablet, smartphone, etc.

[1801] A "natural language generation model" is an artificial intelligence technology used to generate natural dialogue with users, and specifically refers to a computer program that performs language processing.

[1802] "Computer system" refers generally to a computer system for executing generative models and analyzing and managing interaction data.

[1803] "Data Collection" refers to a database system that stores user interaction data and makes it accessible as needed.

[1804] "Natural dialogue" refers to text that is generated naturally in interaction with a user, just like human language.

[1805] "Interaction Data" refers to data including voice, text, and other related information relating to a conversation between a user and a system.

[1806] "Health status" refers to the state of the user's physical and psychological health, and specifically includes physical condition, presence or absence of pain, mental state, etc.

[1807] "Abnormal" refers to any symptom or sign that deviates from normal health, especially one that requires medical attention.

[1808] "Notification" refers to the process of communicating information about an abnormality in a user's health condition to family members or medical institutions.

[1809] "Monitoring" refers to the process of continuously observing a user's health status and assessing it based on specific conditions.

[1810] MODE FOR CARRYING OUT THE INVENTION

[1811] The present invention provides an interactive medical AI service system for elderly people. Specific embodiments of the system are described below.

[1812] Hardware and Software Configuration

[1813] Device:

[1814] This refers to tablets and smartphones that are used daily by seniors, including iPads, Android tablets, iPhones, and Android smartphones. These devices have dedicated applications installed.

[1815] server:

[1816] The server is a computing system equipped with a high-performance CPU, memory, and an NVIDIA GPU specialized for running generative models. This is hardware for efficiently running generative models on the server side.

[1817] Generative model:

[1818] Generative models are natural language processing technologies, including OpenAI's GPT series and Google's BERT model, that are used to generate natural dialogue with users.

[1819] Database:

[1820] This is a database system for managing dialogue data, and uses database engines such as MySQL or MongoDB.

[1821] Program processing

[1822] 1. Initial Setup:

[1823] When a user uses the system for the first time, they launch the app installed on their device and enter personal information such as their name, age, and health information. This information is sent from the device to the server, which then stores the received information in a database.

[1824] 2. Start a conversation:

[1825] The user presses the "Start Conversation" button on the device app. This request is sent from the device to the server, and the server uses the generative model to generate an initial greeting message. The generated greeting message is then sent to the device, where the panda character displays and plays the greeting message.

[1826] 3. Collection and analysis of everyday conversations:

[1827] As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server (using the Google Cloud Speech-to-Text API). The server analyzes the received text data and generates the next question or response. The generated new conversation data is sent to the device and displayed and played back by the panda character.

[1828] 4. Health monitoring:

[1829] The server continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. Based on this information, the generative model generates appropriate responses and questions. For example, if a user says, "My lower back hurts," the generative model creates a question like, "That's worrying. Where does it hurt?"

[1830] 5. Anomaly detection and notification:

[1831] If the server detects any abnormalities in the user's health, it will use Twilio's SMS API to send a notification to family members or medical institutions. For example, if the user says something like "I have chest pain," it will call an emergency number.

[1832] 6. Report Generation:

[1833] The server periodically generates a report on the user's health status and emails it to designated family members or healthcare providers using SendGrid, including a summary of the conversation, any anomalies detected, and recommended next steps.

[1834] Specific examples

[1835] Scenario: An elderly person complains of back pain.

[1836] 1. Start the conversation:

[1837] Terminal: "Hello! How are you today?" asks the panda character to the user.

[1838] User: "I've been having back pain lately..."

[1839] Terminal: Converts speech into text and sends it to the server.

[1840] 2. Conversation analysis and anomaly detection:

[1841] Server: Analyzes the keyword "my lower back hurts" and uses a generative model (GPT-3) to generate a new dialogue: "That's worrying. Where does it hurt?"

[1842] Server: Sends the generated dialogue to the terminal.

[1843] Device: The new dialogue will be displayed and played by the panda character.

[1844] 3. Notifications and Reporting:

[1845] Server: Analyzes data related to lower back pain and detects abnormalities.

[1846] Server: Send a notification to the family member via Twilio's SMS API saying, "You've been complaining of back pain recently. A more detailed diagnosis may be needed."

[1847] Server: Generates periodic health status reports and sends them to family members using SendGrid.

[1848] This system allows elderly people to enjoy daily interactions with panda characters while simultaneously managing their own health conditions, allowing their families and medical institutions to monitor their health conditions in real time and contributing to improving the quality of life of the elderly.

[1849] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1850] Step 1: Initial Setup

[1851] Input: A user installs the app on their tablet or smartphone and enters personal information such as their name, age, and health information.

[1852] Specific behavior:

[1853] The user launches the app and enters their personal information on the "New Registration" screen.

[1854] The terminal sends the entered information to the server in JSON format.

[1855] Data processing:

[1856] The server parses the received JSON data, converts it into the required format, and saves it in the database.

[1857] Output: Personal information is stored on the server and recorded in a database.

[1858] Step 2: Start a conversation

[1859] Input: The user presses the "Start conversation" button in the device app.

[1860] Specific behavior:

[1861] When the user presses the "Start conversation" button, the device sends an HTTP request to the server.

[1862] The server receives the HTTP request and inputs the initial prompt "greet the user" into the generative model.

[1863] Data Calculation:

[1864] The generative model generates an initial greeting message based on the prompt sentence.

[1865] Output: The generated greeting message is sent back to the device, and the panda character displays and plays "Hello! How are you today?"

[1866] Step 3: Collecting and analyzing everyday conversations

[1867] Input: The user continues to talk to the panda character.

[1868] Specific behavior:

[1869] When a user says, "My back has been hurting lately...", the device uses the Google Cloud Speech-to-Text API to convert the speech to text.

[1870] The terminal transmits the converted text data to the server.

[1871] Data Calculation:

[1872] The server analyzes the text data and detects the keyword "my lower back hurts."

[1873] The server inputs the prompt statement "If the user says that their lower back hurts, generate a response" into the generative model.

[1874] Output: The generative model generates a response saying, "That's worrying. Where does it hurt?" and sends it to the device.

[1875] The panda character displays new dialogue and plays it aloud: "That's worrying. Where does it hurt?"

[1876] Step 4: Monitor your health

[1877] Input: Continuous conversation data between the user and the panda character

[1878] Specific behavior:

[1879] The server analyzes all received interaction data in real time.

[1880] To detect specific keywords or signs of anomalies, the server inputs the prompt statement "Generate a new question containing specific keywords" into the generative model.

[1881] Data Calculation:

[1882] The generative model generates appropriate responses and questions to prepare for the next dialogue.

[1883] Output: The appropriate questions and responses are generated and sent to the device.

[1884] A panda character presents the generated questions and responses to the user.

[1885] Step 5: Anomaly detection and notification

[1886] Input: Anomaly information based on conversation data analyzed by the server

[1887] Specific behavior:

[1888] If the server detects any abnormalities in health status, it will use Twilio's SMS API to send notifications to family members or medical institutions.

[1889] For example, if a serious abnormality such as "chest pain" is detected, the server will make an emergency call.

[1890] Data Calculation:

[1891] Generate a notification message containing the anomaly information.

[1892] Output: A notification message is sent to family members and healthcare providers.

[1893] Step 6: Generate reports

[1894] Input: Interaction data periodically collected by the server

[1895] Specific behavior:

[1896] The server aggregates past interaction data and generates periodic health status reports.

[1897] The report includes a summary of the conversation, any anomalies detected, and recommended next steps.

[1898] Data Calculation:

[1899] Generate scheduled reports and convert them into email format.

[1900] Output: The report is sent to the designated family member or healthcare provider using SendGrid.

[1901] (Application example 1)

[1902] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1903] Since there is no system that simultaneously supports the health management and meal selection of the elderly, there is a need for a system that can grasp the health status of the elderly and suggest and order appropriate meals. The lack of such a system has led to issues such as the deterioration of the health status of the elderly and the inconvenience of meal selection.

[1904] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1905] In this invention, the server includes a means for proposing an appropriate meal menu based on the dialogue data and the user's health condition, a means for analyzing the dialogue data and evaluating the user's health condition, and a means for notifying the user and accepting an order if an abnormality in the user's health condition or a meal menu is detected. This enables meal suggestions that take the user's health condition into consideration and easy ordering.

[1906] "Devices used by the elderly" refers to electronic devices operated by the elderly, such as smartphones and tablets.

[1907] A "generative model" is an artificial intelligence technology that generates natural dialogue with users and healthy meal menus.

[1908] The "server" is a computer system that executes the generative model and manages and analyzes user interaction data and health data.

[1909] "User dialogue data" refers to data that records and manages the content of conversations with users.

[1910] The "means for proposing an appropriate meal menu based on health condition" is a means having a function of analyzing the user's health condition and presenting a meal menu suitable for that condition.

[1911] A "health status management database" is a collection of data that stores a user's health information and interaction history and can be accessed as needed.

[1912] The "means for presenting natural dialogue" is a function for displaying natural and easy-to-understand dialogue to the user.

[1913] The "means for proposing healthy meal menus" is a function for proposing healthy meal menus based on the user's health condition and dialogue data.

[1914] The "means for analyzing dialogue data and evaluating health status" is a function for analyzing dialogue data with the user and evaluating the health status.

[1915] "Means for notifying and accepting orders when abnormalities in health status or meal menus are detected" is a function that notifies and accepts orders when abnormalities in the user's health status occur or when selecting or ordering a meal menu.

[1916] This invention is an interactive system that supports elderly people in managing their health and choosing meals simultaneously. The system includes a terminal operated by the elderly, a server that executes generative models, and a database that manages dialogue data and health status.

[1917] 1. System Components

[1918] Devices used by the elderly

[1919] The devices are electronic devices operated by elderly people, such as smartphones and tablets. Through these devices, users can interact with a panda character and receive health assessments and meal menu suggestions.

[1920] Generative Model

[1921] A generative model is an AI technology that generates natural dialogue with users and healthy meal menus. Specifically, it detects keywords from user dialogue data and generates appropriate responses and meal menus based on them.

[1922] server

[1923] The server is a computer system that executes the generative model and manages and analyzes user interaction data and health data. For example, the server analyzes the user's health status and notifies the user if an abnormality is detected.

[1924] Database

[1925] The database is a collection of data that stores user interaction history and health information and can be accessed as needed, enabling continuous health monitoring.

[1926] 2. System Operation

[1927] Start a conversation

[1928] When the user presses the "Start Conversation" button on the device, the device sends this request to the server, which uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[1929] Collection and analysis of everyday conversations

[1930] As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The device then presents the generated new conversation data to the user by displaying and playing it back on the panda character.

[1931] Health monitoring

[1932] The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities. For example, if a user says, "I've been having stomach problems lately," the server uses a generative model to generate detailed questions and present them to the user.

[1933] Anomaly detection and notification

[1934] If the server detects any abnormalities in health, it will send the information to family members or medical institutions in real time via a notification function. For example, if someone says "I have chest pain," a notification will be sent immediately to family members or emergency medical institutions, and an emergency call can even be made.

[1935] Meal menu suggestions and ordering

[1936] Based on the dialogue data generated based on the user's health status, the server proposes a meal menu suitable for the user. The user can select from the proposed menu and place an order through the terminal. The order details are sent to the server, which then notifies the meal service provider.

[1937] 3. Specific Examples

[1938] Scenario: A user complains of stomach upset.

[1939] 1. Start a conversation

[1940] Terminal: "Hello! How are you feeling today?" asks the panda character to the user.

[1941] User: "I've been having stomach problems lately."

[1942] Terminal: Converts the user's speech into text and sends it to the server.

[1943] 2. Conversation analysis and health assessment

[1944] Server: Analyzes the keyword "I've been having stomach problems lately," and based on this, the generative model generates a new dialogue: "That's worrying. What's the best thing to eat to ease your stomach?"

[1945] Server: Sends the generated dialogue to the terminal.

[1946] Device: A panda character will display and play a voice saying, "That's worrying. What is the best food to eat for your stomach?"

[1947] 3. Meal menu suggestions

[1948] Server: Based on the user's health condition, the server suggests meal menus such as "healthy soup with less carbonated drinks" or "nutritiously balanced porridge with plenty of vegetables."

[1949] Terminal: A panda character displays the suggested menu.

[1950] 4. Order Confirmation

[1951] User: Selects from the proposed menu and places an order through the terminal.

[1952] Server: Notifies the food service provider of the order.

[1953] Server: Sends a confirmation message to the terminal saying, "Your order has been confirmed. Delivery will be in 30 minutes."

[1954] Prompt Sentence Examples

[1955] You: I've been having stomach problems lately.

[1956] Panda: That's worrying. What's the easiest thing to eat for your stomach?

[1957] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1958] Step 1:

[1959] The user presses the "Start conversation" button on the device.

[1960] Input: User presses a button

[1961] Output: Sending a conversation initiation request to the server

[1962] Specific operation: The device sends a conversation start request to the server. The server receives this request, uses the generative model to generate an initial greeting message, and sends it to the device.

[1963] Step 2:

[1964] The device will display and play a voice message to the user through the panda character as a greeting message.

[1965] Input: Initial greeting message from the server

[1966] Output: Present natural dialogue to the user

[1967] Specific operation: The device receives a greeting message from the server and displays it as a panda character, playing it aloud.

[1968] Step 3:

[1969] The user responds to the panda character.

[1970] Input: User's voice response

[1971] Output: Audio data

[1972] Specific operation: The device receives the user's voice response and converts the voice into text.

[1973] Step 4:

[1974] The terminal transmits the user's text data to the server.

[1975] Input: Text data converted from audio

[1976] Output: Sending text data to the server

[1977] Specific operation: The device sends the text data converted from the voice to the server.

[1978] Step 5:

[1979] The server analyzes the text data and generates the next question or response.

[1980] Input: User's text data

[1981] Output: The next question or response generated

[1982] Specific operation: The server analyzes the received text data using a generative model and generates the next appropriate question or response.

[1983] Step 6:

[1984] The server transmits the generated dialogue to the terminal.

[1985] Input: Generated dialogue

[1986] Output: Send dialogue to terminal

[1987] Specific operation: The server sends the generated dialogue to the terminal.

[1988] Step 7:

[1989] The device displays and plays aloud the dialogue generated through the panda character to the user.

[1990] Input: Generated dialogue from the server

[1991] Output: Presents dialogue to the user

[1992] Specific operation: The terminal displays and plays aloud the dialogue received from the server using a panda character.

[1993] Step 8:

[1994] The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities.

[1995] Input: Interaction data

[1996] Output: Health status assessment results, abnormality detection results

[1997] Specific operation: The server uses the generative model to analyze the dialogue data and detect specific keywords and signs of anomalies.

[1998] Step 9:

[1999] The server will suggest an appropriate meal menu based on the health assessment results.

[2000] Input: Health status assessment results

[2001] Output: Suggested meal menu

[2002] Specific operation: The server uses the health status assessment results to suggest health-conscious meal menus through a generative model.

[2003] Step 10:

[2004] The terminal displays the suggested meal menu to the user, who places an order.

[2005] Input: Suggested meal menu, user selection

[2006] Output: Order data

[2007] Specific operation: The terminal displays the proposed meal menu and the user selects. The user's order is sent from the terminal to the server.

[2008] Step 11:

[2009] The server transmits the order data to the food service provider.

[2010] Input: Order data

[2011] Output: Notification to food service provider

[2012] Specific operation: The server notifies the meal service provider of the user's order data, confirms the order, and sends a confirmation message to the user's terminal.

[2013] Through the above processing steps, the user can order an appropriate meal menu while monitoring their health condition.

[2014] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2015] MODE FOR CARRYING OUT THE INVENTION

[2016] The present invention combines an emotion engine with an interactive medical AI service system for the elderly. Specific embodiments of the system are described below.

[2017] Components

[2018] 1. Devices: Electronic devices such as tablets and smartphones used by the elderly that provide an interface that allows users to converse with panda characters on a daily basis.

[2019] 2. Server: A computer system that executes the generative model, manages and analyzes the dialogue data with the user, and transmits the generated dialogue and health assessment data to the terminal.

[2020] 3. Generative Model: This is an artificial intelligence technique used to generate natural dialogue with the user, detect specific keywords, and assess health status based on them.

[2021] 4. Database: A collection of data that stores user interaction data and can be accessed as needed.

[2022] 5. Emotion engine: This engine recognizes emotions from the user's voice and text data and adjusts the dialogue with the user based on the recognized emotional data.

[2023] System Operation

[2024] 1. Initial Setup: When a user uses the service for the first time, they install the app on their device and enter their personal information (name, age, health information, etc.). This information is sent to the server and stored in a database.

[2025] 2. Start conversation: The user presses the "Start conversation" button on the device. The device sends this request to the server, which uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[2026] 3. Collection and analysis of everyday conversations: As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The device presents the generated new conversation data to the user by displaying and playing it back on the panda character.

[2027] 4. Health monitoring: The server continuously analyzes the conversation data to detect specific keywords and signs of abnormalities. Based on this, the generative model generates appropriate responses and questions. For example, if the user says, "My lower back hurts," the server instructs the generative model to ask more detailed questions about the extent and location of the pain.

[2028] 5. Emotion recognition and adjustment: The device sends the user's voice and facial expression data to the emotion engine, which then recognizes the user's emotions. For example, if the user has an anxious expression or voice, the emotion engine sends that information to the server.

[2029] 6. Emotion-based dialogue adjustment: The server analyzes the emotion data received from the emotion engine and generates appropriate dialogue based on the emotion. For example, if the user is anxious, the panda character will generate a dialogue such as, "Is there anything you're worried about? Please tell me."

[2030] 7. Abnormality detection and notification: If the server detects an abnormality in health status, the notification function will send the information to family members and medical institutions in real time. For example, if someone says, "I have chest pain," a notification will be sent immediately to family members and emergency medical institutions, and if necessary, an emergency call will be made.

[2031] 8. Report Generation: The server periodically generates a report summarizing the user's health status and emails it to designated family members or healthcare providers. The report includes a summary of the conversation, any abnormalities detected, and recommended next steps.

[2032] Specific examples

[2033] Scenario: An elderly person complains of back pain and becomes emotionally unstable.

[2034] 1. Start a conversation

[2035] Terminal: "Hello! How are you today?" asks the panda character to the user.

[2036] User: "I've been having back pain lately..."

[2037] Terminal: Converts the user's speech into text and sends it to the server.

[2038] 2. Conversation Analysis and Anomaly Detection

[2039] Server: Analyzes the keyword "my lower back hurts," and based on this, the generative model generates a new dialogue: "That's worrying. Where does it hurt?"

[2040] Server: Sends the generated dialogue to the terminal.

[2041] Device: Panda will say and listen, "That's worrying. Where does it hurt?"

[2042] 3. Emotional Recognition

[2043] Device: The user answers in an anxious voice, "My whole lower back hurts."

[2044] Device: Sends voice data to the emotion engine, which detects anxiety.

[2045] Server: Receives emotion data from the emotion engine and generates dialogue based on it.

[2046] 4. Adjusting dialogue based on emotions

[2047] Server: Based on the information from the emotion engine, it generates the dialogue, "You're okay. Take a rest and don't push yourself too hard. Is there anything I can help you with?"

[2048] Server: Sends the generated dialogue to the terminal.

[2049] Device: Panda will say, "It's okay. Take care and rest. Is there anything I can help you with?"

[2050] This invention allows elderly people to enjoy daily conversations with panda characters while simultaneously managing their health. It also responds sensitively to the user's emotions and provides appropriate dialogue, providing a greater sense of security. Family members and medical institutions can monitor the elderly's health status in real time, contributing to improving the quality of life for the elderly.

[2051] The processing flow will be explained below.

[2052] Step 1:

[2053] Device: Display the app installation screen.

[2054] User: Installs and launches the app.

[2055] Step 2:

[2056] Device: Display the initial setup screen (name, age, health information, etc.).

[2057] User: Enter the required information and press the send button.

[2058] Terminal: Sends the entered information to the server.

[2059] Server: Stores the received information in a database and sends a response to the terminal indicating that initial setup is complete.

[2060] Terminal: Notify the user that the initial setup is complete.

[2061] Step 3:

[2062] Device: Display a "Start Conversation" button on the home screen.

[2063] User: Press the start conversation button.

[2064] Terminal: Sends a request to start a conversation to the server.

[2065] Step 4:

[2066] Server: Initiates the conversation session and generates the initial greeting message.

[2067] Server: Sends a greeting message to the terminal.

[2068] On your device: A panda character will display and / or play a greeting message (e.g., "Hello! How was your day today?").

[2069] Step 5:

[2070] User: Answer the panda character's questions (e.g., "It was nice weather today so I went for a walk").

[2071] Terminal: Converts the user's speech into text and sends it to the server.

[2072] Step 6:

[2073] Server: Analyzes the received text data and generates the next question or response.

[2074] Server: Sends the generated conversation data to the terminal.

[2075] On your device: The panda character will display and play your new message (e.g., "That's lovely! Which way did you walk?").

[2076] Step 7:

[2077] Server: Continuously analyzes conversation data and detects keywords and anomalies.

[2078] Server: Generates new responses and questions based on the detected information.

[2079] Device: View and play new questions and responses from the panda character.

[2080] Step 8:

[2081] User: Provides health information as the conversation continues (e.g., "I've been having a lot of back pain lately...").

[2082] Terminal: Converts the user's speech into text and sends it to the server.

[2083] Step 9:

[2084] Server: Analyzes keywords that indicate abnormal health conditions, such as "lower back pain," and detects abnormalities.

[2085] Server: If an abnormality is detected, it generates a notification to notify family members or medical institutions.

[2086] Server: Send notifications to family members or medical institutions (e.g., "You've recently been complaining of back pain. You may need further evaluation.").

[2087] Step 10:

[2088] Strategic terminal: Sends the user's voice and facial expression data to the emotion engine.

[2089] Emotion engine: Recognizes user emotions and detects emotions such as anxiety.

[2090] Emotion Engine: Sends detected emotion data to the server.

[2091] Step 11:

[2092] Server: Analyzes the received emotion data and generates appropriate dialogue based on the emotion.

[2093] Server: Sends the generated dialogue to the terminal.

[2094] On your device: The panda character will display and / or play a sensitive message (e.g., "It's okay. Take care and rest. Is there anything I can help you with?").

[2095] Step 12:

[2096] Server: Periodically aggregates conversation data and generates a health status report for the elderly.

[2097] Server: Sends the generated report to the family or medical institution.

[2098] Step 13:

[2099] Device: Continue everyday conversations with panda characters.

[2100] User: Continue to provide information about health status through interactions with the panda character.

[2101] Example 2

[2102] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2103] Elderly people need to be sensitive to changes in their health and emotions in their daily lives. However, it is often difficult for them to visit medical institutions or communicate with specialists on a daily basis. For this reason, there is a need for a system that allows elderly people to easily manage their health at home and receive appropriate support in response to emotional changes.

[2104] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2105] In this invention, the server includes means for analyzing the dialogue data to evaluate the health condition, means for notifying when an abnormality in the health condition is detected, and means for adjusting the dialogue based on the emotion data, thereby enabling the user to manage their health condition while at home and receive appropriate dialogue in response to changes in their emotions.

[2106] "Communication devices" are electronic devices such as tablets and smartphones used by elderly people, and are devices that provide an interface for users to interact with each other on a daily basis.

[2107] An "information processing device" is a computer system that executes generative models and manages and analyzes user interaction data.

[2108] A "storage device" is a collection of data that stores user interaction data and can be accessed as needed.

[2109] A "generative model" is an artificial intelligence technique used to generate natural-sounding dialogue with a user.

[2110] An "emotion recognition engine" is an engine that recognizes emotions from the user's voice and text data and adjusts the dialogue based on that information.

[2111] "Dialogue data" refers to data such as messages and voice recordings exchanged between a user and a communication device.

[2112] "Health status assessment" is the process of analyzing dialogue data and determining the user's health status.

[2113] "Notification means" is a function that sends information to family members or medical institutions when an abnormality in health condition is detected.

[2114] "Emotion data" is data that indicates the emotional state of the user analyzed by the emotion recognition engine.

[2115] "Dialogue adjustment" is the process of modifying the dialogue content and providing an appropriate response to the user based on emotion data.

[2116] This invention relates to an interactive medical support system for the elderly, and in particular to a system that utilizes a generative AI model with emotion recognition capabilities. The specific operation and usage of the system are described below.

[2117] System Overview

[2118] The system includes a communication device (e.g., a tablet or smartphone) used by the elderly, an information processing device that executes a generative model, and a storage device that manages interaction data with the user. The system includes the following means:

[2119] A means for communication devices to present natural dialogue to users

[2120] The means by which the generative model generates interactions with the user

[2121] A means for an information processing device to analyze the dialogue data and evaluate the health condition

[2122] A means for notifying when an information processing device detects an abnormality in health status

[2123] A means for communication devices to send user voice and facial expression data to an emotion recognition engine

[2124] The emotion recognition engine analyzes the user's emotions and transmits the results to the information processing device.

[2125] A means for an information processing device to adjust a dialogue based on emotion data

[2126] Initial Setup

[2127] When a user uses the system for the first time, they install a dedicated app on their communication device and enter personal information such as their name, age, and health information, which is then sent to the server and stored in a database.

[2128] Start a conversation

[2129] When the user presses the "Start Conversation" button on the communication device, the communication device sends this request to the server. The server uses the generative model to generate an initial greeting message and sends it to the communication device. The communication device then displays and plays the greeting message aloud using a panda character.

[2130] Collection and analysis of everyday conversations

[2131] As the user engages in everyday conversation with the panda character, the communication device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. The generated dialogue is sent to the communication device, where it is displayed and played aloud by the panda character.

[2132] Health monitoring

[2133] The server continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. For example, if a user says, "My lower back hurts," the server instructs the generative model to ask more detailed questions about the extent and location of the pain.

[2134] Emotional awareness and regulation

[2135] The communication device sends the user's voice and facial expression data to an emotion recognition engine, which then analyzes the user's emotions. For example, if the user has an anxious expression or voice, that information is sent to the server. The server analyzes the emotional data, generates appropriate dialogue based on the emotion, and sends it to the communication device.

[2136] Health Assessment and Notification

[2137] If the server detects any abnormalities in a person's health, it will notify family members or medical institutions in real time. For example, if someone says, "I have chest pain," a notification will be sent immediately, and an emergency call will be made if necessary.

[2138] Report Generation

[2139] The server periodically generates a report summarizing the user's health status and emails it to designated family members or healthcare providers, including a summary of the conversation, any abnormalities detected, and recommended next steps.

[2140] Specific examples

[2141] Scenario: An elderly person complains of back pain and becomes emotionally unstable.

[2142] 1. Start a conversation

[2143] Terminal: "Hello! How are you today?" asks the panda character to the user.

[2144] User: "I've been having back pain lately..."

[2145] Terminal: Converts the user's speech into text and sends it to the server.

[2146] 2. Conversation Analysis and Anomaly Detection

[2147] Server: Analyzes the keyword "my lower back hurts," and based on this, the generative model generates a new dialogue: "That's worrying. Where does it hurt?"

[2148] Server: Sends the generated dialogue to the terminal.

[2149] Device: A panda character will display and play a voice message saying, "That's worrying. Where does it hurt?"

[2150] 3. Emotional Recognition

[2151] User: In an anxious voice, replies, "My lower back hurts all over."

[2152] Device: Sends voice data to the emotion engine to detect anxiety.

[2153] Server: Receives the emotion data and generates a dialogue such as, "You're okay. Take care and rest. Is there anything I can help you with?"

[2154] 4. Coordinating dialogue

[2155] Server: Sends the generated dialogue to the terminal.

[2156] Device: A panda character will display and play a voice saying, "It's okay. Please take it easy and rest. Is there anything I can help you with?"

[2157] Prompt Sentence Examples

[2158] "The user complains of back pain and appears anxious. Based on this, please generate a dialogue that will reassure the user."

[2159] This invention allows elderly people to manage their health through the panda character in their daily lives and provides appropriate dialogue, allowing them to spend their days with peace of mind.Furthermore, family members and medical institutions can monitor their health status in real time, contributing to improving the quality of life for elderly people.

[2160] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2161] Step 1: Initial Setup

[2162] User: Installs the app on the device and enters personal information (name, age, health information, etc.) through the app's input form.

[2163] Terminal: Receives input personal information and converts it into JSON format to send to the server. The input is personal information, and the output is the converted JSON data.

[2164] Server: Analyzes the received JSON data and saves the user information in a database. This allows basic user information to be managed on the server side.

[2165] Step 2: Start a conversation

[2166] User: Presses the "Start conversation" button on the device. Input is the action of clicking the "Start conversation" button.

[2167] Terminal: Generates a conversation initiation request and sends it to the server. The output is the generated request data.

[2168] Server: Receives the request and generates the initial greeting message using the generative model. The input is the request and the output is the greeting message.

[2169] Server: Sends the generated greeting message to the terminal. The output is the generated message data.

[2170] Terminal: The panda character displays and plays audibly the received greeting message. The input is the greeting message, and the output is the display and audio playback for the user.

[2171] Step 3: Collecting and analyzing everyday conversations

[2172] User: Have everyday conversations with a panda character. Input is via user voice and text.

[2173] Terminal: Converts user speech from voice to text. This conversion is done by speech recognition software. The input is the user's voice, and the output is text data.

[2174] Terminal: Transmits converted text data to the server. The output is text data.

[2175] Server: Analyzes the received text data and generates the next question or response. This analysis is performed using a natural language processing model. The input is text data, and the output is a new dialogue.

[2176] Server: Sends the generated dialogue to the terminal. The output is dialogue data.

[2177] Terminal: The panda character displays and plays audibly the received dialogue. The input is the dialogue, and the output is a display and audible playback for the user.

[2178] Step 4: Monitor your health

[2179] Server: Continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. A keyword extraction algorithm is used for analysis. The input is dialogue data, and the output is health assessment data.

[2180] Server: When an anomaly is detected, it uses a generative model to generate appropriate questions and responses. The input is the anomaly detection result, and the output is the new questions and responses.

[2181] Server: Sends the generated dialogue to the terminal. The output is dialogue data.

[2182] Terminal: The panda character displays and plays audibly the received dialogue. The input is the dialogue, and the output is a display and audible playback for the user.

[2183] Step 5: Recognize and regulate your emotions

[2184] Terminal: Sends the user's voice and facial expression data to the emotion recognition engine. This transmission is done in real time. The input is voice and facial expression data, and the output is the transmitted data.

[2185] Emotion recognition engine: Analyzes the user's emotions and sends the results to the server. The input is voice and facial expression data, and the output is emotional data.

[2186] Server: Receives emotional data and adjusts the dialogue based on the analysis results. The input is emotional data, and the output is the adjusted dialogue.

[2187] Server: Sends the adjusted dialogue to the terminal. The output is dialogue data.

[2188] Terminal: The panda character displays and plays audibly the received dialogue. The input is the dialogue, and the output is a display and audible playback for the user.

[2189] Step 6: Health Assessment and Notification

[2190] Server: When an abnormality in health status is detected, the notification function is activated. This function includes push notifications and email notifications. The input is the abnormality detection result, and the output is the notification data.

[2191] Server: Notifies family and medical institutions of health status in real time. Makes emergency calls if necessary. Input is notification data, output is notification and alert.

[2192] Step 7: Generate reports

[2193] Server: Periodically generates a report summarizing the user's health status. This report includes a summary of the conversation, any abnormalities detected, and recommended next steps. The input is the conversation data and health assessment data, and the output is the report.

[2194] Server: Sends the generated report to the designated family or medical institution by email. The input is the report data, and the output is the sent report.

[2195] (Application example 2)

[2196] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2197] Health management, especially for the elderly, is an important issue in modern society. Elderly people require daily health monitoring, and prompt action is required when abnormalities occur. However, conventional systems do not provide natural dialogue and are difficult to respond to, taking into account the user's emotions. Therefore, there is a need for a system that combines natural dialogue and emotion recognition that can be used comfortably by the elderly.

[2198] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for analyzing dialogue data and evaluating health status; means for the server to notify when it detects an abnormality in health status; means for the terminal and the server to recognize emotions and adjust the dialogue based on the emotions; means for the terminal to check the health status of residents of the welfare facility through everyday conversation; and means for the server to periodically generate health status reports and provide them to a specified contact. This enables elderly people to manage their health with a sense of security through more natural dialogue.

[2199] The term "elderly" refers to people over a certain age, and in healthcare and welfare in particular, generally refers to people aged 65 and over.

[2200] A "terminal" is an electronic device used by a user, which inputs and outputs data via an interface.

[2201] A "generative model" is an algorithm or program that uses artificial intelligence technology to generate natural-sounding dialogue.

[2202] A "server" is a computer system that manages data and programs and provides services to terminals via a network.

[2203] A "database" is a collection of data that stores user interaction data and health information and can be accessed as needed.

[2204] "Emotions" refer to psychological states such as joy, sadness, and anxiety that humans experience, and can be read from voice and facial expressions.

[2205] A "dialogue" is a written communication between a user and a terminal.

[2206] "Abnormal" refers to a state that deviates from normal health and requires immediate action.

[2207] "Notification" is the act of communicating specific information to designated recipients in real time.

[2208] A "welfare facility" refers to a dedicated facility where elderly people, people with disabilities, etc. can live while receiving support and care.

[2209] MODE FOR CARRYING OUT THE INVENTION

[2210] This invention describes an embodiment of an interactive medical AI service system for elderly people specifically implemented in a welfare facility. The components and operation of the system are described in detail below.

[2211] System Configuration

[2212] 1. Devices: Electronic devices such as tablets and smartphones used by the elderly. Devices provide an interface with users and have the ability to display and play natural dialogue.

[2213] 2. Generative model: An artificial intelligence algorithm that uses natural language processing to generate dialogue with the user. Specifically, it incorporates Spacy and BERT.

[2214] 3. Server: A computer system that manages the database and generative model and sends dialogues and notifications to devices via a network. The server is built using Python or Flask and is responsible for data analysis and dialogue generation.

[2215] 4. Emotion Engine: An algorithm that recognizes emotions from the user's voice and text data and adjusts the dialogue accordingly. The EmotionEngine class is an example of this.

[2216] 5. Database: A system for storing conversation data and health information. MySQL, PostgreSQL, etc. are used.

[2217] Operation overview

[2218] 1. Initial setup: The server stores the user's basic information (name, age, health information, etc.) in a database. During the initial setup, the user installs the app on their device and enters the required information.

[2219] 2. Start of conversation: When the user presses the "Start conversation" button on the device, the server uses the generative model to generate an initial greeting message and sends it to the device. The device then displays and plays the greeting message as a panda character.

[2220] 3. Collection and analysis of everyday conversations: As the user continues to converse with the panda character, the device converts the user's speech into text and sends it to the server. The server analyzes the received text data and generates the next question or response. For example, if the user says, "I've been having a pain in my lower back lately...", the generative model will generate a response such as, "That's worrying. Where does it hurt?"

[2221] 4. Health status monitoring: The server continuously analyzes the dialogue data to detect specific keywords and signs of abnormalities. Based on this, the generative model generates appropriate responses and questions. For example, if the keyword "chest pain" is detected, this information will be promptly notified to family members or medical institutions.

[2222] 5. Emotion recognition and adjustment: The device sends the user's voice and facial expression data to the emotion engine, which then recognizes the user's emotions. For example, if the user's facial expression or voice looks anxious, the emotion engine detects the anxiety and sends this information to the server. The server analyzes the emotion data and generates a dialogue based on the emotion.

[2223] 6. Periodic Report Generation: The server periodically generates a report summarizing the user's health status and provides it to designated contacts (family members or healthcare providers), including a summary of the conversation, any abnormalities detected, and recommended next steps.

[2224] Specific examples

[2225] Scenario: An elderly person complains of lower back pain

[2226] 1. Start a conversation

[2227] User: "I've been having back pain lately..."

[2228] System: "That's concerning. Where does it hurt?"

[2229] 2. Emotional Recognition

[2230] User: "My lower back hurts all over."

[2231] System: The emotion engine detects the user's anxiety and generates a response such as, "You're OK. Take care and get some rest. Is there anything I can help you with?"

[2232] With these actions, elderly people can manage their health with peace of mind through everyday conversation. In addition, if any abnormalities occur, family members and medical institutions will be notified promptly, allowing them to live their lives with peace of mind.

[2233] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2234] Step 1: Initial Setup

[2235] Users install a medical app for the elderly on their device and enter their personal information (name, age, health information, etc.).

[2236] Input: User personal information

[2237] Processing: The device sends this information to the server, which stores it in a database.

[2238] Output: User information stored in the database

[2239] Step 2: Start a conversation

[2240] The user presses the "Start conversation" button on the device.

[2241] Input: User operation (pressing the "Start conversation" button)

[2242] Processing: The server uses the generative model to generate an initial greeting message and sends it to the terminal.

[2243] Output: A greeting message displayed on the terminal (natural dialogue)

[2244] Step 3: Collecting and analyzing everyday conversations

[2245] Users can converse with a panda character.

[2246] Input: User's voice data

[2247] Processing: The device converts the voice data into text and sends it to the server. The server analyzes the received text, and the generative model generates the next question or response. The server then sends the generated dialogue back to the device.

[2248] Output: The new dialogue displayed on the terminal.

[2249] Step 4: Monitor your health

[2250] The server analyzes the conversation data and detects specific keywords and signs of abnormalities.

[2251] Input: User's text data

[2252] Processing: The server detects specific keywords (e.g., "my lower back hurts") and evaluates the health status. If an abnormality is detected, it instructs the generative model on the next course of action.

[2253] Output: The next dialogue generated, and the anomaly detection results.

[2254] Step 5: Recognize and regulate your emotions

[2255] The device sends the user's voice and facial expression data to the emotion engine.

[2256] Input: User's voice and facial expression data

[2257] Processing: The emotion engine analyzes these data to recognize emotions and sends the results to the server. The server analyzes the emotion data, and the generative model generates appropriate dialogue based on the emotion.

[2258] Output: Generated dialogue (content corresponding to emotions)

[2259] Step 6: Anomaly detection and notification

[2260] If the server detects any abnormalities in health, it will use the notification function to send information to family members or medical institutions in real time.

[2261] Input: Anomaly detection results

[2262] Processing: The server generates a notification and sends it to the specified contacts, including calling emergency services if necessary.

[2263] Output: Notification sent to family and healthcare providers

[2264] Step 7: Generate reports

[2265] The server periodically generates a report summarizing the user's health status and emails it to designated contacts.

[2266] Input: Dialogue data and emotion data

[2267] Processing: The server compiles and analyzes the data, generates a report, and sends the report to the designated contact.

[2268] Output: Health report sent to family and healthcare providers

[2269] Through the above processing steps, elderly people can manage their health through natural dialogue and emotion recognition, and can live with peace of mind even in welfare facilities.

[2270] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. ...

Claims

1. A system including a terminal used by an elderly person, a server that executes a generative model, and a database that manages interaction data with the user, A terminal has a means for presenting natural dialogue to a user; a means by which the generative model generates an interaction with the user; A means for the server to analyze the dialogue data and evaluate the health state; A system including a means for notifying when the server detects an abnormality in health status.

2. The system of claim 1 , wherein the generative model detects specific keywords through everyday conversation and evaluates health status based on the keywords.

3. The system according to claim 1 , wherein the server periodically generates a health status report and provides it to a designated family member or medical institution.

Images