system

Abstract

Provide a system. The elderly person terminal includes a means for starting to speak at a predetermined time and asking a predetermined question; A means for recording the elderly person's speech response in the elderly person terminal; A means for converting the recorded voice data into text data in the elderly person's device; A means for transmitting the converted text data to the central processing unit in the elderly person's terminal; A means for the central processing unit to analyze the received text data and evaluate memory ability and cognitive function; A system including a means for generating and notifying an alert when a central processing unit detects an abnormality based on the analysis results.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's world, the number of elderly people living alone is increasing, making the prevention and early detection of dementia a major challenge. Dementia is difficult to detect in its early stages, and it often goes unnoticed until it has progressed. This can result in missed opportunities to receive appropriate medical care and support, reducing the quality of life of elderly people. To solve this problem, a system is needed that can detect cognitive decline early through everyday conversations and quickly implement necessary measures. [Means for solving the problem]

[0005] This invention provides a system for assessing cognitive function through voice conversations with elderly people and detecting abnormalities at an early stage. Specifically, the system includes: (1) a means for an elderly person's terminal to start speaking at a predetermined time and ask predetermined questions; (2) a means for the elderly person's terminal to record the elderly person's spoken responses; (3) a means for converting the recorded voice data into text data; (4) a means for transmitting the converted text data to a central processing unit; (5) a means for analyzing the received text data and evaluating memory ability and cognitive function; and (6) a means for generating and notifying an alert if an abnormality is detected based on the analysis results. This system can detect early signs of dementia through everyday conversations and improve the quality of life of elderly people.

[0006] An "elderly terminal" is an electronic device for use by elderly people that has the function of transmitting and receiving voice and processing data.

[0007] A "predetermined time" refers to a specific, pre-determined time period or time slot.

[0008] "Start speaking" means that the audio output device outputs pre-set content as voice.

[0009] "Predetermined questions" refer to the pre-determined questions that the system asks the elderly.

[0010] "Spoken responses" refer to the content of the voices that elderly people verbally respond to questions posed by the system.

[0011] "Recording means" refers to a device or software that has the function of recording and storing audio as electronic data.

[0012] "Voice data" refers to data that has been digitally recorded as audio information spoken by an elderly person.

[0013] "Text data" refers to voice data converted into text using voice recognition technology.

[0014] "Central Processing Unit" refers to a computer system or server for analyzing received data and outputting evaluation results.

[0015] "Analysis tools" refers to algorithms or software used to process data and extract specific patterns or information.

[0016] "Memory ability" refers to a person's ability to remember information and later recall it.

[0017] "Cognitive function" refers to the ability to perform intellectual activities such as perception, thinking, learning, problem-solving, and decision-making.

[0018] "Means for evaluation" refers to a device or software that has the function of analyzing the collected data and generating evaluation results according to specific criteria.

[0019] "Means for generating and notifying warnings" refers to a device or system that has the function of generating a warning message and notifying relevant parties when an abnormality is detected.

[0020] "Report" refers to a document or data summarizing the results of an analysis, which is subsequently provided to interested parties.

[0021] "Abnormal" refers to patterns of data that deviate from the normal range or signs of cognitive decline. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] System Overview

[0044] The system of the present invention combines an automated voice conversation system with data analysis algorithms to assess cognitive function and detect abnormalities in elderly people. The system includes an elderly person's terminal (hereinafter referred to as "terminal"), a central processing unit (hereinafter referred to as "server"), and related notification means.

[0045] Program processing flow

[0046] Conducting a voice conversation

[0047] Terminal

[0048] The terminal automatically starts the automated voice conversation system at a preset time.

[0049] The device speaks everyday questions to the elderly, such as "What did you do today?" and "What is your grandchild's name?"

[0050] When the elderly person answers these questions verbally, the device records the answers.

[0051] Specific examples

[0052] Terminal: "What did you do today?"

[0053] User: "Today I went to the park with my friends."

[0054] Audio data conversion and transmission

[0055] Terminal

[0056] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[0057] The converted text data is sent to the server.

[0058] Specific examples

[0059] Audio data: "Today I went to the park with my friends."

[0060] Text data: "Today I went to the park with my friends."

[0061] Receiving and storing data

[0062] server

[0063] The server receives the text data sent from the terminal.

[0064] The received text data is securely stored in a database for each elderly person.

[0065] Specific examples

[0066] Server: "New conversation data saved to database."

[0067] Data analysis

[0068] server

[0069] The server retrieves the stored text data for analysis.

[0070] The server applies algorithms to assess the elderly person's cognitive function, checking their memory ability and coherence.

[0071] Check the following points:

[0072] Conflicting answers

[0073] Memory lapses (e.g., forgetting grandchildren's names)

[0074] Confused conversation flow

[0075] Specific examples

[0076] Conversation log: "Your grandchildren came over yesterday." "No, no one came." (Inconsistency resolved)

[0077] Alert generation and notification

[0078] server

[0079] Based on the analysis results, the server generates a warning message if an abnormality is detected.

[0080] The server notifies the family or caregiver of the generated warning message.

[0081] Specific examples

[0082] Alert message: "An elderly person has recently exhibited inconsistencies in their conversations and urgent medical attention is recommended."

[0083] Report generation and delivery

[0084] server

[0085] The server periodically generates a report based on the analysis of the elderly person's conversation data.

[0086] The server sends the generated reports to family members or caregivers via email or a dedicated portal.

[0087] Specific examples

[0088] Report reads: "Analysis of conversations over the past month. Multiple inconsistencies and lapses in memory have been observed."

[0089] In this way, the system of the present invention utilizes an automated voice conversation system and data analysis algorithms to assess the cognitive function of elderly people and detect abnormalities early, and by sending appropriate alerts and reports to family members and caregivers, they can take necessary measures promptly.

[0090] The processing flow will be explained below.

[0091] Step 1:

[0092] Terminal

[0093] The terminal activates the automated voice conversation system at a preset time.

[0094] The device will ask the elderly everyday questions aloud, such as "What did you do today?" and "What is your grandchild's name?"

[0095] The user (elderly person) answers these questions orally, and the terminal records the answers in audio format.

[0096] Step 2:

[0097] Terminal

[0098] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[0099] It uses advanced speech recognition algorithms to generate accurate text data from audio data.

[0100] Step 3:

[0101] Terminal

[0102] The terminal transmits the converted text data to the server.

[0103] The text data also includes the elderly person's identification information and the date and time of the conversation.

[0104] Step 4:

[0105] server

[0106] The server receives the text data sent from the terminal.

[0107] The server securely stores the received text data in a database for each elderly person.

[0108] Step 5:

[0109] server

[0110] The server retrieves the stored text data for analysis.

[0111] The server applies natural language processing algorithms to assess memory and cognitive function, specifically checking the following points:

[0112] Conflicting answers

[0113] Memory deficiencies (e.g., forgetting grandchildren's names)

[0114] Confused conversation flow

[0115] Step 6:

[0116] server

[0117] If an abnormality is detected based on the analysis results, the server generates a warning message.

[0118] The generated warning message is sent to the elderly person's family or caregiver.

[0119] Step 7:

[0120] server

[0121] The server periodically (e.g., weekly or monthly) generates detailed reports based on the elderly person's conversation data.

[0122] The server sends the generated reports to family members or caregivers via email or a dedicated portal.

[0123] Through this series of steps, the system will enable daily monitoring of the cognitive function of elderly people, enabling early detection of abnormalities and follow-up.

[0124] Example 1

[0125] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0126] There is a need to detect cognitive decline in the elderly early and provide appropriate care promptly. Conventional methods often rely on home observation and regular medical examinations, which can easily miss cognitive decline. Furthermore, it is difficult to promptly notify family members or caregivers when abnormalities are detected and provide instructions for appropriate care. This reduces the quality of life of the elderly and increases the risk of inadequate care.

[0127] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0128] In this invention, the server includes: means for the elderly person terminal to start speaking at a predetermined time and ask predetermined questions; means for recording the elderly person's spoken responses; means for converting the recorded voice data into text data; means for transmitting the text data to a central processing unit; means for analyzing the received text data and evaluating memory ability and cognitive function; means for generating and notifying warnings if abnormalities are detected based on the analysis results; means for periodically generating reports based on the analysis results of the conversation data and sending them to relevant parties; means for automatically starting the conversation system at a predetermined time using scheduling software; means for analyzing the text data using natural language processing technology and evaluating consistency and memory ability for multiple situations; means for automatically generating and notifying warning messages of various levels based on abnormalities in the analysis results; and means for periodically reporting details of detected abnormalities to relevant parties. This enables automatic and continuous evaluation of the elderly person's cognitive function, early detection of abnormalities, and prompt notification and detailed reports to relevant parties.

[0129] An "elderly terminal" is a device with voice conversation and recording functions designed for use by elderly people.

[0130] The "predetermined time" refers to a specific time that has been set in advance, and is the reference time at which the system automatically starts operating.

[0131] "Speaking" refers to the act of the elderly person terminal asking a question to the elderly person by voice.

[0132] "Spoken response" refers to a verbal response given by the elderly person to the elderly person terminal.

[0133] "Recording" is the act of recording a speech response as audio data.

[0134] "Voice data" refers to digital voice information obtained by recording the elderly person's speech responses.

[0135] "Text data" is voice data converted into text information using natural language processing technology.

[0136] A "central processing unit" is a computer system that analyzes and evaluates received data, and detects and notifies abnormalities.

[0137] "Analysis" refers to the act of evaluating the memory ability and cognitive function of elderly people based on the received text data.

[0138] "Memory ability and cognitive function assessment" refers to the act of checking the consistency of an elderly person's answers and memory deficiencies, and dynamically assessing the state of their cognitive function.

[0139] "Abnormalities" refer to problems with the elderly person's cognitive function that are different from normal, such as inconsistencies or memory loss.

[0140] A "warning" is an alert message that is generated when an abnormality is detected, and is a notification that prompts a prompt response.

[0141] A "report" is a document summarizing the results of analysis of elderly people's conversation data, and is sent to relevant parties on a regular basis.

[0142] "Scheduling software" is a program that automatically starts system operations at pre-set times.

[0143] "Natural language processing technology" is a technology that converts voice data into text information and analyzes text data.

[0144] "Analysis results" are the results of evaluations and judgments made based on text data.

[0145] "Stakeholders" are people or organizations with an interest in the condition of older people, such as their families, caregivers, and medical professionals.

[0146] MODE FOR CARRYING OUT THE INVENTION

[0147] The system of the present invention utilizes an automated voice conversation system and data analysis algorithms to assess cognitive function in elderly people and detect abnormalities at an early stage. The system includes an elderly person terminal (hereinafter referred to as "terminal"), a central processing unit (hereinafter referred to as "server"), and related notification means.

[0148] Conducting a voice conversation

[0149] Terminal

[0150] The terminal automatically starts the automated voice conversation system at a preset time, using scheduling software such as a Cron job for scheduling.

[0151] The device speaks everyday questions to the elderly, such as "What did you do today?" and "What is your grandchild's name?", using Google® Text-to-Speech technology.

[0152] When the elderly responds to these questions verbally, the device uses a built-in microphone to record the responses, which are then temporarily stored in local storage.

[0153] Specific examples

[0154] Terminal: "What did you do today?"

[0155] User: "Today I went to the park with my friends."

[0156] Audio data conversion and transmission

[0157] Terminal

[0158] The device converts the recorded voice data into text data using natural language processing (NLP) technology, which uses the Google Cloud Speech-to-Text API.

[0159] After converting the data into text, the device sends the text to a remote server using an HTTP POST request, with end-to-end encryption applied to the communication.

[0160] Specific examples

[0161] Audio data: "Today I went to the park with my friends."

[0162] Text data: "Today I went to the park with my friends."

[0163] Receiving and storing data

[0164] server

[0165] The server receives the HTTP POST request sent from the terminal and extracts the text data from it. This process uses the Apache (registered trademark) or Nginx web server software.

[0166] The extracted text data is securely stored in a database for each elderly person. MySQL (registered trademark) or PostgreSQL is used as the database management system, and data encryption is performed.

[0167] Specific examples

[0168] Server: "New conversation data saved to database."

[0169] Data analysis

[0170] server

[0171] The server periodically retrieves the stored text data for analysis, using a Cron job.

[0172] The stored text data is then used to run an algorithm that assesses the cognitive function of older adults. This algorithm uses natural language processing (NLP) techniques and libraries such as Python's NLTK and spaCy.

[0173] During processing, check for inconsistent answers, memory gaps, and confused conversation flow.

[0174] Specific examples

[0175] Conversation log: "Your grandchildren came over yesterday." "No, no one came." (Inconsistency resolved)

[0176] Alert generation and notification

[0177] server

[0178] If an abnormality is detected based on the analysis results, the server generates a warning message. This message is automatically generated using a template.

[0179] The server then sends the generated warning message to family members or caregivers via email, SMS, or a dedicated app. Email is sent using the SMTP protocol, and SMS is sent using services such as Twilio.

[0180] Specific examples

[0181] Alert message: "An elderly person has recently exhibited inconsistencies in their conversations and urgent medical attention is recommended."

[0182] Report generation and delivery

[0183] server

[0184] The server periodically generates reports based on the analysis of the elderly's conversation data. The reports are generated in PDF format using libraries such as Python's ReportLab.

[0185] The server generates reports and sends them to family members or caregivers via email or a dedicated portal, which can be run periodically using a scheduler.

[0186] Specific examples

[0187] Report reads: "Analysis of conversations over the past month. Multiple inconsistencies and lapses in memory have been observed."

[0188] Example prompts to input to the generative AI model

[0189] Examples:

[0190] Terminal: "What did you do today?"

[0191] User: "Today I went to the park with my friends."

[0192] Example prompt sentence:

[0193] "Please suggest ways to predict how the elderly person will respond to the device and ensure consistency in the next conversation. Also, please suggest improvements to the algorithm to identify gaps or inconsistencies in memory."

[0194] The system of this invention can effectively evaluate the cognitive function of elderly people and detect abnormalities early. Detailed information can be obtained through conversational interaction, and appropriate nursing and medical measures can be taken in real time based on that information. In addition, appropriate alerts and regular reports enable prompt and accurate responses.

[0195] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0196] Step 1:

[0197] Starting a voice conversation

[0198] The terminal automatically starts the automated voice conversation system at a preset time, using scheduling software such as a Cron job for scheduling.

[0199] Input: Scheduling time

[0200] Operation: The device monitors its internal clock and starts the voice conversation system at the set time.

[0201] Output: Voice conversation system activated

[0202] Step 2:

[0203] Asking a question

[0204] The device uses Google Text-to-Speech technology to ask everyday questions to the elderly, such as "What did you do today?" or "What is your grandchild's name?"

[0205] Input: Preset question content

[0206] How it works: The device passes preset question text to the Google Text-to-Speech API, generates audio data, and speaks it to the elderly person.

[0207] Output: Question spoken to the elderly person

[0208] Step 3:

[0209] Spoken response recording

[0210] When the elderly responds verbally to the questions, the device uses a built-in microphone to record the response, which is then temporarily stored in local storage.

[0211] Input: Verbal response from the elderly person

[0212] How it works: The device captures audio data through the microphone and stores it in local storage.

[0213] Output: Recorded audio data file

[0214] Step 4:

[0215] Converting audio data to text

[0216] The device converts the recorded voice data into text data using natural language processing (NLP) technology, which uses the Google Cloud Speech-to-Text API.

[0217] Input: Recorded audio data

[0218] How it works: The device sends audio data to the Google Cloud Speech-to-Text API and retrieves text data.

[0219] Output: Converted text data

[0220] Step 5:

[0221] Sending text data

[0222] The device then sends the converted text data to a remote server using an HTTP POST request, with end-to-end encryption applied to the communication.

[0223] Input: Converted text data

[0224] How it works: The device generates an HTTP POST request and sends data to the specified endpoint on the server.

[0225] Output: Text data sent to the server

[0226] Step 6:

[0227] Receiving and storing text data

[0228] The server receives the HTTP POST request sent from the device and extracts the text data from it, using the Apache or Nginx web server software.

[0229] Input: HTTP POST request

[0230] What happens: The server parses the POST request and extracts the text data.

[0231] Output: Extracted text data

[0232] Step 7:

[0233] Data storage

[0234] The server securely stores the text data for each elderly person in a database, using MySQL or PostgreSQL as the database management system, and the data is encrypted.

[0235] Input: Extracted text data

[0236] Operation: The server saves the text data in a database. When saved, it associates it with the elderly person's identification information.

[0237] Output: Securely stored text data

[0238] Step 8:

[0239] Regular analysis of data

[0240] The server periodically retrieves the stored text data for analysis, using a Cron job.

[0241] Input: Text data stored in a database

[0242] Operation: The server periodically retrieves the stored text data and runs an analysis program.

[0243] Output: Text data extracted for analysis

[0244] Step 9:

[0245] Cognitive function assessment

[0246] The server runs an algorithm on the text data to assess cognitive function in older adults, using natural language processing (NLP) techniques and libraries such as Python's NLTK and spaCy.

[0247] Input: Text data extracted for analysis

[0248] How it works: The server runs a program that detects inconsistent answers, memory lapses, and disorganized conversation flow.

[0249] Output: Cognitive function assessment results

[0250] Step 10:

[0251] Anomaly detection and alert generation

[0252] If an abnormality is detected based on the analysis results, the server generates a warning message. This message is automatically generated using a template.

[0253] Input: Cognitive function assessment results

[0254] Operation: The server analyzes the evaluation results and generates a corresponding warning message if an anomaly is detected.

[0255] Output: Warning message

[0256] Step 11:

[0257] Warning Notification

[0258] The server then sends the generated warning message to family members or caregivers via email, SMS, or a dedicated app. Email is sent using the SMTP protocol, and SMS is sent using services such as Twilio.

[0259] Input: warning message

[0260] What it does: The server generates email and / or SMS notifications and sends them to interested parties.

[0261] Output: Warning message sent to family members or caregivers

[0262] Step 12:

[0263] Report generation and delivery

[0264] The server periodically generates reports based on the analysis of the elderly's conversation data. The reports are generated in PDF format using libraries such as Python's ReportLab.

[0265] Input: Analysis results of elderly people's conversation data

[0266] How it works: The server periodically aggregates the analysis results of the conversation data and generates a report in PDF format.

[0267] Output: Generated report

[0268] Step 13:

[0269] Report Notification

[0270] The server then sends the generated reports to family members or caregivers via email or a dedicated portal, and this process is also performed periodically using a scheduler.

[0271] Input: Generated report

[0272] How it works: The server sends reports via email or a dedicated portal.

[0273] Output: Report sent to family or caregiver

[0274] (Application example 1)

[0275] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0276] In modern society, efficient and accurate assessment of the cognitive function of elderly people and early detection of cognitive decline occurring in daily life are important challenges. However, conventional interactive systems lack a well-established method for thoroughly assessing the cognitive function of elderly people through their everyday conversations and appropriately notifying the results. Furthermore, they lack a means for quickly issuing warnings when abnormalities are detected and providing necessary information to family members and caregivers. Furthermore, they lack the ability to regularly store and analyze data to ensure the safety of elderly people. Therefore, the present invention aims to solve these challenges.

[0277] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0278] In this invention, the server includes a means for notifying family members and caregivers of speech-recognized text data, a means for periodically saving and analyzing daily conversation data, a means for analyzing the elderly person's conversation data using natural language processing technology, a means for sending warning messages regarding detected abnormalities to family members and caregivers, and a means for sending generated periodic reports to family members and caregivers. This enables detailed assessment of the elderly person's cognitive function through their daily conversations, early detection of abnormalities, and appropriate alert notifications. Furthermore, periodic data saving and analysis can provide comprehensive support for ensuring the safety of the elderly.

[0279] An "elderly terminal" is an electronic device for use by elderly people that is capable of voice input and output.

[0280] A "central processing unit" is a centralized device for analyzing and evaluating speech-recognized data, and functions as a server.

[0281] The "predetermined time" refers to a specific time or period that has been set in advance, at which the system automatically starts operating.

[0282] "Start speaking" means that the system will output pre-programmed messages and ask questions to the senior.

[0283] The "prescribed questions" are questions about the elderly person's daily life and memory, and are useful for assessing cognitive function.

[0284] "Spoken responses" refer to verbal responses given by the elderly to questions posed by the system.

[0285] "Means for recording" refers to a function for recording the elderly person's spoken responses as digital audio data.

[0286] "Means for converting into text data" refers to technology for converting recorded voice data into text information, and primarily utilizes natural language processing technology.

[0287] "Means of analysis" refers to algorithms for assessing the cognitive function of elderly people based on text data.

[0288] "Means for detecting abnormalities" refers to the function of detecting abnormalities such as a decline in cognitive function from the analysis results and instructing on necessary measures.

[0289] "Means for generating and notifying alerts" refers to a system that has the ability to generate alert messages regarding detected abnormalities and promptly notify family members or caregivers.

[0290] "Natural language processing technology" is a technology for processing voice and text data to understand and generate human language.

[0291] A "regular report" is a report summarizing the results of periodic analyses, and is sent to family members or caregivers.

[0292] "Daily conversation data" is a record of regular voice interactions between the elderly person and the system.

[0293] A "generative AI model" is a collection of algorithms trained to perform specific tasks using artificial intelligence.

[0294] A "prompt" is a question or instruction input to a generative AI model, and is a phrase used to encourage a response from the elderly person.

[0295] System Overview

[0296] The system of the present invention mainly consists of an elderly person's terminal, a central processing unit (server), and a notification means. The elderly person's terminal is equipped with an automated voice conversation system that evaluates the elderly person's cognitive function through everyday conversation. The central processing unit analyzes the received data and generates a warning message to notify family members or caregivers if an abnormality is detected. It also has a function to periodically provide analysis results as a report.

[0297] Hardware and software used

[0298] 1. Devices for the elderly:

[0299] Hardware: Smartphone or tablet (including microphone)

[0300] Software: Natural language processing technology (e.g., Google Speech-to-Text API)

[0301] 2. Central Processing Unit (Server):

[0302] Hardware: High-performance network server

[0303] Software: Database systems, data analysis algorithms, notification systems

[0304] Details of data processing and calculation

[0305] Device operation

[0306] The device will automatically start a voice conversation at a specific time. For example, it will ask questions such as "What did you do today?" or "What is your grandchild's name?" When the elderly person answers these questions, the device will record the voice data.

[0307] Audio data conversion and transmission

[0308] The recorded voice data is converted into text data using natural language processing technology within the device. This converted text data is sent to a server. For example, if an elderly person replies, "Today, I went to the park with my friends," the voice is converted into text data that reads, "Today, I went to the park with my friends."

[0309] Data reception and analysis

[0310] The server receives the text data sent from the device. The received data is stored in a database and analysis begins. The server applies algorithms to assess the elderly person's cognitive function and checks their memory ability and coherence. Analysis detects inconsistent answers, memory gaps, and disorganized conversation flow.

[0311] Generate alerts and reports

[0312] If an abnormality is detected based on the analysis results, the server generates a warning message and notifies the family or caregiver. For example, a message may be sent saying, "There have been inconsistencies in the elderly person's recent conversations. We recommend that you seek medical advice as soon as possible." In addition, reports are generated based on the periodic analysis results and sent to the family or caregiver. The report contents may include, "These are the results of the conversation analysis for the past month. Inconsistent answers and lapses in memory have been observed in multiple situations."

[0313] Examples of prompt statements

[0314] Specific examples of prompts include:

[0315] "What did you do today?"

[0316] "What's your grandson's name?"

[0317] "What did you have for dinner last night?"

[0318] This concludes the detailed description of the embodiment of the present invention. This system enables the evaluation of cognitive function through everyday conversations with elderly people and the early detection of abnormalities, allowing family members or caregivers to take prompt action.

[0319] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0320] Processing Steps

[0321] Step 1:

[0322] Starting up elderly device

[0323] Specific operation: The device automatically starts the automated voice conversation system at the specified time, performs initialization processing, and enables the microphone.

[0324] Input: Predetermined time based on system settings

[0325] Output: Automated voice conversation system startup status

[0326] Step 2:

[0327] Prompt sentence utterance

[0328] Specific operation: The device will speak a pre-defined prompt, such as "What did you do today?", to the elderly person.

[0329] Input: Questionnaires saved in the system

[0330] Output: Voice questions for the elderly

[0331] Step 3:

[0332] Spoken response recording

[0333] Specific operation: The elderly person answers questions, and their answers are recorded by the device's microphone.

[0334] Input: Elderly voice response

[0335] Output: Recorded audio data

[0336] Step 4:

[0337] Converting audio data to text

[0338] Specific operation: The recorded audio data is converted into text data using natural language processing technology (such as Google Speech-to-Text API).

[0339] Input: Recorded audio data

[0340] Output: Text data

[0341] Step 5:

[0342] Sending text data

[0343] Specific operation: The converted text data is sent to the server using an HTTP request.

[0344] Input: Converted text data

[0345] Output: Text data sent to the server

[0346] Step 6:

[0347] Receiving and storing data

[0348] Specific operation: The server receives the text data sent from the device and stores it in a database. When stored, the data is classified for each elderly person.

[0349] Input: Text data sent from the terminal

[0350] Output: Text data stored in a database

[0351] Step 7:

[0352] Data analysis

[0353] How it works: The server processes the stored text data using analytical algorithms to assess the elderly person's cognitive function, checking memory ability, consistency of speech, and inconsistencies.

[0354] Input: Text data stored in a database

[0355] Output: Cognitive function assessment results

[0356] Step 8:

[0357] Anomaly detection and alert generation

[0358] Specific operation: If the server detects an anomaly in the analysis results, it generates a warning message. Different levels of warning are set depending on the type and severity of the anomaly.

[0359] Input: Cognitive function assessment results

[0360] Output: Warning message

[0361] Step 9:

[0362] Sending notifications

[0363] Specific operation: The generated warning message is sent to family members or caregivers via email or a dedicated app.

[0364] Input: warning message

[0365] Output: Notification sent to family members and caregivers

[0366] Step 10:

[0367] Generate and send scheduled reports

[0368] Specific operation: The server periodically generates a report based on the analysis results and sends it to the family or caregiver. The report includes the analysis results of the elderly person's conversation data.

[0369] Input: Accumulated data of cognitive function evaluation results

[0370] Output: Regular reports sent to family members and caregivers

[0371] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0372] System Overview

[0373] The system of the present invention assesses cognitive function through voice conversation with elderly people and simultaneously recognizes their emotional state, enabling early detection of abnormalities. The system includes an elderly person terminal (hereinafter referred to as "terminal"), a central processing unit (hereinafter referred to as "server"), and related notification means. It also incorporates an emotion engine that performs emotion recognition.

[0374] Program processing flow

[0375] Voice conversation execution and emotion recognition

[0376] Terminal

[0377] The terminal activates the automated voice conversation system at a preset time.

[0378] The device will ask the elderly everyday questions such as "What did you do today?" or "What is your grandchild's name?"

[0379] When the elderly person answers these questions verbally, the device records the answers in audio format.

[0380] The emotion engine built into the device recognizes emotions in real time from the voice data of the elderly person's speech responses.

[0381] Specific examples

[0382] Terminal: "What did you do today?"

[0383] User: "I went to the park with my friends today." (Emotion of joy is recognized from the tone of voice)

[0384] Audio data conversion and transmission

[0385] Terminal

[0386] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[0387] It uses advanced speech recognition algorithms to generate accurate text data from audio data.

[0388] The converted text data and emotion data are sent to the server.

[0389] Specific examples

[0390] Audio data: "Today I went to the park with my friends."

[0391] Text data: "Today I went to the park with my friends."

[0392] Emotion data: "Joy"

[0393] Receiving and storing data

[0394] server

[0395] The server receives the text data and emotion data sent from the terminal.

[0396] The server securely stores the received data in a database for each elderly person.

[0397] Specific examples

[0398] Server: "New speech and emotion data saved to database."

[0399] Data analysis

[0400] server

[0401] The server retrieves the stored text data and emotion data for analysis.

[0402] The server applies natural language processing algorithms to assess memory and cognitive function, specifically checking the following:

[0403] Conflicting answers

[0404] Memory deficiencies (e.g., forgetting grandchildren's names)

[0405] Confused conversation flow

[0406] The server also analyzes the emotion data recognized by the emotion engine and evaluates the abnormality of the emotional state.

[0407] Specific examples

[0408] Conversation log: "Your grandchildren came over yesterday." "No, no one came." (Inconsistency resolved)

[0409] Emotional data: "Continued anxiety"

[0410] Alert generation and notification

[0411] server

[0412] If an abnormality in cognitive function or emotional state is detected based on the analysis results, the server generates a warning message.

[0413] The generated warning message is sent to the elderly person's family or caregiver.

[0414] Specific examples

[0415] Alert message: "An older adult has recently shown signs of inconsistency and anxiety in their conversations, so immediate medical attention is recommended."

[0416] Report generation and delivery

[0417] server

[0418] The server periodically (e.g., weekly or monthly) generates detailed reports based on the elderly person's conversational and emotional data.

[0419] The server sends the generated reports to family members or caregivers via email or a dedicated portal.

[0420] Specific examples

[0421] Report reads: "Analysis of conversations over the past month has revealed inconsistent responses, memory lapses, and feelings of anxiety on multiple occasions."

[0422] In this way, the system of the present invention combines an automated voice conversation system and an emotion engine to assess the cognitive function and emotional state of elderly people, detect abnormalities early, and send appropriate alerts and reports to family members and caregivers so that necessary measures can be taken promptly.

[0423] The processing flow will be explained below.

[0424] Step 1:

[0425] Terminal

[0426] The terminal activates the automated voice conversation system at a preset time.

[0427] The terminal speaks predetermined questions to the elderly person, such as "What did you do today?" or "What is your grandchild's name?"

[0428] The user (elderly person) answers these questions verbally.

[0429] Execution example:

[0430] Terminal: "What did you do today?"

[0431] User: "Today I went to the park with my friends."

[0432] Step 2:

[0433] Terminal

[0434] The terminal records the user's spoken response.

[0435] The recorded voice data is converted into text using the device's built-in natural language processing (NLP) engine.

[0436] Prepare the converted text data to be sent to the server.

[0437] Execution example:

[0438] Audio data: "Today I went to the park with my friends."

[0439] Text data: "Today I went to the park with my friends."

[0440] Step 3:

[0441] Terminal

[0442] The emotion engine built into the device recognizes emotions from recorded voice data in real time.

[0443] The recognized emotion data is added to the text data and transmitted to the server.

[0444] Execution example:

[0445] Emotion data sent along with the voice data: "Joy"

[0446] Step 4:

[0447] server

[0448] The server receives the text data and emotion data sent from the terminal.

[0449] The received data is stored in a database for each elderly person.

[0450] Execution example:

[0451] Server: "New speech and emotion data saved to database."

[0452] Step 5:

[0453] server

[0454] The server retrieves the stored text data and emotion data for analysis.

[0455] The server applies natural language processing algorithms to assess memory ability and cognitive function from text data.

[0456] Emotional data will also be analyzed to assess the transition of the elderly person's emotional state.

[0457] Things to check:

[0458] Check for contradictory answers.

[0459] Check for memory lapses (e.g., forgetting the names of grandchildren).

[0460] Detecting confused conversation flow.

[0461] Check to see if certain emotions (e.g., anxiety, anger) are occurring frequently.

[0462] Step 6:

[0463] server

[0464] Based on the analysis results, the server generates a warning message if abnormalities are detected in memory ability, cognitive function, or emotional state.

[0465] Notify family members or caregivers of the generated warning message.

[0466] Execution example:

[0467] Alert message: "An older adult has recently shown signs of inconsistency and anxiety in their conversations, so immediate medical attention is recommended."

[0468] Step 7:

[0469] server

[0470] The server periodically (for example, weekly or monthly) generates detailed reports based on the elderly person's conversational and emotional data.

[0471] Generated reports are sent to family members or caregivers via email or a dedicated portal.

[0472] Execution example:

[0473] Report reads: "Analysis of conversations over the past month has revealed inconsistent responses, memory lapses, and feelings of anxiety on multiple occasions."

[0474] Through these steps, the system of the present invention can routinely monitor the cognitive function and emotional state of elderly people and detect abnormalities early. Appropriate alerts and reports can be sent to family members and caregivers, enabling them to take prompt action to ensure the quality of life and safety of the elderly.

[0475] Example 2

[0476] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0477] It is difficult to routinely assess the cognitive function and emotional state of elderly people and detect abnormalities early. In particular, there is a need for a system that can accurately detect cognitive decline and emotional changes through everyday conversations and promptly notify family members and caregivers. There is also a need for detailed data analysis and reports that enable appropriate responses tailored to the individual condition of each elderly person.

[0478] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for sending periodically generated reports to family members or caregivers, a means for generating and notifying different levels of warnings, a means for analyzing received text data and emotional data to evaluate memory ability and cognitive function, and a means for analyzing emotional data to evaluate emotional state. This makes it possible to quickly and accurately detect changes in the elderly person's cognitive function and emotions, to promptly notify if an abnormality is found, and to provide a detailed report.

[0479] An "elderly terminal" is a device that can carry out voice conversations with the elderly, record audio, convert text, recognize emotions, and transmit data.

[0480] "Predetermined time" refers to a specific date, time or time period for which the system is set.

[0481] "Start speaking" means that the terminal automatically starts a voice conversation.

[0482] The "predetermined question" refers to a question that is set in advance in the system.

[0483] "Recording a speech response" means that the terminal records the voice of the elderly person's response.

[0484] "Natural language processing technology" is a general term for computer technology for converting voice data into text data.

[0485] "Text data" is recorded voice data converted into text information.

[0486] "Emotion data" refers to information about emotions recognized from voice data.

[0487] "Central Processing Unit" is the main computer that processes and analyzes the received data and notifies the results.

[0488] "Analyzing" means analyzing data in detail and extracting meaning and patterns.

[0489] "Memory ability" refers to a subject's ability to accurately recall past information.

[0490] "Cognitive function" is a general term for intellectual functions such as thinking, understanding, learning, and memory.

[0491] "Emotional state" refers to the subject's current emotional state.

[0492] "Generating an alert" means creating a notification mechanism when an anomaly is detected.

[0493] "Notify" refers to sending the generated alert to a family member or caregiver.

[0494] "Generating a report" means creating a report based on the analysis results.

[0495] The present invention is a system for evaluating cognitive function and emotional state through voice conversation with elderly people and detecting abnormalities at an early stage. Specific embodiments of the present invention will be described below.

[0496] System configuration

[0497] The system of the present invention includes an elderly terminal, a central processing unit, and related notification means, and also incorporates an emotion engine for emotion recognition.

[0498] Devices for the elderly

[0499] The elderly terminal has a function to automatically start the voice conversation system at a predetermined time. For example, the system starts up at 9:00 AM every day and starts a conversation with the elderly. The terminal asks everyday questions such as:

[0500] For example: "What did you do today?"

[0501] For example: "What is your grandchild's name?"

[0502] The user (elderly person) answers these questions verbally, and the device records the voice. The recorded voice data is passed through an emotion engine built into the device to recognize emotions in real time. For example, if the user answers, "I went to the park with my friends today," the emotion of joy is recognized from the tone of the voice.

[0503] Audio data conversion and transmission

[0504] The recorded voice data is converted into text data using natural language processing (NLP) technology, using software such as the Google Speech-to-Text API and DeepSpeech (an open-source library developed by Mozilla). Emotion data is generated along with the text data, and this data is sent to a central processing unit (server).

[0505] Central Processing Unit (Server)

[0506] The server receives the text and emotion data sent from the device. The received data is securely stored in a database (e.g., MySQL database) for each elderly person. The server then retrieves the stored data for analysis and uses natural language processing algorithms (e.g., SpaCy) to evaluate memory ability and cognitive function. The analysis includes the following points:

[0507] Conflicting answers

[0508] Memory deficiencies (e.g., forgetting grandchildren's names)

[0509] Confused conversation flow

[0510] Furthermore, the emotional data recognized by the emotion engine is also analyzed to evaluate abnormalities in the emotional state.

[0511] Alerts and Notifications

[0512] If the server detects any abnormalities in cognitive function or emotional state based on the analysis results, it will generate a warning message and notify the family or caregiver, such as, "The elderly person's recent conversations have shown inconsistencies and feelings of anxiety, so we recommend that you seek medical attention immediately."

[0513] Report generation and delivery

[0514] Periodically (e.g., weekly or monthly), the server generates detailed reports based on the elderly person's conversation and emotion data. A report generation tool (e.g., Tableau) is used to generate the reports. The generated reports are sent to family members or caregivers via email or a dedicated portal. For example, a report such as, "Conversation analysis results for the past month. Inconsistent answers, memory lapses, and feelings of anxiety were observed in multiple situations."

[0515] Examples of prompt statements

[0516] Using a generative AI model, you can input prompts like this:

[0517] "Describe a natural language processing algorithm for analyzing speech data from older adults and assessing their emotional state."

[0518] The system of the present invention combines an automated voice conversation system and an emotion engine to assess the cognitive function and emotional state of elderly people, detect abnormalities early, and send appropriate alerts and reports to family members and caregivers, enabling them to take necessary measures promptly.

[0519] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0520] Step 1: Voice conversation execution and emotion recognition (device)

[0521] The terminal activates the automated voice conversation system at a predetermined time.

[0522] Input: A specific time (e.g., 9:00 AM every day)

[0523] What it does: Your device will speak everyday questions like, "What did you do today?"

[0524] Output: The spoken question

[0525] The user verbally answers questions posed by the terminal.

[0526] Input: Terminal question

[0527] Action: The user responds, "Today I went to the park with my friends."

[0528] Output: Audio data (user's response)

[0529] The terminal records the user's response in audio format.

[0530] Input: User's voice response

[0531] How it works: The device's microphone captures audio data and saves it as a recording.

[0532] Output: Recorded audio data

[0533] The device's emotion engine analyzes the recorded voice data in real time and recognizes emotions.

[0534] Input: Recorded audio data

[0535] How it works: Emotion recognition algorithms analyze voice tone, pitch, speed, etc.

[0536] Output: Emotion data (e.g., "joy")

[0537] Step 2: Convert and send audio data (terminal)

[0538] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[0539] Input: Recorded audio data

[0540] What it does: Converts audio data into text using the Google Speech-to-Text API.

[0541] Output: Text data (e.g., "Today I went to the park with my friends.")

[0542] The terminal transmits the converted text data and emotion data to the server.

[0543] Input: Text data, emotion data

[0544] Operation: The terminal packets the data and sends it to the server over the network.

[0545] Output: Data sent to the server

[0546] Step 3: Receiving and storing data (server)

[0547] The server receives the text data and emotion data sent from the terminal.

[0548] Input: Text data, emotion data

[0549] How it works: The server captures packets on the network and analyzes the data.

[0550] Output: Received data

[0551] The server securely stores the received data in a database for each elderly person.

[0552] Input: Received data

[0553] What it does: Saves data to a MySQL database.

[0554] Output: Securely stored data

[0555] Step 4: Data analysis (server)

[0556] The server retrieves the stored text data and emotion data for analysis.

[0557] Input: saved text data, emotion data

[0558] What it does: Reads data from a database.

[0559] Output: Data to be analyzed

[0560] The server analyzes the text data using natural language processing algorithms (e.g., SpaCy) to assess memory ability and cognitive function.

[0561] Input: Data to be analyzed

[0562] How it works: It applies natural language processing algorithms to detect inconsistencies, memory lapses, and confused conversation flow.

[0563] Output: Analysis results (e.g., "Inconsistencies detected," "Memory gaps observed")

[0564] The server also analyzes the emotion data recognized by the emotion engine and evaluates abnormalities in the emotional state.

[0565] Input: Emotion data

[0566] How it works: Analyzes emotion data and detects abnormal emotional states (e.g., "persistent anxiety").

[0567] Output: Emotion analysis results

[0568] Step 5: Alert Generation and Notification (Server)

[0569] If an abnormality in cognitive function or emotional state is detected based on the analysis results, the server generates a warning message.

[0570] Input: Analysis results, emotion analysis results

[0571] Operation: Generates a warning message and determines the content of the message depending on the severity of the abnormality.

[0572] Output: Generated warning message

[0573] The server sends the generated alert message to the family member or caregiver.

[0574] Input: warning message

[0575] Action: Sends notifications via email and SMS.

[0576] Output: Warning message sent

[0577] Step 6: Generate and send report (server)

[0578] The server periodically generates detailed reports based on the elderly person's speech and emotion data.

[0579] Input: Saved conversation data, emotion data

[0580] Behavior: Use a report generation tool (e.g., Tableau) to visualize data and create reports.

[0581] Output: Generated report

[0582] The server sends the generated report to the family member or caregiver.

[0583] Input: Generated report

[0584] How it works: Reports are delivered via email and a dedicated portal.

[0585] Output: Report sent

[0586] (Application example 2)

[0587] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0588] Systems that assess the cognitive function and emotional state of elderly people need to be able to detect abnormalities in real time and quickly notify family members or caregivers, ensuring the safety of the elderly. Conventional systems have difficulty assessing the elderly's emotional state in detail and notifying them in a timely manner. Furthermore, there is a need for integration into a home security system to monitor the safety of elderly people at home.

[0589] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing received text data and evaluating memory ability and cognitive function, means for analyzing emotion data and evaluating emotional abnormalities, and means for generating and notifying an alert when an abnormality is detected based on the analysis results. This makes it possible to detect abnormalities in the cognitive function and emotional state of an elderly person in real time and quickly notify family members or security service operators.

[0590] An "elderly terminal" is an information processing terminal designed for use by elderly people, and is a device that includes voice recognition and emotion recognition functions.

[0591] The "predetermined time" refers to a specific time period that has been set in advance, during which the system will automatically operate.

[0592] "Means for initiating speech and asking specific questions" refers to the function of the elderly terminal to automatically initiate a voice dialogue and ask specific questions to the elderly.

[0593] "Means for recording spoken responses" refers to the function of the elderly person terminal to record the elderly person's responses in audio format.

[0594] "Means for converting recorded voice data into text data" refers to a function that converts recorded voice into text data using voice recognition technology.

[0595] "Means for recognizing emotions" refers to the function of analyzing and recognizing the speaker's emotional state from voice data.

[0596] "Means for transmitting text data and emotion data to a central processing unit" refers to a function for transmitting converted text and analyzed emotion information to a remote server.

[0597] The "central processing unit" refers to a server or the like that receives data sent from the elderly person's terminal and performs analytical processing.

[0598] "Means for analyzing text data to assess memory ability and cognitive function" refers to an algorithm that analyzes received text data to assess the memory ability and cognitive function of elderly people.

[0599] "Means for analyzing emotional data and evaluating emotional abnormalities" refers to a function for analyzing received emotional data and detecting abnormalities in the emotional state.

[0600] "Means for generating and notifying an alert when an abnormality is detected based on the analysis results" refers to the function of generating an alert and notifying family members or caregivers when an abnormality is detected based on the results of data analysis.

[0601] "Means for sending as a report" refers to the function of periodically sending a report summarizing the analysis results to family members or caregivers.

[0602] System configuration

[0603] The system required to implement this invention includes an "elderly terminal," a "central processing unit (server)," and a "notification means." Furthermore, it incorporates engines for speech recognition, natural language processing (NLP), and emotion recognition. Specifically, it uses the Google Cloud Speech-to-Text API for speech recognition, the Microsoft® Emotion API or Amazon Rekognition for emotion recognition, and Firebase or Amazon DynamoDB for the database.

[0604] Process Overview

[0605] 1. Start a voice interaction:

[0606] The elderly device automatically starts up at a specified time and asks the elderly specific questions such as "What did you do today?" or "What is your grandchild's name?" These questions are spoken using voice synthesis technology.

[0607] 2. Audio data collection and text conversion:

[0608] When the elderly responds verbally to these questions, the device records their responses and then converts the recorded audio data into text using the Google Cloud Speech-to-Text API.

[0609] 3. Emotion recognition:

[0610] At the same time, an emotion engine (e.g., Amazon Rekognition) is used to recognize the elderly person's emotions from the voice data in real time, and the recognition results are saved as emotion data.

[0611] 4. Data transmission and storage:

[0612] The converted text data and emotion data are sent from the elderly person's device to a central processing unit (server), which receives the data and stores it securely in a database such as Firebase.

[0613] 5. Data Analysis:

[0614] The central processing unit retrieves the stored data and analyzes the text data to assess memory and cognitive function, specifically for inconsistent responses and memory lapses, and analyzes the emotional data to assess abnormal emotional states.

[0615] 6. Alert Generation and Notification:

[0616] If an abnormality is detected based on the analysis results, the system automatically generates an alert and notifies family members, caregivers, and in some cases security service operators via an SMS API (such as Twilio) or Firebase Cloud Messaging.

[0617] 7. Report generation and sending:

[0618] Detailed reports based on the elderly person's conversation and emotion data are generated periodically (e.g., weekly or monthly) and sent to family members or caregivers via email or a dedicated portal.

[0619] Specific examples

[0620] For example, if an elderly person says, "Today I went to the park with my friends," the system records the audio data and then uses the Google Cloud Speech-to-Text API to convert it into text data: "Today I went to the park with my friends." At the same time, Amazon Rekognition is used to identify the emotion as "joy" and send this data to the server. The server analyzes the text data and emotion data and evaluates whether there are any anomalies. If an anomaly is detected, for example, if the emotion "anxiety" persists, the system uses Twilio's SMS API to send an alert to the family member stating, "Anxiety has been detected in the elderly person's recent conversations, and immediate attention is required."

[0621] Prompt Sentence Examples

[0622] "What did you do today?"

[0623] "What's your grandson's name?"

[0624] "How are you feeling today?"

[0625] Through these example prompts, the system of the invention can efficiently assess the cognitive function and emotional state of elderly people and quickly provide this information to their families and caregivers.

[0626] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0627] Step 1:

[0628] The elderly terminal begins speaking at a predetermined time. Using speech synthesis technology, the elderly terminal speaks prompt sentences such as "What did you do today?" or "What is your grandchild's name?" The input is the prompt sentence, and the output is a voice response from the elderly person.

[0629] Step 2:

[0630] The elderly person's speech response is recorded on the elderly person terminal. The input is the elderly person's voice data, and the output is the recorded voice file.

[0631] Step 3:

[0632] The recorded voice data is converted into text data. The elderly device converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is an audio file, and the output is text data.

[0633] Step 4:

[0634] At the same time, the elderly device uses an emotion engine (e.g., Amazon Rekognition) to recognize emotions from voice data. The input is the voice file, and the output is emotion data.

[0635] Step 5:

[0636] The elderly person's device sends the converted text data and emotion data to the server. The input is the text data and emotion data, and the output is the data sent to the server.

[0637] Step 6:

[0638] The server analyzes the received text data and evaluates memory ability and cognitive function. Specifically, it checks for inconsistent answers and memory deficiencies. The input is text data, and the output is the evaluation results of memory ability and cognitive function.

[0639] Step 7:

[0640] The server analyzes the emotional data and evaluates the abnormality of the emotional state. The input is the emotional data, and the output is the evaluation result of the emotional state.

[0641] Step 8:

[0642] If the server detects an abnormality based on the analysis results, it generates and notifies a warning. When an abnormality is detected, it notifies family members or security service operators using Twilio's SMS API or Firebase Cloud Messaging. The input is the evaluation result, and the output is the notified warning message.

[0643] Step 9:

[0644] The server periodically generates detailed reports based on the elderly person's conversation and emotion data and sends them to family members or caregivers via email or a dedicated portal. The input is the analyzed data, and the output is the sent report.

[0645] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0646] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0647] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0648] [Second embodiment]

[0649] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0650] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0651] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0652] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0653] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0654] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0655] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0656] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0657] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0658] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0659] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0660] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0661] System Overview

[0662] The system of the present invention combines an automated voice conversation system with data analysis algorithms to assess cognitive function and detect abnormalities in elderly people. The system includes an elderly person's terminal (hereinafter referred to as "terminal"), a central processing unit (hereinafter referred to as "server"), and related notification means.

[0663] Program processing flow

[0664] Conducting a voice conversation

[0665] Terminal

[0666] The terminal automatically starts the automated voice conversation system at a preset time.

[0667] The device speaks everyday questions to the elderly, such as "What did you do today?" and "What is your grandchild's name?"

[0668] When the elderly person answers these questions verbally, the device records the answers.

[0669] Specific examples

[0670] Terminal: "What did you do today?"

[0671] User: "Today I went to the park with my friends."

[0672] Audio data conversion and transmission

[0673] Terminal

[0674] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[0675] The converted text data is sent to the server.

[0676] Specific examples

[0677] Audio data: "Today I went to the park with my friends."

[0678] Text data: "Today I went to the park with my friends."

[0679] Receiving and storing data

[0680] server

[0681] The server receives the text data sent from the terminal.

[0682] The received text data is securely stored in a database for each elderly person.

[0683] Specific examples

[0684] Server: "New conversation data saved to database."

[0685] Data analysis

[0686] server

[0687] The server retrieves the stored text data for analysis.

[0688] The server applies algorithms to assess the elderly person's cognitive function, checking their memory ability and coherence.

[0689] Check the following points:

[0690] Conflicting answers

[0691] Memory lapses (e.g., forgetting grandchildren's names)

[0692] Confused conversation flow

[0693] Specific examples

[0694] Conversation log: "Your grandchildren came over yesterday." "No, no one came." (Inconsistency resolved)

[0695] Alert generation and notification

[0696] server

[0697] Based on the analysis results, the server generates a warning message if an abnormality is detected.

[0698] The server notifies the family or caregiver of the generated warning message.

[0699] Specific examples

[0700] Alert message: "An elderly person has recently exhibited inconsistencies in their conversations and urgent medical attention is recommended."

[0701] Report generation and delivery

[0702] server

[0703] The server periodically generates a report based on the analysis of the elderly person's conversation data.

[0704] The server sends the generated reports to family members or caregivers via email or a dedicated portal.

[0705] Specific examples

[0706] Report reads: "Analysis of conversations over the past month. Multiple inconsistencies and lapses in memory have been observed."

[0707] In this way, the system of the present invention utilizes an automated voice conversation system and data analysis algorithms to assess the cognitive function of elderly people and detect abnormalities early, and by sending appropriate alerts and reports to family members and caregivers, they can take necessary measures promptly.

[0708] The processing flow will be explained below.

[0709] Step 1:

[0710] Terminal

[0711] The terminal activates the automated voice conversation system at a preset time.

[0712] The device will ask the elderly everyday questions aloud, such as "What did you do today?" and "What is your grandchild's name?"

[0713] The user (elderly person) answers these questions orally, and the terminal records the answers in audio format.

[0714] Step 2:

[0715] Terminal

[0716] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[0717] It uses advanced speech recognition algorithms to generate accurate text data from audio data.

[0718] Step 3:

[0719] Terminal

[0720] The terminal transmits the converted text data to the server.

[0721] The text data also includes the elderly person's identification information and the date and time of the conversation.

[0722] Step 4:

[0723] server

[0724] The server receives the text data sent from the terminal.

[0725] The server securely stores the received text data in a database for each elderly person.

[0726] Step 5:

[0727] server

[0728] The server retrieves the stored text data for analysis.

[0729] The server applies natural language processing algorithms to assess memory and cognitive function, specifically checking the following points:

[0730] Conflicting answers

[0731] Memory deficiencies (e.g., forgetting grandchildren's names)

[0732] Confused conversation flow

[0733] Step 6:

[0734] server

[0735] If an abnormality is detected based on the analysis results, the server generates a warning message.

[0736] The generated warning message is sent to the elderly person's family or caregiver.

[0737] Step 7:

[0738] server

[0739] The server periodically (e.g., weekly or monthly) generates detailed reports based on the elderly person's conversation data.

[0740] The server sends the generated reports to family members or caregivers via email or a dedicated portal.

[0741] Through this series of steps, the system will enable daily monitoring of the cognitive function of elderly people, enabling early detection of abnormalities and follow-up.

[0742] Example 1

[0743] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0744] There is a need to detect cognitive decline in the elderly early and provide appropriate care promptly. Conventional methods often rely on home observation and regular medical examinations, which can easily miss cognitive decline. Furthermore, it is difficult to promptly notify family members or caregivers when abnormalities are detected and provide instructions for appropriate care. This reduces the quality of life of the elderly and increases the risk of inadequate care.

[0745] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0746] In this invention, the server includes: means for the elderly person terminal to start speaking at a predetermined time and ask predetermined questions; means for recording the elderly person's spoken responses; means for converting the recorded voice data into text data; means for transmitting the text data to a central processing unit; means for analyzing the received text data and evaluating memory ability and cognitive function; means for generating and notifying warnings if abnormalities are detected based on the analysis results; means for periodically generating reports based on the analysis results of the conversation data and sending them to relevant parties; means for automatically starting the conversation system at a predetermined time using scheduling software; means for analyzing the text data using natural language processing technology and evaluating consistency and memory ability for multiple situations; means for automatically generating and notifying warning messages of various levels based on abnormalities in the analysis results; and means for periodically reporting details of detected abnormalities to relevant parties. This enables automatic and continuous evaluation of the elderly person's cognitive function, early detection of abnormalities, and prompt notification and detailed reports to relevant parties.

[0747] An "elderly terminal" is a device with voice conversation and recording functions designed for use by elderly people.

[0748] The "predetermined time" refers to a specific time that has been set in advance, and is the reference time at which the system automatically starts operating.

[0749] "Speaking" refers to the act of the elderly person terminal asking a question to the elderly person by voice.

[0750] "Spoken response" refers to a verbal response given by the elderly person to the elderly person terminal.

[0751] "Recording" is the act of recording a speech response as audio data.

[0752] "Voice data" refers to digital voice information obtained by recording the elderly person's speech responses.

[0753] "Text data" is voice data converted into text information using natural language processing technology.

[0754] A "central processing unit" is a computer system that analyzes and evaluates received data, and detects and notifies abnormalities.

[0755] "Analysis" refers to the act of evaluating the memory ability and cognitive function of elderly people based on the received text data.

[0756] "Memory ability and cognitive function assessment" refers to the act of checking the consistency of an elderly person's answers and memory deficiencies, and dynamically assessing the state of their cognitive function.

[0757] "Abnormalities" refer to problems with the elderly person's cognitive function that are different from normal, such as inconsistencies or memory loss.

[0758] A "warning" is an alert message that is generated when an abnormality is detected, and is a notification that prompts a prompt response.

[0759] A "report" is a document summarizing the results of analysis of elderly people's conversation data, and is sent to relevant parties on a regular basis.

[0760] "Scheduling software" is a program that automatically starts system operations at pre-set times.

[0761] "Natural language processing technology" is a technology that converts voice data into text information and analyzes text data.

[0762] "Analysis results" are the results of evaluations and judgments made based on text data.

[0763] "Stakeholders" are people or organizations with an interest in the condition of older people, such as their families, caregivers, and medical professionals.

[0764] MODE FOR CARRYING OUT THE INVENTION

[0765] The system of the present invention utilizes an automated voice conversation system and data analysis algorithms to assess cognitive function in elderly people and detect abnormalities at an early stage. The system includes an elderly person terminal (hereinafter referred to as "terminal"), a central processing unit (hereinafter referred to as "server"), and related notification means.

[0766] Conducting a voice conversation

[0767] Terminal

[0768] The terminal automatically starts the automated voice conversation system at a preset time, using scheduling software such as a Cron job for scheduling.

[0769] The device asks everyday questions to the elderly, such as "What did you do today?" and "What is your grandchild's name?", using Google Text-to-Speech technology.

[0770] When the elderly responds to these questions verbally, the device uses a built-in microphone to record the responses, which are then temporarily stored in local storage.

[0771] Specific examples

[0772] Terminal: "What did you do today?"

[0773] User: "Today I went to the park with my friends."

[0774] Audio data conversion and transmission

[0775] Terminal

[0776] The device converts the recorded voice data into text data using natural language processing (NLP) technology, which uses the Google Cloud Speech-to-Text API.

[0777] After converting the data into text, the device sends the text to a remote server using an HTTP POST request, with end-to-end encryption applied to the communication.

[0778] Specific examples

[0779] Audio data: "Today I went to the park with my friends."

[0780] Text data: "Today I went to the park with my friends."

[0781] Receiving and storing data

[0782] server

[0783] The server receives the HTTP POST request sent from the device and extracts the text data from it, using the Apache or Nginx web server software.

[0784] The extracted text data is securely stored in a database for each elderly person. MySQL or PostgreSQL is used as the database management system, and data encryption is performed.

[0785] Specific examples

[0786] Server: "New conversation data saved to database."

[0787] Data analysis

[0788] server

[0789] The server periodically retrieves the stored text data for analysis, using a Cron job.

[0790] The stored text data is then used to run an algorithm that assesses the cognitive function of older adults. This algorithm uses natural language processing (NLP) techniques and libraries such as Python's NLTK and spaCy.

[0791] During processing, check for inconsistent answers, memory gaps, and confused conversation flow.

[0792] Specific examples

[0793] Conversation log: "Your grandchildren came over yesterday." "No, no one came." (Inconsistency resolved)

[0794] Alert generation and notification

[0795] server

[0796] If an abnormality is detected based on the analysis results, the server generates a warning message. This message is automatically generated using a template.

[0797] The server then sends the generated warning message to family members or caregivers via email, SMS, or a dedicated app. Email is sent using the SMTP protocol, and SMS is sent using services such as Twilio.

[0798] Specific examples

[0799] Alert message: "An elderly person has recently exhibited inconsistencies in their conversations and urgent medical attention is recommended."

[0800] Report generation and delivery

[0801] server

[0802] The server periodically generates reports based on the analysis of the elderly's conversation data. The reports are generated in PDF format using libraries such as Python's ReportLab.

[0803] The server generates reports and sends them to family members or caregivers via email or a dedicated portal, which can be run periodically using a scheduler.

[0804] Specific examples

[0805] Report reads: "Analysis of conversations over the past month. Multiple inconsistencies and lapses in memory have been observed."

[0806] Example prompts to input to the generative AI model

[0807] Examples:

[0808] Terminal: "What did you do today?"

[0809] User: "Today I went to the park with my friends."

[0810] Example prompt sentence:

[0811] "Please suggest ways to predict how the elderly person will respond to the device and ensure consistency in the next conversation. Also, please suggest improvements to the algorithm to identify gaps or inconsistencies in memory."

[0812] The system of this invention can effectively evaluate the cognitive function of elderly people and detect abnormalities early. Detailed information can be obtained through conversational interaction, and appropriate nursing and medical measures can be taken in real time based on that information. In addition, appropriate alerts and regular reports enable prompt and accurate responses.

[0813] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0814] Step 1:

[0815] Starting a voice conversation

[0816] The terminal automatically starts the automated voice conversation system at a preset time, using scheduling software such as a Cron job for scheduling.

[0817] Input: Scheduling time

[0818] Operation: The device monitors its internal clock and starts the voice conversation system at the set time.

[0819] Output: Voice conversation system activated

[0820] Step 2:

[0821] Asking a question

[0822] The device uses Google Text-to-Speech technology to ask everyday questions to the elderly, such as "What did you do today?" or "What is your grandchild's name?"

[0823] Input: Preset question content

[0824] How it works: The device passes preset question text to the Google Text-to-Speech API, generates audio data, and speaks it to the elderly person.

[0825] Output: Question spoken to the elderly person

[0826] Step 3:

[0827] Spoken response recording

[0828] When the elderly responds verbally to the questions, the device uses a built-in microphone to record the response, which is then temporarily stored in local storage.

[0829] Input: Verbal response from the elderly person

[0830] How it works: The device captures audio data through the microphone and stores it in local storage.

[0831] Output: Recorded audio data file

[0832] Step 4:

[0833] Converting audio data to text

[0834] The device converts the recorded voice data into text data using natural language processing (NLP) technology, which uses the Google Cloud Speech-to-Text API.

[0835] Input: Recorded audio data

[0836] How it works: The device sends audio data to the Google Cloud Speech-to-Text API and retrieves text data.

[0837] Output: Converted text data

[0838] Step 5:

[0839] Sending text data

[0840] The device then sends the converted text data to a remote server using an HTTP POST request, with end-to-end encryption applied to the communication.

[0841] Input: Converted text data

[0842] How it works: The device generates an HTTP POST request and sends data to the specified endpoint on the server.

[0843] Output: Text data sent to the server

[0844] Step 6:

[0845] Receiving and storing text data

[0846] The server receives the HTTP POST request sent from the device and extracts the text data from it, using the Apache or Nginx web server software.

[0847] Input: HTTP POST request

[0848] What happens: The server parses the POST request and extracts the text data.

[0849] Output: Extracted text data

[0850] Step 7:

[0851] Data storage

[0852] The server securely stores the text data for each elderly person in a database, using MySQL or PostgreSQL as the database management system, and the data is encrypted.

[0853] Input: Extracted text data

[0854] Operation: The server saves the text data in a database. When saved, it associates it with the elderly person's identification information.

[0855] Output: Securely stored text data

[0856] Step 8:

[0857] Regular analysis of data

[0858] The server periodically retrieves the stored text data for analysis, using a Cron job.

[0859] Input: Text data stored in a database

[0860] Operation: The server periodically retrieves the stored text data and runs an analysis program.

[0861] Output: Text data extracted for analysis

[0862] Step 9:

[0863] Cognitive function assessment

[0864] The server runs an algorithm on the text data to assess cognitive function in older adults, using natural language processing (NLP) techniques and libraries such as Python's NLTK and spaCy.

[0865] Input: Text data extracted for analysis

[0866] How it works: The server runs a program that detects inconsistent answers, memory lapses, and disorganized conversation flow.

[0867] Output: Cognitive function assessment results

[0868] Step 10:

[0869] Anomaly detection and alert generation

[0870] If an abnormality is detected based on the analysis results, the server generates a warning message. This message is automatically generated using a template.

[0871] Input: Cognitive function assessment results

[0872] Operation: The server analyzes the evaluation results and generates a corresponding warning message if an anomaly is detected.

[0873] Output: Warning message

[0874] Step 11:

[0875] Warning Notification

[0876] The server then sends the generated warning message to family members or caregivers via email, SMS, or a dedicated app. Email is sent using the SMTP protocol, and SMS is sent using services such as Twilio.

[0877] Input: warning message

[0878] What it does: The server generates email and / or SMS notifications and sends them to interested parties.

[0879] Output: Warning message sent to family members or caregivers

[0880] Step 12:

[0881] Report generation and delivery

[0882] The server periodically generates reports based on the analysis of the elderly's conversation data. The reports are generated in PDF format using libraries such as Python's ReportLab.

[0883] Input: Analysis results of elderly people's conversation data

[0884] How it works: The server periodically aggregates the analysis results of the conversation data and generates a report in PDF format.

[0885] Output: Generated report

[0886] Step 13:

[0887] Report Notification

[0888] The server then sends the generated reports to family members or caregivers via email or a dedicated portal, and this process is also performed periodically using a scheduler.

[0889] Input: Generated report

[0890] How it works: The server sends reports via email or a dedicated portal.

[0891] Output: Report sent to family or caregiver

[0892] (Application example 1)

[0893] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0894] In modern society, efficient and accurate assessment of the cognitive function of elderly people and early detection of cognitive decline occurring in daily life are important challenges. However, conventional interactive systems lack a well-established method for thoroughly assessing the cognitive function of elderly people through their everyday conversations and appropriately notifying the results. Furthermore, they lack a means for quickly issuing warnings when abnormalities are detected and providing necessary information to family members and caregivers. Furthermore, they lack the ability to regularly store and analyze data to ensure the safety of elderly people. Therefore, the present invention aims to solve these challenges.

[0895] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0896] In this invention, the server includes a means for notifying family members and caregivers of speech-recognized text data, a means for periodically saving and analyzing daily conversation data, a means for analyzing the elderly person's conversation data using natural language processing technology, a means for sending warning messages regarding detected abnormalities to family members and caregivers, and a means for sending generated periodic reports to family members and caregivers. This enables detailed assessment of the elderly person's cognitive function through their daily conversations, early detection of abnormalities, and appropriate alert notifications. Furthermore, periodic data saving and analysis can provide comprehensive support for ensuring the safety of the elderly.

[0897] An "elderly terminal" is an electronic device for use by elderly people that is capable of voice input and output.

[0898] A "central processing unit" is a centralized device for analyzing and evaluating speech-recognized data, and functions as a server.

[0899] The "predetermined time" refers to a specific time or period that has been set in advance, at which the system automatically starts operating.

[0900] "Start speaking" means that the system will output pre-programmed messages and ask questions to the senior.

[0901] The "prescribed questions" are questions about the elderly person's daily life and memory, and are useful for assessing cognitive function.

[0902] "Spoken responses" refer to verbal responses given by the elderly to questions posed by the system.

[0903] "Means for recording" refers to a function for recording the elderly person's spoken responses as digital audio data.

[0904] "Means for converting into text data" refers to technology for converting recorded voice data into text information, and primarily utilizes natural language processing technology.

[0905] "Means of analysis" refers to algorithms for assessing the cognitive function of elderly people based on text data.

[0906] "Means for detecting abnormalities" refers to the function of detecting abnormalities such as a decline in cognitive function from the analysis results and instructing on necessary measures.

[0907] "Means for generating and notifying alerts" refers to a system that has the ability to generate alert messages regarding detected abnormalities and promptly notify family members or caregivers.

[0908] "Natural language processing technology" is a technology for processing voice and text data to understand and generate human language.

[0909] A "regular report" is a report summarizing the results of periodic analyses, and is sent to family members or caregivers.

[0910] "Daily conversation data" is a record of regular voice interactions between the elderly person and the system.

[0911] A "generative AI model" is a collection of algorithms trained to perform specific tasks using artificial intelligence.

[0912] A "prompt" is a question or instruction input to a generative AI model, and is a phrase used to encourage a response from the elderly person.

[0913] System Overview

[0914] The system of the present invention mainly consists of an elderly person's terminal, a central processing unit (server), and a notification means. The elderly person's terminal is equipped with an automated voice conversation system that evaluates the elderly person's cognitive function through everyday conversation. The central processing unit analyzes the received data and generates a warning message to notify family members or caregivers if an abnormality is detected. It also has a function to periodically provide analysis results as a report.

[0915] Hardware and software used

[0916] 1. Devices for the elderly:

[0917] Hardware: Smartphone or tablet (including microphone)

[0918] Software: Natural language processing technology (e.g., Google Speech-to-Text API)

[0919] 2. Central Processing Unit (Server):

[0920] Hardware: High-performance network server

[0921] Software: Database systems, data analysis algorithms, notification systems

[0922] Details of data processing and calculation

[0923] Device operation

[0924] The device will automatically start a voice conversation at a specific time. For example, it will ask questions such as "What did you do today?" or "What is your grandchild's name?" When the elderly person answers these questions, the device will record the voice data.

[0925] Audio data conversion and transmission

[0926] The recorded voice data is converted into text data using natural language processing technology within the device. This converted text data is sent to a server. For example, if an elderly person replies, "Today, I went to the park with my friends," the voice is converted into text data that reads, "Today, I went to the park with my friends."

[0927] Data reception and analysis

[0928] The server receives the text data sent from the device. The received data is stored in a database and analysis begins. The server applies algorithms to assess the elderly person's cognitive function and checks their memory ability and coherence. Analysis detects inconsistent answers, memory gaps, and disorganized conversation flow.

[0929] Generate alerts and reports

[0930] If an abnormality is detected based on the analysis results, the server generates a warning message and notifies the family or caregiver. For example, a message may be sent saying, "There have been inconsistencies in the elderly person's recent conversations. We recommend that you seek medical advice as soon as possible." In addition, reports are generated based on the periodic analysis results and sent to the family or caregiver. The report contents may include, "These are the results of the conversation analysis for the past month. Inconsistent answers and lapses in memory have been observed in multiple situations."

[0931] Examples of prompt statements

[0932] Specific examples of prompts include:

[0933] "What did you do today?"

[0934] "What's your grandson's name?"

[0935] "What did you have for dinner last night?"

[0936] This concludes the detailed description of the embodiment of the present invention. This system enables the evaluation of cognitive function through everyday conversations with elderly people and the early detection of abnormalities, allowing family members or caregivers to take prompt action.

[0937] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0938] Processing Steps

[0939] Step 1:

[0940] Starting up elderly device

[0941] Specific operation: The device automatically starts the automated voice conversation system at the specified time, performs initialization processing, and enables the microphone.

[0942] Input: Predetermined time based on system settings

[0943] Output: Automated voice conversation system startup status

[0944] Step 2:

[0945] Prompt sentence utterance

[0946] Specific operation: The device will speak a pre-defined prompt, such as "What did you do today?", to the elderly person.

[0947] Input: Questionnaires saved in the system

[0948] Output: Voice questions for the elderly

[0949] Step 3:

[0950] Spoken response recording

[0951] Specific operation: The elderly person answers questions, and their answers are recorded by the device's microphone.

[0952] Input: Elderly voice response

[0953] Output: Recorded audio data

[0954] Step 4:

[0955] Converting audio data to text

[0956] Specific operation: The recorded audio data is converted into text data using natural language processing technology (such as Google Speech-to-Text API).

[0957] Input: Recorded audio data

[0958] Output: Text data

[0959] Step 5:

[0960] Sending text data

[0961] Specific operation: The converted text data is sent to the server using an HTTP request.

[0962] Input: Converted text data

[0963] Output: Text data sent to the server

[0964] Step 6:

[0965] Receiving and storing data

[0966] Specific operation: The server receives the text data sent from the device and stores it in a database. When stored, the data is classified for each elderly person.

[0967] Input: Text data sent from the terminal

[0968] Output: Text data stored in a database

[0969] Step 7:

[0970] Data analysis

[0971] How it works: The server processes the stored text data using analytical algorithms to assess the elderly person's cognitive function, checking memory ability, consistency of speech, and inconsistencies.

[0972] Input: Text data stored in a database

[0973] Output: Cognitive function assessment results

[0974] Step 8:

[0975] Anomaly detection and alert generation

[0976] Specific operation: If the server detects an anomaly in the analysis results, it generates a warning message. Different levels of warning are set depending on the type and severity of the anomaly.

[0977] Input: Cognitive function assessment results

[0978] Output: Warning message

[0979] Step 9:

[0980] Sending notifications

[0981] Specific operation: The generated warning message is sent to family members or caregivers via email or a dedicated app.

[0982] Input: warning message

[0983] Output: Notification sent to family members and caregivers

[0984] Step 10:

[0985] Generate and send scheduled reports

[0986] Specific operation: The server periodically generates a report based on the analysis results and sends it to the family or caregiver. The report includes the analysis results of the elderly person's conversation data.

[0987] Input: Accumulated data of cognitive function evaluation results

[0988] Output: Regular reports sent to family members and caregivers

[0989] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0990] System Overview

[0991] The system of the present invention assesses cognitive function through voice conversation with elderly people and simultaneously recognizes their emotional state, enabling early detection of abnormalities. The system includes an elderly person terminal (hereinafter referred to as "terminal"), a central processing unit (hereinafter referred to as "server"), and related notification means. It also incorporates an emotion engine that performs emotion recognition.

[0992] Program processing flow

[0993] Voice conversation execution and emotion recognition

[0994] Terminal

[0995] The terminal activates the automated voice conversation system at a preset time.

[0996] The device will ask the elderly everyday questions such as "What did you do today?" or "What is your grandchild's name?"

[0997] When the elderly person answers these questions verbally, the device records the answers in audio format.

[0998] The emotion engine built into the device recognizes emotions in real time from the voice data of the elderly person's speech responses.

[0999] Specific examples

[1000] Terminal: "What did you do today?"

[1001] User: "I went to the park with my friends today." (Emotion of joy is recognized from the tone of voice)

[1002] Audio data conversion and transmission

[1003] Terminal

[1004] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[1005] It uses advanced speech recognition algorithms to generate accurate text data from audio data.

[1006] The converted text data and emotion data are sent to the server.

[1007] Specific examples

[1008] Audio data: "Today I went to the park with my friends."

[1009] Text data: "Today I went to the park with my friends."

[1010] Emotion data: "Joy"

[1011] Receiving and storing data

[1012] server

[1013] The server receives the text data and emotion data sent from the terminal.

[1014] The server securely stores the received data in a database for each elderly person.

[1015] Specific examples

[1016] Server: "New speech and emotion data saved to database."

[1017] Data analysis

[1018] server

[1019] The server retrieves the stored text data and emotion data for analysis.

[1020] The server applies natural language processing algorithms to assess memory and cognitive function, specifically checking the following:

[1021] Conflicting answers

[1022] Memory deficiencies (e.g., forgetting grandchildren's names)

[1023] Confused conversation flow

[1024] The server also analyzes the emotion data recognized by the emotion engine and evaluates the abnormality of the emotional state.

[1025] Specific examples

[1026] Conversation log: "Your grandchildren came over yesterday." "No, no one came." (Inconsistency resolved)

[1027] Emotional data: "Continued anxiety"

[1028] Alert generation and notification

[1029] server

[1030] If an abnormality in cognitive function or emotional state is detected based on the analysis results, the server generates a warning message.

[1031] The generated warning message is sent to the elderly person's family or caregiver.

[1032] Specific examples

[1033] Alert message: "An older adult has recently shown signs of inconsistency and anxiety in their conversations, so immediate medical attention is recommended."

[1034] Report generation and delivery

[1035] server

[1036] The server periodically (e.g., weekly or monthly) generates detailed reports based on the elderly person's conversational and emotional data.

[1037] The server sends the generated reports to family members or caregivers via email or a dedicated portal.

[1038] Specific examples

[1039] Report reads: "Analysis of conversations over the past month has revealed inconsistent responses, memory lapses, and feelings of anxiety on multiple occasions."

[1040] In this way, the system of the present invention combines an automated voice conversation system and an emotion engine to assess the cognitive function and emotional state of elderly people, detect abnormalities early, and send appropriate alerts and reports to family members and caregivers so that necessary measures can be taken promptly.

[1041] The processing flow will be explained below.

[1042] Step 1:

[1043] Terminal

[1044] The terminal activates the automated voice conversation system at a preset time.

[1045] The terminal speaks predetermined questions to the elderly person, such as "What did you do today?" or "What is your grandchild's name?"

[1046] The user (elderly person) answers these questions verbally.

[1047] Execution example:

[1048] Terminal: "What did you do today?"

[1049] User: "Today I went to the park with my friends."

[1050] Step 2:

[1051] Terminal

[1052] The terminal records the user's spoken response.

[1053] The recorded voice data is converted into text using the device's built-in natural language processing (NLP) engine.

[1054] Prepare the converted text data to be sent to the server.

[1055] Execution example:

[1056] Audio data: "Today I went to the park with my friends."

[1057] Text data: "Today I went to the park with my friends."

[1058] Step 3:

[1059] Terminal

[1060] The emotion engine built into the device recognizes emotions from recorded voice data in real time.

[1061] The recognized emotion data is added to the text data and transmitted to the server.

[1062] Execution example:

[1063] Emotion data sent along with the voice data: "Joy"

[1064] Step 4:

[1065] server

[1066] The server receives the text data and emotion data sent from the terminal.

[1067] The received data is stored in a database for each elderly person.

[1068] Execution example:

[1069] Server: "New speech and emotion data saved to database."

[1070] Step 5:

[1071] server

[1072] The server retrieves the stored text data and emotion data for analysis.

[1073] The server applies natural language processing algorithms to assess memory ability and cognitive function from text data.

[1074] Emotional data will also be analyzed to assess the transition of the elderly person's emotional state.

[1075] Things to check:

[1076] Check for contradictory answers.

[1077] Check for memory lapses (e.g., forgetting the names of grandchildren).

[1078] Detecting confused conversation flow.

[1079] Check to see if certain emotions (e.g., anxiety, anger) are occurring frequently.

[1080] Step 6:

[1081] server

[1082] Based on the analysis results, the server generates a warning message if abnormalities are detected in memory ability, cognitive function, or emotional state.

[1083] Notify family members or caregivers of the generated warning message.

[1084] Execution example:

[1085] Alert message: "An older adult has recently shown signs of inconsistency and anxiety in their conversations, so immediate medical attention is recommended."

[1086] Step 7:

[1087] server

[1088] The server periodically (for example, weekly or monthly) generates detailed reports based on the elderly person's conversational and emotional data.

[1089] Generated reports are sent to family members or caregivers via email or a dedicated portal.

[1090] Execution example:

[1091] Report reads: "Analysis of conversations over the past month has revealed inconsistent responses, memory lapses, and feelings of anxiety on multiple occasions."

[1092] Through these steps, the system of the present invention can routinely monitor the cognitive function and emotional state of elderly people and detect abnormalities early. Appropriate alerts and reports can be sent to family members and caregivers, enabling them to take prompt action to ensure the quality of life and safety of the elderly.

[1093] Example 2

[1094] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1095] It is difficult to routinely assess the cognitive function and emotional state of elderly people and detect abnormalities early. In particular, there is a need for a system that can accurately detect cognitive decline and emotional changes through everyday conversations and promptly notify family members and caregivers. There is also a need for detailed data analysis and reports that enable appropriate responses tailored to the individual condition of each elderly person.

[1096] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for sending periodically generated reports to family members or caregivers, a means for generating and notifying different levels of warnings, a means for analyzing received text data and emotional data to evaluate memory ability and cognitive function, and a means for analyzing emotional data to evaluate emotional state. This makes it possible to quickly and accurately detect changes in the elderly person's cognitive function and emotions, to promptly notify if an abnormality is found, and to provide a detailed report.

[1097] An "elderly terminal" is a device that can carry out voice conversations with the elderly, record audio, convert text, recognize emotions, and transmit data.

[1098] "Predetermined time" refers to a specific date, time or time period for which the system is set.

[1099] "Start speaking" means that the terminal automatically starts a voice conversation.

[1100] The "predetermined question" refers to a question that is set in advance in the system.

[1101] "Recording a speech response" means that the terminal records the voice of the elderly person's response.

[1102] "Natural language processing technology" is a general term for computer technology for converting voice data into text data.

[1103] "Text data" is recorded voice data converted into text information.

[1104] "Emotion data" refers to information about emotions recognized from voice data.

[1105] "Central Processing Unit" is the main computer that processes and analyzes the received data and notifies the results.

[1106] "Analyzing" means analyzing data in detail and extracting meaning and patterns.

[1107] "Memory ability" refers to a subject's ability to accurately recall past information.

[1108] "Cognitive function" is a general term for intellectual functions such as thinking, understanding, learning, and memory.

[1109] "Emotional state" refers to the subject's current emotional state.

[1110] "Generating an alert" means creating a notification mechanism when an anomaly is detected.

[1111] "Notify" refers to sending the generated alert to a family member or caregiver.

[1112] "Generating a report" means creating a report based on the analysis results.

[1113] The present invention is a system for evaluating cognitive function and emotional state through voice conversation with elderly people and detecting abnormalities at an early stage. Specific embodiments of the present invention will be described below.

[1114] System configuration

[1115] The system of the present invention includes an elderly terminal, a central processing unit, and related notification means, and also incorporates an emotion engine for emotion recognition.

[1116] Devices for the elderly

[1117] The elderly terminal has a function to automatically start the voice conversation system at a predetermined time. For example, the system starts up at 9:00 AM every day and starts a conversation with the elderly. The terminal asks everyday questions such as:

[1118] For example: "What did you do today?"

[1119] For example: "What is your grandchild's name?"

[1120] The user (elderly person) answers these questions verbally, and the device records the voice. The recorded voice data is passed through an emotion engine built into the device to recognize emotions in real time. For example, if the user answers, "I went to the park with my friends today," the emotion of joy is recognized from the tone of the voice.

[1121] Audio data conversion and transmission

[1122] The recorded voice data is converted into text data using natural language processing (NLP) technology, using software such as the Google Speech-to-Text API and DeepSpeech (an open-source library developed by Mozilla). Emotion data is generated along with the text data, and this data is sent to a central processing unit (server).

[1123] Central Processing Unit (Server)

[1124] The server receives the text and emotion data sent from the device. The received data is securely stored in a database (e.g., MySQL database) for each elderly person. The server then retrieves the stored data for analysis and uses natural language processing algorithms (e.g., SpaCy) to evaluate memory ability and cognitive function. The analysis includes the following points:

[1125] Conflicting answers

[1126] Memory deficiencies (e.g., forgetting grandchildren's names)

[1127] Confused conversation flow

[1128] Furthermore, the emotional data recognized by the emotion engine is also analyzed to evaluate abnormalities in the emotional state.

[1129] Alerts and Notifications

[1130] If the server detects any abnormalities in cognitive function or emotional state based on the analysis results, it will generate a warning message and notify the family or caregiver, such as, "The elderly person's recent conversations have shown inconsistencies and feelings of anxiety, so we recommend that you seek medical attention immediately."

[1131] Report generation and delivery

[1132] Periodically (e.g., weekly or monthly), the server generates detailed reports based on the elderly person's conversation and emotion data. A report generation tool (e.g., Tableau) is used to generate the reports. The generated reports are sent to family members or caregivers via email or a dedicated portal. For example, a report such as, "Conversation analysis results for the past month. Inconsistent answers, memory lapses, and feelings of anxiety were observed in multiple situations."

[1133] Examples of prompt statements

[1134] Using a generative AI model, you can input prompts like this:

[1135] "Describe a natural language processing algorithm for analyzing speech data from older adults and assessing their emotional state."

[1136] The system of the present invention combines an automated voice conversation system and an emotion engine to assess the cognitive function and emotional state of elderly people, detect abnormalities early, and send appropriate alerts and reports to family members and caregivers, enabling them to take necessary measures promptly.

[1137] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1138] Step 1: Voice conversation execution and emotion recognition (device)

[1139] The terminal activates the automated voice conversation system at a predetermined time.

[1140] Input: A specific time (e.g., 9:00 AM every day)

[1141] What it does: Your device will speak everyday questions like, "What did you do today?"

[1142] Output: The spoken question

[1143] The user verbally answers questions posed by the terminal.

[1144] Input: Terminal question

[1145] Action: The user responds, "Today I went to the park with my friends."

[1146] Output: Audio data (user's response)

[1147] The terminal records the user's response in audio format.

[1148] Input: User's voice response

[1149] How it works: The device's microphone captures audio data and saves it as a recording.

[1150] Output: Recorded audio data

[1151] The device's emotion engine analyzes the recorded voice data in real time and recognizes emotions.

[1152] Input: Recorded audio data

[1153] How it works: Emotion recognition algorithms analyze voice tone, pitch, speed, etc.

[1154] Output: Emotion data (e.g., "joy")

[1155] Step 2: Convert and send audio data (terminal)

[1156] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[1157] Input: Recorded audio data

[1158] What it does: Converts audio data into text using the Google Speech-to-Text API.

[1159] Output: Text data (e.g., "Today I went to the park with my friends.")

[1160] The terminal transmits the converted text data and emotion data to the server.

[1161] Input: Text data, emotion data

[1162] Operation: The terminal packets the data and sends it to the server over the network.

[1163] Output: Data sent to the server

[1164] Step 3: Receiving and storing data (server)

[1165] The server receives the text data and emotion data sent from the terminal.

[1166] Input: Text data, emotion data

[1167] How it works: The server captures packets on the network and analyzes the data.

[1168] Output: Received data

[1169] The server securely stores the received data in a database for each elderly person.

[1170] Input: Received data

[1171] What it does: Saves data to a MySQL database.

[1172] Output: Securely stored data

[1173] Step 4: Data analysis (server)

[1174] The server retrieves the stored text data and emotion data for analysis.

[1175] Input: saved text data, emotion data

[1176] What it does: Reads data from a database.

[1177] Output: Data to be analyzed

[1178] The server analyzes the text data using natural language processing algorithms (e.g., SpaCy) to assess memory ability and cognitive function.

[1179] Input: Data to be analyzed

[1180] How it works: It applies natural language processing algorithms to detect inconsistencies, memory lapses, and confused conversation flow.

[1181] Output: Analysis results (e.g., "Inconsistencies detected," "Memory gaps observed")

[1182] The server also analyzes the emotion data recognized by the emotion engine and evaluates abnormalities in the emotional state.

[1183] Input: Emotion data

[1184] How it works: Analyzes emotion data and detects abnormal emotional states (e.g., "persistent anxiety").

[1185] Output: Emotion analysis results

[1186] Step 5: Alert Generation and Notification (Server)

[1187] If an abnormality in cognitive function or emotional state is detected based on the analysis results, the server generates a warning message.

[1188] Input: Analysis results, emotion analysis results

[1189] Operation: Generates a warning message and determines the content of the message depending on the severity of the abnormality.

[1190] Output: Generated warning message

[1191] The server sends the generated alert message to the family member or caregiver.

[1192] Input: warning message

[1193] Action: Sends notifications via email and SMS.

[1194] Output: Warning message sent

[1195] Step 6: Generate and send report (server)

[1196] The server periodically generates detailed reports based on the elderly person's speech and emotion data.

[1197] Input: Saved conversation data, emotion data

[1198] Behavior: Use a report generation tool (e.g., Tableau) to visualize data and create reports.

[1199] Output: Generated report

[1200] The server sends the generated report to the family member or caregiver.

[1201] Input: Generated report

[1202] How it works: Reports are delivered via email and a dedicated portal.

[1203] Output: Report sent

[1204] (Application example 2)

[1205] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1206] Systems that assess the cognitive function and emotional state of elderly people need to be able to detect abnormalities in real time and quickly notify family members or caregivers, ensuring the safety of the elderly. Conventional systems have difficulty assessing the elderly's emotional state in detail and notifying them in a timely manner. Furthermore, there is a need for integration into a home security system to monitor the safety of elderly people at home.

[1207] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing received text data and evaluating memory ability and cognitive function, means for analyzing emotion data and evaluating emotional abnormalities, and means for generating and notifying an alert when an abnormality is detected based on the analysis results. This makes it possible to detect abnormalities in the cognitive function and emotional state of an elderly person in real time and quickly notify family members or security service operators.

[1208] An "elderly terminal" is an information processing terminal designed for use by elderly people, and is a device that includes voice recognition and emotion recognition functions.

[1209] The "predetermined time" refers to a specific time period that has been set in advance, during which the system will automatically operate.

[1210] "Means for initiating speech and asking specific questions" refers to the function of the elderly terminal to automatically initiate a voice dialogue and ask specific questions to the elderly.

[1211] "Means for recording spoken responses" refers to the function of the elderly person terminal to record the elderly person's responses in audio format.

[1212] "Means for converting recorded voice data into text data" refers to a function that converts recorded voice into text data using voice recognition technology.

[1213] "Means for recognizing emotions" refers to the function of analyzing and recognizing the speaker's emotional state from voice data.

[1214] "Means for transmitting text data and emotion data to a central processing unit" refers to a function for transmitting converted text and analyzed emotion information to a remote server.

[1215] The "central processing unit" refers to a server or the like that receives data sent from the elderly person's terminal and performs analytical processing.

[1216] "Means for analyzing text data to assess memory ability and cognitive function" refers to an algorithm that analyzes received text data to assess the memory ability and cognitive function of elderly people.

[1217] "Means for analyzing emotional data and evaluating emotional abnormalities" refers to a function for analyzing received emotional data and detecting abnormalities in the emotional state.

[1218] "Means for generating and notifying an alert when an abnormality is detected based on the analysis results" refers to the function of generating an alert and notifying family members or caregivers when an abnormality is detected based on the results of data analysis.

[1219] "Means for sending as a report" refers to the function of periodically sending a report summarizing the analysis results to family members or caregivers.

[1220] System configuration

[1221] The system required to implement this invention includes an "elderly terminal," a "central processing unit (server)," and a "notification means." Furthermore, it incorporates engines for speech recognition, natural language processing (NLP), and emotion recognition. Specifically, it uses the Google Cloud Speech-to-Text API for speech recognition, the Microsoft Emotion API or Amazon Rekognition for emotion recognition, and Firebase or Amazon DynamoDB for the database.

[1222] Process Overview

[1223] 1. Start a voice interaction:

[1224] The elderly device automatically starts up at a specified time and asks the elderly specific questions such as "What did you do today?" or "What is your grandchild's name?" These questions are spoken using voice synthesis technology.

[1225] 2. Audio data collection and text conversion:

[1226] When the elderly responds verbally to these questions, the device records their responses and then converts the recorded audio data into text using the Google Cloud Speech-to-Text API.

[1227] 3. Emotion recognition:

[1228] At the same time, an emotion engine (e.g., Amazon Rekognition) is used to recognize the elderly person's emotions from the voice data in real time, and the recognition results are saved as emotion data.

[1229] 4. Data transmission and storage:

[1230] The converted text data and emotion data are sent from the elderly person's device to a central processing unit (server), which receives the data and stores it securely in a database such as Firebase.

[1231] 5. Data Analysis:

[1232] The central processing unit retrieves the stored data and analyzes the text data to assess memory and cognitive function, specifically for inconsistent responses and memory lapses, and analyzes the emotional data to assess abnormal emotional states.

[1233] 6. Alert Generation and Notification:

[1234] If an abnormality is detected based on the analysis results, the system automatically generates an alert and notifies family members, caregivers, and in some cases security service operators via an SMS API (such as Twilio) or Firebase Cloud Messaging.

[1235] 7. Report generation and sending:

[1236] Detailed reports based on the elderly person's conversation and emotion data are generated periodically (e.g., weekly or monthly) and sent to family members or caregivers via email or a dedicated portal.

[1237] Specific examples

[1238] For example, if an elderly person says, "Today I went to the park with my friends," the system records the audio data and then uses the Google Cloud Speech-to-Text API to convert it into text data: "Today I went to the park with my friends." At the same time, Amazon Rekognition is used to identify the emotion as "joy" and send this data to the server. The server analyzes the text data and emotion data and evaluates whether there are any anomalies. If an anomaly is detected, for example, if the emotion "anxiety" persists, the system uses Twilio's SMS API to send an alert to the family member stating, "Anxiety has been detected in the elderly person's recent conversations, and immediate attention is required."

[1239] Prompt Sentence Examples

[1240] "What did you do today?"

[1241] "What's your grandson's name?"

[1242] "How are you feeling today?"

[1243] Through these example prompts, the system of the invention can efficiently assess the cognitive function and emotional state of elderly people and quickly provide this information to their families and caregivers.

[1244] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1245] Step 1:

[1246] The elderly terminal begins speaking at a predetermined time. Using speech synthesis technology, the elderly terminal speaks prompt sentences such as "What did you do today?" or "What is your grandchild's name?" The input is the prompt sentence, and the output is a voice response from the elderly person.

[1247] Step 2:

[1248] The elderly person's speech response is recorded on the elderly person terminal. The input is the elderly person's voice data, and the output is the recorded voice file.

[1249] Step 3:

[1250] The recorded voice data is converted into text data. The elderly device converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is an audio file, and the output is text data.

[1251] Step 4:

[1252] At the same time, the elderly device uses an emotion engine (e.g., Amazon Rekognition) to recognize emotions from voice data. The input is the voice file, and the output is emotion data.

[1253] Step 5:

[1254] The elderly person's device sends the converted text data and emotion data to the server. The input is the text data and emotion data, and the output is the data sent to the server.

[1255] Step 6:

[1256] The server analyzes the received text data and evaluates memory ability and cognitive function. Specifically, it checks for inconsistent answers and memory deficiencies. The input is text data, and the output is the evaluation results of memory ability and cognitive function.

[1257] Step 7:

[1258] The server analyzes the emotional data and evaluates the abnormality of the emotional state. The input is the emotional data, and the output is the evaluation result of the emotional state.

[1259] Step 8:

[1260] If the server detects an abnormality based on the analysis results, it generates and notifies a warning. When an abnormality is detected, it notifies family members or security service operators using Twilio's SMS API or Firebase Cloud Messaging. The input is the evaluation result, and the output is the notified warning message.

[1261] Step 9:

[1262] The server periodically generates detailed reports based on the elderly person's conversation and emotion data and sends them to family members or caregivers via email or a dedicated portal. The input is the analyzed data, and the output is the sent report.

[1263] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1264] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1265] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1266] [Third embodiment]

[1267] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1268] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1269] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1270] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1271] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1272] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1273] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1274] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1275] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1276] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1277] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1278] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1279] System Overview

[1280] The system of the present invention combines an automated voice conversation system with data analysis algorithms to assess cognitive function and detect abnormalities in elderly people. The system includes an elderly person's terminal (hereinafter referred to as "terminal"), a central processing unit (hereinafter referred to as "server"), and related notification means.

[1281] Program processing flow

[1282] Conducting a voice conversation

[1283] Terminal

[1284] The terminal automatically starts the automated voice conversation system at a preset time.

[1285] The device speaks everyday questions to the elderly, such as "What did you do today?" and "What is your grandchild's name?"

[1286] When the elderly person answers these questions verbally, the device records the answers.

[1287] Specific examples

[1288] Terminal: "What did you do today?"

[1289] User: "Today I went to the park with my friends."

[1290] Audio data conversion and transmission

[1291] Terminal

[1292] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[1293] The converted text data is sent to the server.

[1294] Specific examples

[1295] Audio data: "Today I went to the park with my friends."

[1296] Text data: "Today I went to the park with my friends."

[1297] Receiving and storing data

[1298] server

[1299] The server receives the text data sent from the terminal.

[1300] The received text data is securely stored in a database for each elderly person.

[1301] Specific examples

[1302] Server: "New conversation data saved to database."

[1303] Data analysis

[1304] server

[1305] The server retrieves the stored text data for analysis.

[1306] The server applies algorithms to assess the elderly person's cognitive function, checking their memory ability and coherence.

[1307] Check the following points:

[1308] Conflicting answers

[1309] Memory lapses (e.g., forgetting grandchildren's names)

[1310] Confused conversation flow

[1311] Specific examples

[1312] Conversation log: "Your grandchildren came over yesterday." "No, no one came." (Inconsistency resolved)

[1313] Alert generation and notification

[1314] server

[1315] Based on the analysis results, the server generates a warning message if an abnormality is detected.

[1316] The server notifies the family or caregiver of the generated warning message.

[1317] Specific examples

[1318] Alert message: "An elderly person has recently exhibited inconsistencies in their conversations and urgent medical attention is recommended."

[1319] Report generation and delivery

[1320] server

[1321] The server periodically generates a report based on the analysis of the elderly person's conversation data.

[1322] The server sends the generated reports to family members or caregivers via email or a dedicated portal.

[1323] Specific examples

[1324] Report reads: "Analysis of conversations over the past month. Multiple inconsistencies and lapses in memory have been observed."

[1325] In this way, the system of the present invention utilizes an automated voice conversation system and data analysis algorithms to assess the cognitive function of elderly people and detect abnormalities early, and by sending appropriate alerts and reports to family members and caregivers, they can take necessary measures promptly.

[1326] The processing flow will be explained below.

[1327] Step 1:

[1328] Terminal

[1329] The terminal activates the automated voice conversation system at a preset time.

[1330] The device will ask the elderly everyday questions aloud, such as "What did you do today?" and "What is your grandchild's name?"

[1331] The user (elderly person) answers these questions orally, and the terminal records the answers in audio format.

[1332] Step 2:

[1333] Terminal

[1334] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[1335] It uses advanced speech recognition algorithms to generate accurate text data from audio data.

[1336] Step 3:

[1337] Terminal

[1338] The terminal transmits the converted text data to the server.

[1339] The text data also includes the elderly person's identification information and the date and time of the conversation.

[1340] Step 4:

[1341] server

[1342] The server receives the text data sent from the terminal.

[1343] The server securely stores the received text data in a database for each elderly person.

[1344] Step 5:

[1345] server

[1346] The server retrieves the stored text data for analysis.

[1347] The server applies natural language processing algorithms to assess memory and cognitive function, specifically checking the following points:

[1348] Conflicting answers

[1349] Memory deficiencies (e.g., forgetting grandchildren's names)

[1350] Confused conversation flow

[1351] Step 6:

[1352] server

[1353] If an abnormality is detected based on the analysis results, the server generates a warning message.

[1354] The generated warning message is sent to the elderly person's family or caregiver.

[1355] Step 7:

[1356] server

[1357] The server periodically (e.g., weekly or monthly) generates detailed reports based on the elderly person's conversation data.

[1358] The server sends the generated reports to family members or caregivers via email or a dedicated portal.

[1359] Through this series of steps, the system will enable daily monitoring of the cognitive function of elderly people, enabling early detection of abnormalities and follow-up.

[1360] Example 1

[1361] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1362] There is a need to detect cognitive decline in the elderly early and provide appropriate care promptly. Conventional methods often rely on home observation and regular medical examinations, which can easily miss cognitive decline. Furthermore, it is difficult to promptly notify family members or caregivers when abnormalities are detected and provide instructions for appropriate care. This reduces the quality of life of the elderly and increases the risk of inadequate care.

[1363] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1364] In this invention, the server includes: means for the elderly person terminal to start speaking at a predetermined time and ask predetermined questions; means for recording the elderly person's spoken responses; means for converting the recorded voice data into text data; means for transmitting the text data to a central processing unit; means for analyzing the received text data and evaluating memory ability and cognitive function; means for generating and notifying warnings if abnormalities are detected based on the analysis results; means for periodically generating reports based on the analysis results of the conversation data and sending them to relevant parties; means for automatically starting the conversation system at a predetermined time using scheduling software; means for analyzing the text data using natural language processing technology and evaluating consistency and memory ability for multiple situations; means for automatically generating and notifying warning messages of various levels based on abnormalities in the analysis results; and means for periodically reporting details of detected abnormalities to relevant parties. This enables automatic and continuous evaluation of the elderly person's cognitive function, early detection of abnormalities, and prompt notification and detailed reports to relevant parties.

[1365] An "elderly terminal" is a device with voice conversation and recording functions designed for use by elderly people.

[1366] The "predetermined time" refers to a specific time that has been set in advance, and is the reference time at which the system automatically starts operating.

[1367] "Speaking" refers to the act of the elderly person terminal asking a question to the elderly person by voice.

[1368] "Spoken response" refers to a verbal response given by the elderly person to the elderly person terminal.

[1369] "Recording" is the act of recording a speech response as audio data.

[1370] "Voice data" refers to digital voice information obtained by recording the elderly person's speech responses.

[1371] "Text data" is voice data converted into text information using natural language processing technology.

[1372] A "central processing unit" is a computer system that analyzes and evaluates received data, and detects and notifies abnormalities.

[1373] "Analysis" refers to the act of evaluating the memory ability and cognitive function of elderly people based on the received text data.

[1374] "Memory ability and cognitive function assessment" refers to the act of checking the consistency of an elderly person's answers and memory deficiencies, and dynamically assessing the state of their cognitive function.

[1375] "Abnormalities" refer to problems with the elderly person's cognitive function that are different from normal, such as inconsistencies or memory loss.

[1376] A "warning" is an alert message that is generated when an abnormality is detected, and is a notification that prompts a prompt response.

[1377] A "report" is a document summarizing the results of analysis of elderly people's conversation data, and is sent to relevant parties on a regular basis.

[1378] "Scheduling software" is a program that automatically starts system operations at pre-set times.

[1379] "Natural language processing technology" is a technology that converts voice data into text information and analyzes text data.

[1380] "Analysis results" are the results of evaluations and judgments made based on text data.

[1381] "Stakeholders" are people or organizations with an interest in the condition of older people, such as their families, caregivers, and medical professionals.

[1382] MODE FOR CARRYING OUT THE INVENTION

[1383] The system of the present invention utilizes an automated voice conversation system and data analysis algorithms to assess cognitive function in elderly people and detect abnormalities at an early stage. The system includes an elderly person terminal (hereinafter referred to as "terminal"), a central processing unit (hereinafter referred to as "server"), and related notification means.

[1384] Conducting a voice conversation

[1385] Terminal

[1386] The terminal automatically starts the automated voice conversation system at a preset time, using scheduling software such as a Cron job for scheduling.

[1387] The device asks everyday questions to the elderly, such as "What did you do today?" and "What is your grandchild's name?", using Google Text-to-Speech technology.

[1388] When the elderly responds to these questions verbally, the device uses a built-in microphone to record the responses, which are then temporarily stored in local storage.

[1389] Specific examples

[1390] Terminal: "What did you do today?"

[1391] User: "Today I went to the park with my friends."

[1392] Audio data conversion and transmission

[1393] Terminal

[1394] The device converts the recorded voice data into text data using natural language processing (NLP) technology, which uses the Google Cloud Speech-to-Text API.

[1395] After converting the data into text, the device sends the text to a remote server using an HTTP POST request, with end-to-end encryption applied to the communication.

[1396] Specific examples

[1397] Audio data: "Today I went to the park with my friends."

[1398] Text data: "Today I went to the park with my friends."

[1399] Receiving and storing data

[1400] server

[1401] The server receives the HTTP POST request sent from the device and extracts the text data from it, using the Apache or Nginx web server software.

[1402] The extracted text data is securely stored in a database for each elderly person. MySQL or PostgreSQL is used as the database management system, and data encryption is performed.

[1403] Specific examples

[1404] Server: "New conversation data saved to database."

[1405] Data analysis

[1406] server

[1407] The server periodically retrieves the stored text data for analysis, using a Cron job.

[1408] The stored text data is then used to run an algorithm that assesses the cognitive function of older adults. This algorithm uses natural language processing (NLP) techniques and libraries such as Python's NLTK and spaCy.

[1409] During processing, check for inconsistent answers, memory gaps, and confused conversation flow.

[1410] Specific examples

[1411] Conversation log: "Your grandchildren came over yesterday." "No, no one came." (Inconsistency resolved)

[1412] Alert generation and notification

[1413] server

[1414] If an abnormality is detected based on the analysis results, the server generates a warning message. This message is automatically generated using a template.

[1415] The server then sends the generated warning message to family members or caregivers via email, SMS, or a dedicated app. Email is sent using the SMTP protocol, and SMS is sent using services such as Twilio.

[1416] Specific examples

[1417] Alert message: "An elderly person has recently exhibited inconsistencies in their conversations and urgent medical attention is recommended."

[1418] Report generation and delivery

[1419] server

[1420] The server periodically generates reports based on the analysis of the elderly's conversation data. The reports are generated in PDF format using libraries such as Python's ReportLab.

[1421] The server generates reports and sends them to family members or caregivers via email or a dedicated portal, which can be run periodically using a scheduler.

[1422] Specific examples

[1423] Report reads: "Analysis of conversations over the past month. Multiple inconsistencies and lapses in memory have been observed."

[1424] Example prompts to input to the generative AI model

[1425] Examples:

[1426] Terminal: "What did you do today?"

[1427] User: "Today I went to the park with my friends."

[1428] Example prompt sentence:

[1429] "Please suggest ways to predict how the elderly person will respond to the device and ensure consistency in the next conversation. Also, please suggest improvements to the algorithm to identify gaps or inconsistencies in memory."

[1430] The system of this invention can effectively evaluate the cognitive function of elderly people and detect abnormalities early. Detailed information can be obtained through conversational interaction, and appropriate nursing and medical measures can be taken in real time based on that information. In addition, appropriate alerts and regular reports enable prompt and accurate responses.

[1431] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1432] Step 1:

[1433] Starting a voice conversation

[1434] The terminal automatically starts the automated voice conversation system at a preset time, using scheduling software such as a Cron job for scheduling.

[1435] Input: Scheduling time

[1436] Operation: The device monitors its internal clock and starts the voice conversation system at the set time.

[1437] Output: Voice conversation system activated

[1438] Step 2:

[1439] Asking a question

[1440] The device uses Google Text-to-Speech technology to ask everyday questions to the elderly, such as "What did you do today?" or "What is your grandchild's name?"

[1441] Input: Preset question content

[1442] How it works: The device passes preset question text to the Google Text-to-Speech API, generates audio data, and speaks it to the elderly person.

[1443] Output: Question spoken to the elderly person

[1444] Step 3:

[1445] Spoken response recording

[1446] When the elderly responds verbally to the questions, the device uses a built-in microphone to record the response, which is then temporarily stored in local storage.

[1447] Input: Verbal response from the elderly person

[1448] How it works: The device captures audio data through the microphone and stores it in local storage.

[1449] Output: Recorded audio data file

[1450] Step 4:

[1451] Converting audio data to text

[1452] The device converts the recorded voice data into text data using natural language processing (NLP) technology, which uses the Google Cloud Speech-to-Text API.

[1453] Input: Recorded audio data

[1454] How it works: The device sends audio data to the Google Cloud Speech-to-Text API and retrieves text data.

[1455] Output: Converted text data

[1456] Step 5:

[1457] Sending text data

[1458] The device then sends the converted text data to a remote server using an HTTP POST request, with end-to-end encryption applied to the communication.

[1459] Input: Converted text data

[1460] How it works: The device generates an HTTP POST request and sends data to the specified endpoint on the server.

[1461] Output: Text data sent to the server

[1462] Step 6:

[1463] Receiving and storing text data

[1464] The server receives the HTTP POST request sent from the device and extracts the text data from it, using the Apache or Nginx web server software.

[1465] Input: HTTP POST request

[1466] What happens: The server parses the POST request and extracts the text data.

[1467] Output: Extracted text data

[1468] Step 7:

[1469] Data storage

[1470] The server securely stores the text data for each elderly person in a database, using MySQL or PostgreSQL as the database management system, and the data is encrypted.

[1471] Input: Extracted text data

[1472] Operation: The server saves the text data in a database. When saved, it associates it with the elderly person's identification information.

[1473] Output: Securely stored text data

[1474] Step 8:

[1475] Regular analysis of data

[1476] The server periodically retrieves the stored text data for analysis, using a Cron job.

[1477] Input: Text data stored in a database

[1478] Operation: The server periodically retrieves the stored text data and runs an analysis program.

[1479] Output: Text data extracted for analysis

[1480] Step 9:

[1481] Cognitive function assessment

[1482] The server runs an algorithm on the text data to assess cognitive function in older adults, using natural language processing (NLP) techniques and libraries such as Python's NLTK and spaCy.

[1483] Input: Text data extracted for analysis

[1484] How it works: The server runs a program that detects inconsistent answers, memory lapses, and disorganized conversation flow.

[1485] Output: Cognitive function assessment results

[1486] Step 10:

[1487] Anomaly detection and alert generation

[1488] If an abnormality is detected based on the analysis results, the server generates a warning message. This message is automatically generated using a template.

[1489] Input: Cognitive function assessment results

[1490] Operation: The server analyzes the evaluation results and generates a corresponding warning message if an anomaly is detected.

[1491] Output: Warning message

[1492] Step 11:

[1493] Warning Notification

[1494] The server then sends the generated warning message to family members or caregivers via email, SMS, or a dedicated app. Email is sent using the SMTP protocol, and SMS is sent using services such as Twilio.

[1495] Input: warning message

[1496] What it does: The server generates email and / or SMS notifications and sends them to interested parties.

[1497] Output: Warning message sent to family members or caregivers

[1498] Step 12:

[1499] Report generation and delivery

[1500] The server periodically generates reports based on the analysis of the elderly's conversation data. The reports are generated in PDF format using libraries such as Python's ReportLab.

[1501] Input: Analysis results of elderly people's conversation data

[1502] How it works: The server periodically aggregates the analysis results of the conversation data and generates a report in PDF format.

[1503] Output: Generated report

[1504] Step 13:

[1505] Report Notification

[1506] The server then sends the generated reports to family members or caregivers via email or a dedicated portal, and this process is also performed periodically using a scheduler.

[1507] Input: Generated report

[1508] How it works: The server sends reports via email or a dedicated portal.

[1509] Output: Report sent to family or caregiver

[1510] (Application example 1)

[1511] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1512] In modern society, efficient and accurate assessment of the cognitive function of elderly people and early detection of cognitive decline occurring in daily life are important challenges. However, conventional interactive systems lack a well-established method for thoroughly assessing the cognitive function of elderly people through their everyday conversations and appropriately notifying the results. Furthermore, they lack a means for quickly issuing warnings when abnormalities are detected and providing necessary information to family members and caregivers. Furthermore, they lack the ability to regularly store and analyze data to ensure the safety of elderly people. Therefore, the present invention aims to solve these challenges.

[1513] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1514] In this invention, the server includes a means for notifying family members and caregivers of speech-recognized text data, a means for periodically saving and analyzing daily conversation data, a means for analyzing the elderly person's conversation data using natural language processing technology, a means for sending warning messages regarding detected abnormalities to family members and caregivers, and a means for sending generated periodic reports to family members and caregivers. This enables detailed assessment of the elderly person's cognitive function through their daily conversations, early detection of abnormalities, and appropriate alert notifications. Furthermore, periodic data saving and analysis can provide comprehensive support for ensuring the safety of the elderly.

[1515] An "elderly terminal" is an electronic device for use by elderly people that is capable of voice input and output.

[1516] A "central processing unit" is a centralized device for analyzing and evaluating speech-recognized data, and functions as a server.

[1517] The "predetermined time" refers to a specific time or period that has been set in advance, at which the system automatically starts operating.

[1518] "Start speaking" means that the system will output pre-programmed messages and ask questions to the senior.

[1519] The "prescribed questions" are questions about the elderly person's daily life and memory, and are useful for assessing cognitive function.

[1520] "Spoken responses" refer to verbal responses given by the elderly to questions posed by the system.

[1521] "Means for recording" refers to a function for recording the elderly person's spoken responses as digital audio data.

[1522] "Means for converting into text data" refers to technology for converting recorded voice data into text information, and primarily utilizes natural language processing technology.

[1523] "Means of analysis" refers to algorithms for assessing the cognitive function of elderly people based on text data.

[1524] "Means for detecting abnormalities" refers to the function of detecting abnormalities such as a decline in cognitive function from the analysis results and instructing on necessary measures.

[1525] "Means for generating and notifying alerts" refers to a system that has the ability to generate alert messages regarding detected abnormalities and promptly notify family members or caregivers.

[1526] "Natural language processing technology" is a technology for processing voice and text data to understand and generate human language.

[1527] A "regular report" is a report summarizing the results of periodic analyses, and is sent to family members or caregivers.

[1528] "Daily conversation data" is a record of regular voice interactions between the elderly person and the system.

[1529] A "generative AI model" is a collection of algorithms trained to perform specific tasks using artificial intelligence.

[1530] A "prompt" is a question or instruction input to a generative AI model, and is a phrase used to encourage a response from the elderly person.

[1531] System Overview

[1532] The system of the present invention mainly consists of an elderly person's terminal, a central processing unit (server), and a notification means. The elderly person's terminal is equipped with an automated voice conversation system that evaluates the elderly person's cognitive function through everyday conversation. The central processing unit analyzes the received data and generates a warning message to notify family members or caregivers if an abnormality is detected. It also has a function to periodically provide analysis results as a report.

[1533] Hardware and software used

[1534] 1. Devices for the elderly:

[1535] Hardware: Smartphone or tablet (including microphone)

[1536] Software: Natural language processing technology (e.g., Google Speech-to-Text API)

[1537] 2. Central Processing Unit (Server):

[1538] Hardware: High-performance network server

[1539] Software: Database systems, data analysis algorithms, notification systems

[1540] Details of data processing and calculation

[1541] Device operation

[1542] The device will automatically start a voice conversation at a specific time. For example, it will ask questions such as "What did you do today?" or "What is your grandchild's name?" When the elderly person answers these questions, the device will record the voice data.

[1543] Audio data conversion and transmission

[1544] The recorded voice data is converted into text data using natural language processing technology within the device. This converted text data is sent to a server. For example, if an elderly person replies, "Today, I went to the park with my friends," the voice is converted into text data that reads, "Today, I went to the park with my friends."

[1545] Data reception and analysis

[1546] The server receives the text data sent from the device. The received data is stored in a database and analysis begins. The server applies algorithms to assess the elderly person's cognitive function and checks their memory ability and coherence. Analysis detects inconsistent answers, memory gaps, and disorganized conversation flow.

[1547] Generate alerts and reports

[1548] If an abnormality is detected based on the analysis results, the server generates a warning message and notifies the family or caregiver. For example, a message may be sent saying, "There have been inconsistencies in the elderly person's recent conversations. We recommend that you seek medical advice as soon as possible." In addition, reports are generated based on the periodic analysis results and sent to the family or caregiver. The report contents may include, "These are the results of the conversation analysis for the past month. Inconsistent answers and lapses in memory have been observed in multiple situations."

[1549] Examples of prompt statements

[1550] Specific examples of prompts include:

[1551] "What did you do today?"

[1552] "What's your grandson's name?"

[1553] "What did you have for dinner last night?"

[1554] This concludes the detailed description of the embodiment of the present invention. This system enables the evaluation of cognitive function through everyday conversations with elderly people and the early detection of abnormalities, allowing family members or caregivers to take prompt action.

[1555] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1556] Processing Steps

[1557] Step 1:

[1558] Starting up elderly device

[1559] Specific operation: The device automatically starts the automated voice conversation system at the specified time, performs initialization processing, and enables the microphone.

[1560] Input: Predetermined time based on system settings

[1561] Output: Automated voice conversation system startup status

[1562] Step 2:

[1563] Prompt sentence utterance

[1564] Specific operation: The device will speak a pre-defined prompt, such as "What did you do today?", to the elderly person.

[1565] Input: Questionnaires saved in the system

[1566] Output: Voice questions for the elderly

[1567] Step 3:

[1568] Spoken response recording

[1569] Specific operation: The elderly person answers questions, and their answers are recorded by the device's microphone.

[1570] Input: Elderly voice response

[1571] Output: Recorded audio data

[1572] Step 4:

[1573] Converting audio data to text

[1574] Specific operation: The recorded audio data is converted into text data using natural language processing technology (such as Google Speech-to-Text API).

[1575] Input: Recorded audio data

[1576] Output: Text data

[1577] Step 5:

[1578] Sending text data

[1579] Specific operation: The converted text data is sent to the server using an HTTP request.

[1580] Input: Converted text data

[1581] Output: Text data sent to the server

[1582] Step 6:

[1583] Receiving and storing data

[1584] Specific operation: The server receives the text data sent from the device and stores it in a database. When stored, the data is classified for each elderly person.

[1585] Input: Text data sent from the terminal

[1586] Output: Text data stored in a database

[1587] Step 7:

[1588] Data analysis

[1589] How it works: The server processes the stored text data using analytical algorithms to assess the elderly person's cognitive function, checking memory ability, consistency of speech, and inconsistencies.

[1590] Input: Text data stored in a database

[1591] Output: Cognitive function assessment results

[1592] Step 8:

[1593] Anomaly detection and alert generation

[1594] Specific operation: If the server detects an anomaly in the analysis results, it generates a warning message. Different levels of warning are set depending on the type and severity of the anomaly.

[1595] Input: Cognitive function assessment results

[1596] Output: Warning message

[1597] Step 9:

[1598] Sending notifications

[1599] Specific operation: The generated warning message is sent to family members or caregivers via email or a dedicated app.

[1600] Input: warning message

[1601] Output: Notification sent to family members and caregivers

[1602] Step 10:

[1603] Generate and send scheduled reports

[1604] Specific operation: The server periodically generates a report based on the analysis results and sends it to the family or caregiver. The report includes the analysis results of the elderly person's conversation data.

[1605] Input: Accumulated data of cognitive function evaluation results

[1606] Output: Regular reports sent to family members and caregivers

[1607] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1608] System Overview

[1609] The system of the present invention assesses cognitive function through voice conversation with elderly people and simultaneously recognizes their emotional state, enabling early detection of abnormalities. The system includes an elderly person terminal (hereinafter referred to as "terminal"), a central processing unit (hereinafter referred to as "server"), and related notification means. It also incorporates an emotion engine that performs emotion recognition.

[1610] Program processing flow

[1611] Voice conversation execution and emotion recognition

[1612] Terminal

[1613] The terminal activates the automated voice conversation system at a preset time.

[1614] The device will ask the elderly everyday questions such as "What did you do today?" or "What is your grandchild's name?"

[1615] When the elderly person answers these questions verbally, the device records the answers in audio format.

[1616] The emotion engine built into the device recognizes emotions in real time from the voice data of the elderly person's speech responses.

[1617] Specific examples

[1618] Terminal: "What did you do today?"

[1619] User: "I went to the park with my friends today." (Emotion of joy is recognized from the tone of voice)

[1620] Audio data conversion and transmission

[1621] Terminal

[1622] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[1623] It uses advanced speech recognition algorithms to generate accurate text data from audio data.

[1624] The converted text data and emotion data are sent to the server.

[1625] Specific examples

[1626] Audio data: "Today I went to the park with my friends."

[1627] Text data: "Today I went to the park with my friends."

[1628] Emotion data: "Joy"

[1629] Receiving and storing data

[1630] server

[1631] The server receives the text data and emotion data sent from the terminal.

[1632] The server securely stores the received data in a database for each elderly person.

[1633] Specific examples

[1634] Server: "New speech and emotion data saved to database."

[1635] Data analysis

[1636] server

[1637] The server retrieves the stored text data and emotion data for analysis.

[1638] The server applies natural language processing algorithms to assess memory and cognitive function, specifically checking the following:

[1639] Conflicting answers

[1640] Memory deficiencies (e.g., forgetting grandchildren's names)

[1641] Confused conversation flow

[1642] The server also analyzes the emotion data recognized by the emotion engine and evaluates the abnormality of the emotional state.

[1643] Specific examples

[1644] Conversation log: "Your grandchildren came over yesterday." "No, no one came." (Inconsistency resolved)

[1645] Emotional data: "Continued anxiety"

[1646] Alert generation and notification

[1647] server

[1648] If an abnormality in cognitive function or emotional state is detected based on the analysis results, the server generates a warning message.

[1649] The generated warning message is sent to the elderly person's family or caregiver.

[1650] Specific examples

[1651] Alert message: "An older adult has recently shown signs of inconsistency and anxiety in their conversations, so immediate medical attention is recommended."

[1652] Report generation and delivery

[1653] server

[1654] The server periodically (e.g., weekly or monthly) generates detailed reports based on the elderly person's conversational and emotional data.

[1655] The server sends the generated reports to family members or caregivers via email or a dedicated portal.

[1656] Specific examples

[1657] Report reads: "Analysis of conversations over the past month has revealed inconsistent responses, memory lapses, and feelings of anxiety on multiple occasions."

[1658] In this way, the system of the present invention combines an automated voice conversation system and an emotion engine to assess the cognitive function and emotional state of elderly people, detect abnormalities early, and send appropriate alerts and reports to family members and caregivers so that necessary measures can be taken promptly.

[1659] The processing flow will be explained below.

[1660] Step 1:

[1661] Terminal

[1662] The terminal activates the automated voice conversation system at a preset time.

[1663] The terminal speaks predetermined questions to the elderly person, such as "What did you do today?" or "What is your grandchild's name?"

[1664] The user (elderly person) answers these questions verbally.

[1665] Execution example:

[1666] Terminal: "What did you do today?"

[1667] User: "Today I went to the park with my friends."

[1668] Step 2:

[1669] Terminal

[1670] The terminal records the user's spoken response.

[1671] The recorded voice data is converted into text using the device's built-in natural language processing (NLP) engine.

[1672] Prepare the converted text data to be sent to the server.

[1673] Execution example:

[1674] Audio data: "Today I went to the park with my friends."

[1675] Text data: "Today I went to the park with my friends."

[1676] Step 3:

[1677] Terminal

[1678] The emotion engine built into the device recognizes emotions from recorded voice data in real time.

[1679] The recognized emotion data is added to the text data and transmitted to the server.

[1680] Execution example:

[1681] Emotion data sent along with the voice data: "Joy"

[1682] Step 4:

[1683] server

[1684] The server receives the text data and emotion data sent from the terminal.

[1685] The received data is stored in a database for each elderly person.

[1686] Execution example:

[1687] Server: "New speech and emotion data saved to database."

[1688] Step 5:

[1689] server

[1690] The server retrieves the stored text data and emotion data for analysis.

[1691] The server applies natural language processing algorithms to assess memory ability and cognitive function from text data.

[1692] Emotional data will also be analyzed to assess the transition of the elderly person's emotional state.

[1693] Things to check:

[1694] Check for contradictory answers.

[1695] Check for memory lapses (e.g., forgetting the names of grandchildren).

[1696] Detecting confused conversation flow.

[1697] Check to see if certain emotions (e.g., anxiety, anger) are occurring frequently.

[1698] Step 6:

[1699] server

[1700] Based on the analysis results, the server generates a warning message if abnormalities are detected in memory ability, cognitive function, or emotional state.

[1701] Notify family members or caregivers of the generated warning message.

[1702] Execution example:

[1703] Alert message: "An older adult has recently shown signs of inconsistency and anxiety in their conversations, so immediate medical attention is recommended."

[1704] Step 7:

[1705] server

[1706] The server periodically (for example, weekly or monthly) generates detailed reports based on the elderly person's conversational and emotional data.

[1707] Generated reports are sent to family members or caregivers via email or a dedicated portal.

[1708] Execution example:

[1709] Report reads: "Analysis of conversations over the past month has revealed inconsistent responses, memory lapses, and feelings of anxiety on multiple occasions."

[1710] Through these steps, the system of the present invention can routinely monitor the cognitive function and emotional state of elderly people and detect abnormalities early. Appropriate alerts and reports can be sent to family members and caregivers, enabling them to take prompt action to ensure the quality of life and safety of the elderly.

[1711] Example 2

[1712] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1713] It is difficult to routinely assess the cognitive function and emotional state of elderly people and detect abnormalities early. In particular, there is a need for a system that can accurately detect cognitive decline and emotional changes through everyday conversations and promptly notify family members and caregivers. There is also a need for detailed data analysis and reports that enable appropriate responses tailored to the individual condition of each elderly person.

[1714] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for sending periodically generated reports to family members or caregivers, a means for generating and notifying different levels of warnings, a means for analyzing received text data and emotional data to evaluate memory ability and cognitive function, and a means for analyzing emotional data to evaluate emotional state. This makes it possible to quickly and accurately detect changes in the elderly person's cognitive function and emotions, to promptly notify if an abnormality is found, and to provide a detailed report.

[1715] An "elderly terminal" is a device that can carry out voice conversations with the elderly, record audio, convert text, recognize emotions, and transmit data.

[1716] "Predetermined time" refers to a specific date, time or time period for which the system is set.

[1717] "Start speaking" means that the terminal automatically starts a voice conversation.

[1718] The "predetermined question" refers to a question that is set in advance in the system.

[1719] "Recording a speech response" means that the terminal records the voice of the elderly person's response.

[1720] "Natural language processing technology" is a general term for computer technology for converting voice data into text data.

[1721] "Text data" is recorded voice data converted into text information.

[1722] "Emotion data" refers to information about emotions recognized from voice data.

[1723] "Central Processing Unit" is the main computer that processes and analyzes the received data and notifies the results.

[1724] "Analyzing" means analyzing data in detail and extracting meaning and patterns.

[1725] "Memory ability" refers to a subject's ability to accurately recall past information.

[1726] "Cognitive function" is a general term for intellectual functions such as thinking, understanding, learning, and memory.

[1727] "Emotional state" refers to the subject's current emotional state.

[1728] "Generating an alert" means creating a notification mechanism when an anomaly is detected.

[1729] "Notify" refers to sending the generated alert to a family member or caregiver.

[1730] "Generating a report" means creating a report based on the analysis results.

[1731] The present invention is a system for evaluating cognitive function and emotional state through voice conversation with elderly people and detecting abnormalities at an early stage. Specific embodiments of the present invention will be described below.

[1732] System configuration

[1733] The system of the present invention includes an elderly terminal, a central processing unit, and related notification means, and also incorporates an emotion engine for emotion recognition.

[1734] Devices for the elderly

[1735] The elderly terminal has a function to automatically start the voice conversation system at a predetermined time. For example, the system starts up at 9:00 AM every day and starts a conversation with the elderly. The terminal asks everyday questions such as:

[1736] For example: "What did you do today?"

[1737] For example: "What is your grandchild's name?"

[1738] The user (elderly person) answers these questions verbally, and the device records the voice. The recorded voice data is passed through an emotion engine built into the device to recognize emotions in real time. For example, if the user answers, "I went to the park with my friends today," the emotion of joy is recognized from the tone of the voice.

[1739] Audio data conversion and transmission

[1740] The recorded voice data is converted into text data using natural language processing (NLP) technology, using software such as the Google Speech-to-Text API and DeepSpeech (an open-source library developed by Mozilla). Emotion data is generated along with the text data, and this data is sent to a central processing unit (server).

[1741] Central Processing Unit (Server)

[1742] The server receives the text and emotion data sent from the device. The received data is securely stored in a database (e.g., MySQL database) for each elderly person. The server then retrieves the stored data for analysis and uses natural language processing algorithms (e.g., SpaCy) to evaluate memory ability and cognitive function. The analysis includes the following points:

[1743] Conflicting answers

[1744] Memory deficiencies (e.g., forgetting grandchildren's names)

[1745] Confused conversation flow

[1746] Furthermore, the emotional data recognized by the emotion engine is also analyzed to evaluate abnormalities in the emotional state.

[1747] Alerts and Notifications

[1748] If the server detects any abnormalities in cognitive function or emotional state based on the analysis results, it will generate a warning message and notify the family or caregiver, such as, "The elderly person's recent conversations have shown inconsistencies and feelings of anxiety, so we recommend that you seek medical attention immediately."

[1749] Report generation and delivery

[1750] Periodically (e.g., weekly or monthly), the server generates detailed reports based on the elderly person's conversation and emotion data. A report generation tool (e.g., Tableau) is used to generate the reports. The generated reports are sent to family members or caregivers via email or a dedicated portal. For example, a report such as, "Conversation analysis results for the past month. Inconsistent answers, memory lapses, and feelings of anxiety were observed in multiple situations."

[1751] Examples of prompt statements

[1752] Using a generative AI model, you can input prompts like this:

[1753] "Describe a natural language processing algorithm for analyzing speech data from older adults and assessing their emotional state."

[1754] The system of the present invention combines an automated voice conversation system and an emotion engine to assess the cognitive function and emotional state of elderly people, detect abnormalities early, and send appropriate alerts and reports to family members and caregivers, enabling them to take necessary measures promptly.

[1755] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1756] Step 1: Voice conversation execution and emotion recognition (device)

[1757] The terminal activates the automated voice conversation system at a predetermined time.

[1758] Input: A specific time (e.g., 9:00 AM every day)

[1759] What it does: Your device will speak everyday questions like, "What did you do today?"

[1760] Output: The spoken question

[1761] The user verbally answers questions posed by the terminal.

[1762] Input: Terminal question

[1763] Action: The user responds, "Today I went to the park with my friends."

[1764] Output: Audio data (user's response)

[1765] The terminal records the user's response in audio format.

[1766] Input: User's voice response

[1767] How it works: The device's microphone captures audio data and saves it as a recording.

[1768] Output: Recorded audio data

[1769] The device's emotion engine analyzes the recorded voice data in real time and recognizes emotions.

[1770] Input: Recorded audio data

[1771] How it works: Emotion recognition algorithms analyze voice tone, pitch, speed, etc.

[1772] Output: Emotion data (e.g., "joy")

[1773] Step 2: Convert and send audio data (terminal)

[1774] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[1775] Input: Recorded audio data

[1776] What it does: Converts audio data into text using the Google Speech-to-Text API.

[1777] Output: Text data (e.g., "Today I went to the park with my friends.")

[1778] The terminal transmits the converted text data and emotion data to the server.

[1779] Input: Text data, emotion data

[1780] Operation: The terminal packets the data and sends it to the server over the network.

[1781] Output: Data sent to the server

[1782] Step 3: Receiving and storing data (server)

[1783] The server receives the text data and emotion data sent from the terminal.

[1784] Input: Text data, emotion data

[1785] How it works: The server captures packets on the network and analyzes the data.

[1786] Output: Received data

[1787] The server securely stores the received data in a database for each elderly person.

[1788] Input: Received data

[1789] What it does: Saves data to a MySQL database.

[1790] Output: Securely stored data

[1791] Step 4: Data analysis (server)

[1792] The server retrieves the stored text data and emotion data for analysis.

[1793] Input: saved text data, emotion data

[1794] What it does: Reads data from a database.

[1795] Output: Data to be analyzed

[1796] The server analyzes the text data using natural language processing algorithms (e.g., SpaCy) to assess memory ability and cognitive function.

[1797] Input: Data to be analyzed

[1798] How it works: It applies natural language processing algorithms to detect inconsistencies, memory lapses, and confused conversation flow.

[1799] Output: Analysis results (e.g., "Inconsistencies detected," "Memory gaps observed")

[1800] The server also analyzes the emotion data recognized by the emotion engine and evaluates abnormalities in the emotional state.

[1801] Input: Emotion data

[1802] How it works: Analyzes emotion data and detects abnormal emotional states (e.g., "persistent anxiety").

[1803] Output: Emotion analysis results

[1804] Step 5: Alert Generation and Notification (Server)

[1805] If an abnormality in cognitive function or emotional state is detected based on the analysis results, the server generates a warning message.

[1806] Input: Analysis results, emotion analysis results

[1807] Operation: Generates a warning message and determines the content of the message depending on the severity of the abnormality.

[1808] Output: Generated warning message

[1809] The server sends the generated alert message to the family member or caregiver.

[1810] Input: warning message

[1811] Action: Sends notifications via email and SMS.

[1812] Output: Warning message sent

[1813] Step 6: Generate and send report (server)

[1814] The server periodically generates detailed reports based on the elderly person's speech and emotion data.

[1815] Input: Saved conversation data, emotion data

[1816] Behavior: Use a report generation tool (e.g., Tableau) to visualize data and create reports.

[1817] Output: Generated report

[1818] The server sends the generated report to the family member or caregiver.

[1819] Input: Generated report

[1820] How it works: Reports are delivered via email and a dedicated portal.

[1821] Output: Report sent

[1822] (Application example 2)

[1823] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1824] Systems that assess the cognitive function and emotional state of elderly people need to be able to detect abnormalities in real time and quickly notify family members or caregivers, ensuring the safety of the elderly. Conventional systems have difficulty assessing the elderly's emotional state in detail and notifying them in a timely manner. Furthermore, there is a need for integration into a home security system to monitor the safety of elderly people at home.

[1825] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing received text data and evaluating memory ability and cognitive function, means for analyzing emotion data and evaluating emotional abnormalities, and means for generating and notifying an alert when an abnormality is detected based on the analysis results. This makes it possible to detect abnormalities in the cognitive function and emotional state of an elderly person in real time and quickly notify family members or security service operators.

[1826] An "elderly terminal" is an information processing terminal designed for use by elderly people, and is a device that includes voice recognition and emotion recognition functions.

[1827] The "predetermined time" refers to a specific time period that has been set in advance, during which the system will automatically operate.

[1828] "Means for initiating speech and asking specific questions" refers to the function of the elderly terminal to automatically initiate a voice dialogue and ask specific questions to the elderly.

[1829] "Means for recording spoken responses" refers to the function of the elderly person terminal to record the elderly person's responses in audio format.

[1830] "Means for converting recorded voice data into text data" refers to a function that converts recorded voice into text data using voice recognition technology.

[1831] "Means for recognizing emotions" refers to the function of analyzing and recognizing the speaker's emotional state from voice data.

[1832] "Means for transmitting text data and emotion data to a central processing unit" refers to a function for transmitting converted text and analyzed emotion information to a remote server.

[1833] The "central processing unit" refers to a server or the like that receives data sent from the elderly person's terminal and performs analytical processing.

[1834] "Means for analyzing text data to assess memory ability and cognitive function" refers to an algorithm that analyzes received text data to assess the memory ability and cognitive function of elderly people.

[1835] "Means for analyzing emotional data and evaluating emotional abnormalities" refers to a function for analyzing received emotional data and detecting abnormalities in the emotional state.

[1836] "Means for generating and notifying an alert when an abnormality is detected based on the analysis results" refers to the function of generating an alert and notifying family members or caregivers when an abnormality is detected based on the results of data analysis.

[1837] "Means for sending as a report" refers to the function of periodically sending a report summarizing the analysis results to family members or caregivers.

[1838] System configuration

[1839] The system required to implement this invention includes an "elderly terminal," a "central processing unit (server)," and a "notification means." Furthermore, it incorporates engines for speech recognition, natural language processing (NLP), and emotion recognition. Specifically, it uses the Google Cloud Speech-to-Text API for speech recognition, the Microsoft Emotion API or Amazon Rekognition for emotion recognition, and Firebase or Amazon DynamoDB for the database.

[1840] Process Overview

[1841] 1. Start a voice interaction:

[1842] The elderly device automatically starts up at a specified time and asks the elderly specific questions such as "What did you do today?" or "What is your grandchild's name?" These questions are spoken using voice synthesis technology.

[1843] 2. Audio data collection and text conversion:

[1844] When the elderly responds verbally to these questions, the device records their responses and then converts the recorded audio data into text using the Google Cloud Speech-to-Text API.

[1845] 3. Emotion recognition:

[1846] At the same time, an emotion engine (e.g., Amazon Rekognition) is used to recognize the elderly person's emotions from the voice data in real time, and the recognition results are saved as emotion data.

[1847] 4. Data transmission and storage:

[1848] The converted text data and emotion data are sent from the elderly person's device to a central processing unit (server), which receives the data and stores it securely in a database such as Firebase.

[1849] 5. Data Analysis:

[1850] The central processing unit retrieves the stored data and analyzes the text data to assess memory and cognitive function, specifically for inconsistent responses and memory lapses, and analyzes the emotional data to assess abnormal emotional states.

[1851] 6. Alert Generation and Notification:

[1852] If an abnormality is detected based on the analysis results, the system automatically generates an alert and notifies family members, caregivers, and in some cases security service operators via an SMS API (such as Twilio) or Firebase Cloud Messaging.

[1853] 7. Report generation and sending:

[1854] Detailed reports based on the elderly person's conversation and emotion data are generated periodically (e.g., weekly or monthly) and sent to family members or caregivers via email or a dedicated portal.

[1855] Specific examples

[1856] For example, if an elderly person says, "Today I went to the park with my friends," the system records the audio data and then uses the Google Cloud Speech-to-Text API to convert it into text data: "Today I went to the park with my friends." At the same time, Amazon Rekognition is used to identify the emotion as "joy" and send this data to the server. The server analyzes the text data and emotion data and evaluates whether there are any anomalies. If an anomaly is detected, for example, if the emotion "anxiety" persists, the system uses Twilio's SMS API to send an alert to the family member stating, "Anxiety has been detected in the elderly person's recent conversations, and immediate attention is required."

[1857] Prompt Sentence Examples

[1858] "What did you do today?"

[1859] "What's your grandson's name?"

[1860] "How are you feeling today?"

[1861] Through these example prompts, the system of the invention can efficiently assess the cognitive function and emotional state of elderly people and quickly provide this information to their families and caregivers.

[1862] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1863] Step 1:

[1864] The elderly terminal begins speaking at a predetermined time. Using speech synthesis technology, the elderly terminal speaks prompt sentences such as "What did you do today?" or "What is your grandchild's name?" The input is the prompt sentence, and the output is a voice response from the elderly person.

[1865] Step 2:

[1866] The elderly person's speech response is recorded on the elderly person terminal. The input is the elderly person's voice data, and the output is the recorded voice file.

[1867] Step 3:

[1868] The recorded voice data is converted into text data. The elderly device converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is an audio file, and the output is text data.

[1869] Step 4:

[1870] At the same time, the elderly device uses an emotion engine (e.g., Amazon Rekognition) to recognize emotions from voice data. The input is the voice file, and the output is emotion data.

[1871] Step 5:

[1872] The elderly person's device sends the converted text data and emotion data to the server. The input is the text data and emotion data, and the output is the data sent to the server.

[1873] Step 6:

[1874] The server analyzes the received text data and evaluates memory ability and cognitive function. Specifically, it checks for inconsistent answers and memory deficiencies. The input is text data, and the output is the evaluation results of memory ability and cognitive function.

[1875] Step 7:

[1876] The server analyzes the emotional data and evaluates the abnormality of the emotional state. The input is the emotional data, and the output is the evaluation result of the emotional state.

[1877] Step 8:

[1878] If the server detects an abnormality based on the analysis results, it generates and notifies a warning. When an abnormality is detected, it notifies family members or security service operators using Twilio's SMS API or Firebase Cloud Messaging. The input is the evaluation result, and the output is the notified warning message.

[1879] Step 9:

[1880] The server periodically generates detailed reports based on the elderly person's conversation and emotion data and sends them to family members or caregivers via email or a dedicated portal. The input is the analyzed data, and the output is the sent report.

[1881] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1882] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1883] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1884] [Fourth embodiment]

[1885] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1886] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1887] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1888] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1889] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1890] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1891] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1892] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1893] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1894] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1895] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1896] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1897] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1898] System Overview

[1899] The system of the present invention combines an automated voice conversation system with data analysis algorithms to assess cognitive function and detect abnormalities in elderly people. The system includes an elderly person's terminal (hereinafter referred to as "terminal"), a central processing unit (hereinafter referred to as "server"), and related notification means.

[1900] Program processing flow

[1901] Conducting a voice conversation

[1902] Terminal

[1903] The terminal automatically starts the automated voice conversation system at a preset time.

[1904] The device speaks everyday questions to the elderly, such as "What did you do today?" and "What is your grandchild's name?"

[1905] When the elderly person answers these questions verbally, the device records the answers.

[1906] Specific examples

[1907] Terminal: "What did you do today?"

[1908] User: "Today I went to the park with my friends."

[1909] Audio data conversion and transmission

[1910] Terminal

[1911] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[1912] The converted text data is sent to the server.

[1913] Specific examples

[1914] Audio data: "Today I went to the park with my friends."

[1915] Text data: "Today I went to the park with my friends."

[1916] Receiving and storing data

[1917] server

[1918] The server receives the text data sent from the terminal.

[1919] The received text data is securely stored in a database for each elderly person.

[1920] Specific examples

[1921] Server: "New conversation data saved to database."

[1922] Data analysis

[1923] server

[1924] The server retrieves the stored text data for analysis.

[1925] The server applies algorithms to assess the elderly person's cognitive function, checking their memory ability and coherence.

[1926] Check the following points:

[1927] Conflicting answers

[1928] Memory lapses (e.g., forgetting grandchildren's names)

[1929] Confused conversation flow

[1930] Specific examples

[1931] Conversation log: "Your grandchildren came over yesterday." "No, no one came." (Inconsistency resolved)

[1932] Alert generation and notification

[1933] server

[1934] Based on the analysis results, the server generates a warning message if an abnormality is detected.

[1935] The server notifies the family or caregiver of the generated warning message.

[1936] Specific examples

[1937] Alert message: "An elderly person has recently exhibited inconsistencies in their conversations and urgent medical attention is recommended."

[1938] Report generation and delivery

[1939] server

[1940] The server periodically generates a report based on the analysis of the elderly person's conversation data.

[1941] The server sends the generated reports to family members or caregivers via email or a dedicated portal.

[1942] Specific examples

[1943] Report reads: "Analysis of conversations over the past month. Multiple inconsistencies and lapses in memory have been observed."

[1944] In this way, the system of the present invention utilizes an automated voice conversation system and data analysis algorithms to assess the cognitive function of elderly people and detect abnormalities early, and by sending appropriate alerts and reports to family members and caregivers, they can take necessary measures promptly.

[1945] The processing flow will be explained below.

[1946] Step 1:

[1947] Terminal

[1948] The terminal activates the automated voice conversation system at a preset time.

[1949] The device will ask the elderly everyday questions aloud, such as "What did you do today?" and "What is your grandchild's name?"

[1950] The user (elderly person) answers these questions orally, and the terminal records the answers in audio format.

[1951] Step 2:

[1952] Terminal

[1953] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[1954] It uses advanced speech recognition algorithms to generate accurate text data from audio data.

[1955] Step 3:

[1956] Terminal

[1957] The terminal transmits the converted text data to the server.

[1958] The text data also includes the elderly person's identification information and the date and time of the conversation.

[1959] Step 4:

[1960] server

[1961] The server receives the text data sent from the terminal.

[1962] The server securely stores the received text data in a database for each elderly person.

[1963] Step 5:

[1964] server

[1965] The server retrieves the stored text data for analysis.

[1966] The server applies natural language processing algorithms to assess memory and cognitive function, specifically checking the following points:

[1967] Conflicting answers

[1968] Memory deficiencies (e.g., forgetting grandchildren's names)

[1969] Confused conversation flow

[1970] Step 6:

[1971] server

[1972] If an abnormality is detected based on the analysis results, the server generates a warning message.

[1973] The generated warning message is sent to the elderly person's family or caregiver.

[1974] Step 7:

[1975] server

[1976] The server periodically (e.g., weekly or monthly) generates detailed reports based on the elderly person's conversation data.

[1977] The server sends the generated reports to family members or caregivers via email or a dedicated portal.

[1978] Through this series of steps, the system will enable daily monitoring of the cognitive function of elderly people, enabling early detection of abnormalities and follow-up.

[1979] Example 1

[1980] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1981] There is a need to detect cognitive decline in the elderly early and provide appropriate care promptly. Conventional methods often rely on home observation and regular medical examinations, which can easily miss cognitive decline. Furthermore, it is difficult to promptly notify family members or caregivers when abnormalities are detected and provide instructions for appropriate care. This reduces the quality of life of the elderly and increases the risk of inadequate care.

[1982] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1983] In this invention, the server includes: means for the elderly person terminal to start speaking at a predetermined time and ask predetermined questions; means for recording the elderly person's spoken responses; means for converting the recorded voice data into text data; means for transmitting the text data to a central processing unit; means for analyzing the received text data and evaluating memory ability and cognitive function; means for generating and notifying warnings if abnormalities are detected based on the analysis results; means for periodically generating reports based on the analysis results of the conversation data and sending them to relevant parties; means for automatically starting the conversation system at a predetermined time using scheduling software; means for analyzing the text data using natural language processing technology and evaluating consistency and memory ability for multiple situations; means for automatically generating and notifying warning messages of various levels based on abnormalities in the analysis results; and means for periodically reporting details of detected abnormalities to relevant parties. This enables automatic and continuous evaluation of the elderly person's cognitive function, early detection of abnormalities, and prompt notification and detailed reports to relevant parties.

[1984] An "elderly terminal" is a device with voice conversation and recording functions designed for use by elderly people.

[1985] The "predetermined time" refers to a specific time that has been set in advance, and is the reference time at which the system automatically starts operating.

[1986] "Speaking" refers to the act of the elderly person terminal asking a question to the elderly person by voice.

[1987] "Spoken response" refers to a verbal response given by the elderly person to the elderly person terminal.

[1988] "Recording" is the act of recording a speech response as audio data.

[1989] "Voice data" refers to digital voice information obtained by recording the elderly person's speech responses.

[1990] "Text data" is voice data converted into text information using natural language processing technology.

[1991] A "central processing unit" is a computer system that analyzes and evaluates received data, and detects and notifies abnormalities.

[1992] "Analysis" refers to the act of evaluating the memory ability and cognitive function of elderly people based on the received text data.

[1993] "Memory ability and cognitive function assessment" refers to the act of checking the consistency of an elderly person's answers and memory deficiencies, and dynamically assessing the state of their cognitive function.

[1994] "Abnormalities" refer to problems with the elderly person's cognitive function that are different from normal, such as inconsistencies or memory loss.

[1995] A "warning" is an alert message that is generated when an abnormality is detected, and is a notification that prompts a prompt response.

[1996] A "report" is a document summarizing the results of analysis of elderly people's conversation data, and is sent to relevant parties on a regular basis.

[1997] "Scheduling software" is a program that automatically starts system operations at pre-set times.

[1998] "Natural language processing technology" is a technology that converts voice data into text information and analyzes text data.

[1999] "Analysis results" are the results of evaluations and judgments made based on text data.

[2000] "Stakeholders" are people or organizations with an interest in the condition of older people, such as their families, caregivers, and medical professionals.

[2001] MODE FOR CARRYING OUT THE INVENTION

[2002] The system of the present invention utilizes an automated voice conversation system and data analysis algorithms to assess cognitive function in elderly people and detect abnormalities at an early stage. The system includes an elderly person terminal (hereinafter referred to as "terminal"), a central processing unit (hereinafter referred to as "server"), and related notification means.

[2003] Conducting a voice conversation

[2004] Terminal

[2005] The terminal automatically starts the automated voice conversation system at a preset time, using scheduling software such as a Cron job for scheduling.

[2006] The device asks everyday questions to the elderly, such as "What did you do today?" and "What is your grandchild's name?", using Google Text-to-Speech technology.

[2007] When the elderly responds to these questions verbally, the device uses a built-in microphone to record the responses, which are then temporarily stored in local storage.

[2008] Specific examples

[2009] Terminal: "What did you do today?"

[2010] User: "Today I went to the park with my friends."

[2011] Audio data conversion and transmission

[2012] Terminal

[2013] The device converts the recorded voice data into text data using natural language processing (NLP) technology, which uses the Google Cloud Speech-to-Text API.

[2014] After converting the data into text, the device sends the text to a remote server using an HTTP POST request, with end-to-end encryption applied to the communication.

[2015] Specific examples

[2016] Audio data: "Today I went to the park with my friends."

[2017] Text data: "Today I went to the park with my friends."

[2018] Receiving and storing data

[2019] server

[2020] The server receives the HTTP POST request sent from the device and extracts the text data from it, using the Apache or Nginx web server software.

[2021] The extracted text data is securely stored in a database for each elderly person. MySQL or PostgreSQL is used as the database management system, and data encryption is performed.

[2022] Specific examples

[2023] Server: "New conversation data saved to database."

[2024] Data analysis

[2025] server

[2026] The server periodically retrieves the stored text data for analysis, using a Cron job.

[2027] The stored text data is then used to run an algorithm that assesses the cognitive function of older adults. This algorithm uses natural language processing (NLP) techniques and libraries such as Python's NLTK and spaCy.

[2028] During processing, check for inconsistent answers, memory gaps, and confused conversation flow.

[2029] Specific examples

[2030] Conversation log: "Your grandchildren came over yesterday." "No, no one came." (Inconsistency resolved)

[2031] Alert generation and notification

[2032] server

[2033] If an abnormality is detected based on the analysis results, the server generates a warning message. This message is automatically generated using a template.

[2034] The server then sends the generated warning message to family members or caregivers via email, SMS, or a dedicated app. Email is sent using the SMTP protocol, and SMS is sent using services such as Twilio.

[2035] Specific examples

[2036] Alert message: "An elderly person has recently exhibited inconsistencies in their conversations and urgent medical attention is recommended."

[2037] Report generation and delivery

[2038] server

[2039] The server periodically generates reports based on the analysis of the elderly's conversation data. The reports are generated in PDF format using libraries such as Python's ReportLab.

[2040] The server generates reports and sends them to family members or caregivers via email or a dedicated portal, which can be run periodically using a scheduler.

[2041] Specific examples

[2042] Report reads: "Analysis of conversations over the past month. Multiple inconsistencies and lapses in memory have been observed."

[2043] Example prompts to input to the generative AI model

[2044] Examples:

[2045] Terminal: "What did you do today?"

[2046] User: "Today I went to the park with my friends."

[2047] Example prompt sentence:

[2048] "Please suggest ways to predict how the elderly person will respond to the device and ensure consistency in the next conversation. Also, please suggest improvements to the algorithm to identify gaps or inconsistencies in memory."

[2049] The system of this invention can effectively evaluate the cognitive function of elderly people and detect abnormalities early. Detailed information can be obtained through conversational interaction, and appropriate nursing and medical measures can be taken in real time based on that information. In addition, appropriate alerts and regular reports enable prompt and accurate responses.

[2050] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2051] Step 1:

[2052] Starting a voice conversation

[2053] The terminal automatically starts the automated voice conversation system at a preset time, using scheduling software such as a Cron job for scheduling.

[2054] Input: Scheduling time

[2055] Operation: The device monitors its internal clock and starts the voice conversation system at the set time.

[2056] Output: Voice conversation system activated

[2057] Step 2:

[2058] Asking a question

[2059] The device uses Google Text-to-Speech technology to ask everyday questions to the elderly, such as "What did you do today?" or "What is your grandchild's name?"

[2060] Input: Preset question content

[2061] How it works: The device passes preset question text to the Google Text-to-Speech API, generates audio data, and speaks it to the elderly person.

[2062] Output: Question spoken to the elderly person

[2063] Step 3:

[2064] Spoken response recording

[2065] When the elderly responds verbally to the questions, the device uses a built-in microphone to record the response, which is then temporarily stored in local storage.

[2066] Input: Verbal response from the elderly person

[2067] How it works: The device captures audio data through the microphone and stores it in local storage.

[2068] Output: Recorded audio data file

[2069] Step 4:

[2070] Converting audio data to text

[2071] The device converts the recorded voice data into text data using natural language processing (NLP) technology, which uses the Google Cloud Speech-to-Text API.

[2072] Input: Recorded audio data

[2073] How it works: The device sends audio data to the Google Cloud Speech-to-Text API and retrieves text data.

[2074] Output: Converted text data

[2075] Step 5:

[2076] Sending text data

[2077] The device then sends the converted text data to a remote server using an HTTP POST request, with end-to-end encryption applied to the communication.

[2078] Input: Converted text data

[2079] How it works: The device generates an HTTP POST request and sends data to the specified endpoint on the server.

[2080] Output: Text data sent to the server

[2081] Step 6:

[2082] Receiving and storing text data

[2083] The server receives the HTTP POST request sent from the device and extracts the text data from it, using the Apache or Nginx web server software.

[2084] Input: HTTP POST request

[2085] What happens: The server parses the POST request and extracts the text data.

[2086] Output: Extracted text data

[2087] Step 7:

[2088] Data storage

[2089] The server securely stores the text data for each elderly person in a database, using MySQL or PostgreSQL as the database management system, and the data is encrypted.

[2090] Input: Extracted text data

[2091] Operation: The server saves the text data in a database. When saved, it associates it with the elderly person's identification information.

[2092] Output: Securely stored text data

[2093] Step 8:

[2094] Regular analysis of data

[2095] The server periodically retrieves the stored text data for analysis, using a Cron job.

[2096] Input: Text data stored in a database

[2097] Operation: The server periodically retrieves the stored text data and runs an analysis program.

[2098] Output: Text data extracted for analysis

[2099] Step 9:

[2100] Cognitive function assessment

[2101] The server runs an algorithm on the text data to assess cognitive function in older adults, using natural language processing (NLP) techniques and libraries such as Python's NLTK and spaCy.

[2102] Input: Text data extracted for analysis

[2103] How it works: The server runs a program that detects inconsistent answers, memory lapses, and disorganized conversation flow.

[2104] Output: Cognitive function assessment results

[2105] Step 10:

[2106] Anomaly detection and alert generation

[2107] If an abnormality is detected based on the analysis results, the server generates a warning message. This message is automatically generated using a template.

[2108] Input: Cognitive function assessment results

[2109] Operation: The server analyzes the evaluation results and generates a corresponding warning message if an anomaly is detected.

[2110] Output: Warning message

[2111] Step 11:

[2112] Warning Notification

[2113] The server then sends the generated warning message to family members or caregivers via email, SMS, or a dedicated app. Email is sent using the SMTP protocol, and SMS is sent using services such as Twilio.

[2114] Input: warning message

[2115] What it does: The server generates email and / or SMS notifications and sends them to interested parties.

[2116] Output: Warning message sent to family members or caregivers

[2117] Step 12:

[2118] Report generation and delivery

[2119] The server periodically generates reports based on the analysis of the elderly's conversation data. The reports are generated in PDF format using libraries such as Python's ReportLab.

[2120] Input: Analysis results of elderly people's conversation data

[2121] How it works: The server periodically aggregates the analysis results of the conversation data and generates a report in PDF format.

[2122] Output: Generated report

[2123] Step 13:

[2124] Report Notification

[2125] The server then sends the generated reports to family members or caregivers via email or a dedicated portal, and this process is also performed periodically using a scheduler.

[2126] Input: Generated report

[2127] How it works: The server sends reports via email or a dedicated portal.

[2128] Output: Report sent to family or caregiver

[2129] (Application example 1)

[2130] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2131] In modern society, efficient and accurate assessment of the cognitive function of elderly people and early detection of cognitive decline occurring in daily life are important challenges. However, conventional interactive systems lack a well-established method for thoroughly assessing the cognitive function of elderly people through their everyday conversations and appropriately notifying the results. Furthermore, they lack a means for quickly issuing warnings when abnormalities are detected and providing necessary information to family members and caregivers. Furthermore, they lack the ability to regularly store and analyze data to ensure the safety of elderly people. Therefore, the present invention aims to solve these challenges.

[2132] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2133] In this invention, the server includes a means for notifying family members and caregivers of speech-recognized text data, a means for periodically saving and analyzing daily conversation data, a means for analyzing the elderly person's conversation data using natural language processing technology, a means for sending warning messages regarding detected abnormalities to family members and caregivers, and a means for sending generated periodic reports to family members and caregivers. This enables detailed assessment of the elderly person's cognitive function through their daily conversations, early detection of abnormalities, and appropriate alert notifications. Furthermore, periodic data saving and analysis can provide comprehensive support for ensuring the safety of the elderly.

[2134] An "elderly terminal" is an electronic device for use by elderly people that is capable of voice input and output.

[2135] A "central processing unit" is a centralized device for analyzing and evaluating speech-recognized data, and functions as a server.

[2136] The "predetermined time" refers to a specific time or period that has been set in advance, at which the system automatically starts operating.

[2137] "Start speaking" means that the system will output pre-programmed messages and ask questions to the senior.

[2138] The "prescribed questions" are questions about the elderly person's daily life and memory, and are useful for assessing cognitive function.

[2139] "Spoken responses" refer to verbal responses given by the elderly to questions posed by the system.

[2140] "Means for recording" refers to a function for recording the elderly person's spoken responses as digital audio data.

[2141] "Means for converting into text data" refers to technology for converting recorded voice data into text information, and primarily utilizes natural language processing technology.

[2142] "Means of analysis" refers to algorithms for assessing the cognitive function of elderly people based on text data.

[2143] "Means for detecting abnormalities" refers to the function of detecting abnormalities such as a decline in cognitive function from the analysis results and instructing on necessary measures.

[2144] "Means for generating and notifying alerts" refers to a system that has the ability to generate alert messages regarding detected abnormalities and promptly notify family members or caregivers.

[2145] "Natural language processing technology" is a technology for processing voice and text data to understand and generate human language.

[2146] A "regular report" is a report summarizing the results of periodic analyses, and is sent to family members or caregivers.

[2147] "Daily conversation data" is a record of regular voice interactions between the elderly person and the system.

[2148] A "generative AI model" is a collection of algorithms trained to perform specific tasks using artificial intelligence.

[2149] A "prompt" is a question or instruction input to a generative AI model, and is a phrase used to encourage a response from the elderly person.

[2150] System Overview

[2151] The system of the present invention mainly consists of an elderly person's terminal, a central processing unit (server), and a notification means. The elderly person's terminal is equipped with an automated voice conversation system that evaluates the elderly person's cognitive function through everyday conversation. The central processing unit analyzes the received data and generates a warning message to notify family members or caregivers if an abnormality is detected. It also has a function to periodically provide analysis results as a report.

[2152] Hardware and software used

[2153] 1. Devices for the elderly:

[2154] Hardware: Smartphone or tablet (including microphone)

[2155] Software: Natural language processing technology (e.g., Google Speech-to-Text API)

[2156] 2. Central Processing Unit (Server):

[2157] Hardware: High-performance network server

[2158] Software: Database systems, data analysis algorithms, notification systems

[2159] Details of data processing and calculation

[2160] Device operation

[2161] The device will automatically start a voice conversation at a specific time. For example, it will ask questions such as "What did you do today?" or "What is your grandchild's name?" When the elderly person answers these questions, the device will record the voice data.

[2162] Audio data conversion and transmission

[2163] The recorded voice data is converted into text data using natural language processing technology within the device. This converted text data is sent to a server. For example, if an elderly person replies, "Today, I went to the park with my friends," the voice is converted into text data that reads, "Today, I went to the park with my friends."

[2164] Data reception and analysis

[2165] The server receives the text data sent from the device. The received data is stored in a database and analysis begins. The server applies algorithms to assess the elderly person's cognitive function and checks their memory ability and coherence. Analysis detects inconsistent answers, memory gaps, and disorganized conversation flow.

[2166] Generate alerts and reports

[2167] If an abnormality is detected based on the analysis results, the server generates a warning message and notifies the family or caregiver. For example, a message may be sent saying, "There have been inconsistencies in the elderly person's recent conversations. We recommend that you seek medical advice as soon as possible." In addition, reports are generated based on the periodic analysis results and sent to the family or caregiver. The report contents may include, "These are the results of the conversation analysis for the past month. Inconsistent answers and lapses in memory have been observed in multiple situations."

[2168] Examples of prompt statements

[2169] Specific examples of prompts include:

[2170] "What did you do today?"

[2171] "What's your grandson's name?"

[2172] "What did you have for dinner last night?"

[2173] This concludes the detailed description of the embodiment of the present invention. This system enables the evaluation of cognitive function through everyday conversations with elderly people and the early detection of abnormalities, allowing family members or caregivers to take prompt action.

[2174] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2175] Processing Steps

[2176] Step 1:

[2177] Starting up elderly device

[2178] Specific operation: The device automatically starts the automated voice conversation system at the specified time, performs initialization processing, and enables the microphone.

[2179] Input: Predetermined time based on system settings

[2180] Output: Automated voice conversation system startup status

[2181] Step 2:

[2182] Prompt sentence utterance

[2183] Specific operation: The device will speak a pre-defined prompt, such as "What did you do today?", to the elderly person.

[2184] Input: Questionnaires saved in the system

[2185] Output: Voice questions for the elderly

[2186] Step 3:

[2187] Spoken response recording

[2188] Specific operation: The elderly person answers questions, and their answers are recorded by the device's microphone.

[2189] Input: Elderly voice response

[2190] Output: Recorded audio data

[2191] Step 4:

[2192] Converting audio data to text

[2193] Specific operation: The recorded audio data is converted into text data using natural language processing technology (such as Google Speech-to-Text API).

[2194] Input: Recorded audio data

[2195] Output: Text data

[2196] Step 5:

[2197] Sending text data

[2198] Specific operation: The converted text data is sent to the server using an HTTP request.

[2199] Input: Converted text data

[2200] Output: Text data sent to the server

[2201] Step 6:

[2202] Receiving and storing data

[2203] Specific operation: The server receives the text data sent from the device and stores it in a database. When stored, the data is classified for each elderly person.

[2204] Input: Text data sent from the terminal

[2205] Output: Text data stored in a database

[2206] Step 7:

[2207] Data analysis

[2208] How it works: The server processes the stored text data using analytical algorithms to assess the elderly person's cognitive function, checking memory ability, consistency of speech, and inconsistencies.

[2209] Input: Text data stored in a database

[2210] Output: Cognitive function assessment results

[2211] Step 8:

[2212] Anomaly detection and alert generation

[2213] Specific operation: If the server detects an anomaly in the analysis results, it generates a warning message. Different levels of warning are set depending on the type and severity of the anomaly.

[2214] Input: Cognitive function assessment results

[2215] Output: Warning message

[2216] Step 9:

[2217] Sending notifications

[2218] Specific operation: The generated warning message is sent to family members or caregivers via email or a dedicated app.

[2219] Input: warning message

[2220] Output: Notification sent to family members and caregivers

[2221] Step 10:

[2222] Generate and send scheduled reports

[2223] Specific operation: The server periodically generates a report based on the analysis results and sends it to the family or caregiver. The report includes the analysis results of the elderly person's conversation data.

[2224] Input: Accumulated data of cognitive function evaluation results

[2225] Output: Regular reports sent to family members and caregivers

[2226] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2227] System Overview

[2228] The system of the present invention assesses cognitive function through voice conversation with elderly people and simultaneously recognizes their emotional state, enabling early detection of abnormalities. The system includes an elderly person terminal (hereinafter referred to as "terminal"), a central processing unit (hereinafter referred to as "server"), and related notification means. It also incorporates an emotion engine that performs emotion recognition.

[2229] Program processing flow

[2230] Voice conversation execution and emotion recognition

[2231] Terminal

[2232] The terminal activates the automated voice conversation system at a preset time.

[2233] The device will ask the elderly everyday questions such as "What did you do today?" or "What is your grandchild's name?"

[2234] When the elderly person answers these questions verbally, the device records the answers in audio format.

[2235] The emotion engine built into the device recognizes emotions in real time from the voice data of the elderly person's speech responses.

[2236] Specific examples

[2237] Terminal: "What did you do today?"

[2238] User: "I went to the park with my friends today." (Emotion of joy is recognized from the tone of voice)

[2239] Audio data conversion and transmission

[2240] Terminal

[2241] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[2242] It uses advanced speech recognition algorithms to generate accurate text data from audio data.

[2243] The converted text data and emotion data are sent to the server.

[2244] Specific examples

[2245] Audio data: "Today I went to the park with my friends."

[2246] Text data: "Today I went to the park with my friends."

[2247] Emotion data: "Joy"

[2248] Receiving and storing data

[2249] server

[2250] The server receives the text data and emotion data sent from the terminal.

[2251] The server securely stores the received data in a database for each elderly person.

[2252] Specific examples

[2253] Server: "New speech and emotion data saved to database."

[2254] Data analysis

[2255] server

[2256] The server retrieves the stored text data and emotion data for analysis.

[2257] The server applies natural language processing algorithms to assess memory and cognitive function, specifically checking the following:

[2258] Conflicting answers

[2259] Memory deficiencies (e.g., forgetting grandchildren's names)

[2260] Confused conversation flow

[2261] The server also analyzes the emotion data recognized by the emotion engine and evaluates the abnormality of the emotional state.

[2262] Specific examples

[2263] Conversation log: "Your grandchildren came over yesterday." "No, no one came." (Inconsistency resolved)

[2264] Emotional data: "Continued anxiety"

[2265] Alert generation and notification

[2266] server

[2267] If an abnormality in cognitive function or emotional state is detected based on the analysis results, the server generates a warning message.

[2268] The generated warning message is sent to the elderly person's family or caregiver.

[2269] Specific examples

[2270] Alert message: "An older adult has recently shown signs of inconsistency and anxiety in their conversations, so immediate medical attention is recommended."

[2271] Report generation and delivery

[2272] server

[2273] The server periodically (e.g., weekly or monthly) generates detailed reports based on the elderly person's conversational and emotional data.

[2274] The server sends the generated reports to family members or caregivers via email or a dedicated portal.

[2275] Specific examples

[2276] Report reads: "Analysis of conversations over the past month has revealed inconsistent responses, memory lapses, and feelings of anxiety on multiple occasions."

[2277] In this way, the system of the present invention combines an automated voice conversation system and an emotion engine to assess the cognitive function and emotional state of elderly people, detect abnormalities early, and send appropriate alerts and reports to family members and caregivers so that necessary measures can be taken promptly.

[2278] The processing flow will be explained below.

[2279] Step 1:

[2280] Terminal

[2281] The terminal activates the automated voice conversation system at a preset time.

[2282] The terminal speaks predetermined questions to the elderly person, such as "What did you do today?" or "What is your grandchild's name?"

[2283] The user (elderly person) answers these questions verbally.

[2284] Execution example:

[2285] Terminal: "What did you do today?"

[2286] User: "Today I went to the park with my friends."

[2287] Step 2:

[2288] Terminal

[2289] The terminal records the user's spoken response.

[2290] The recorded voice data is converted into text using the device's built-in natural language processing (NLP) engine.

[2291] Prepare the converted text data to be sent to the server.

[2292] Execution example:

[2293] Audio data: "Today I went to the park with my friends."

[2294] Text data: "Today I went to the park with my friends."

[2295] Step 3:

[2296] Terminal

[2297] The emotion engine built into the device recognizes emotions from recorded voice data in real time.

[2298] The recognized emotion data is added to the text data and transmitted to the server.

[2299] Execution example:

[2300] Emotion data sent along with the voice data: "Joy"

[2301] Step 4:

[2302] server

[2303] The server receives the text data and emotion data sent from the terminal.

[2304] The received data is stored in a database for each elderly person.

[2305] Execution example:

[2306] Server: "New speech and emotion data saved to database."

[2307] Step 5:

[2308] server

[2309] The server retrieves the stored text data and emotion data for analysis.

[2310] The server applies natural language processing algorithms to assess memory ability and cognitive function from text data.

[2311] Emotional data will also be analyzed to assess the transition of the elderly person's emotional state.

[2312] Things to check:

[2313] Check for contradictory answers.

[2314] Check for memory lapses (e.g., forgetting the names of grandchildren).

[2315] Detecting confused conversation flow.

[2316] Check to see if certain emotions (e.g., anxiety, anger) are occurring frequently.

[2317] Step 6:

[2318] server

[2319] Based on the analysis results, the server generates a warning message if abnormalities are detected in memory ability, cognitive function, or emotional state.

[2320] Notify family members or caregivers of the generated warning message.

[2321] Execution example:

[2322] Alert message: "An older adult has recently shown signs of inconsistency and anxiety in their conversations, so immediate medical attention is recommended."

[2323] Step 7:

[2324] server

[2325] The server periodically (for example, weekly or monthly) generates detailed reports based on the elderly person's conversational and emotional data.

[2326] Generated reports are sent to family members or caregivers via email or a dedicated portal.

[2327] Execution example:

[2328] Report reads: "Analysis of conversations over the past month has revealed inconsistent responses, memory lapses, and feelings of anxiety on multiple occasions."

[2329] Through these steps, the system of the present invention can routinely monitor the cognitive function and emotional state of elderly people and detect abnormalities early. Appropriate alerts and reports can be sent to family members and caregivers, enabling them to take prompt action to ensure the quality of life and safety of the elderly.

[2330] Example 2

[2331] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2332] It is difficult to routinely assess the cognitive function and emotional state of elderly people and detect abnormalities early. In particular, there is a need for a system that can accurately detect cognitive decline and emotional changes through everyday conversations and promptly notify family members and caregivers. There is also a need for detailed data analysis and reports that enable appropriate responses tailored to the individual condition of each elderly person.

[2333] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for sending periodically generated reports to family members or caregivers, a means for generating and notifying different levels of warnings, a means for analyzing received text data and emotional data to evaluate memory ability and cognitive function, and a means for analyzing emotional data to evaluate emotional state. This makes it possible to quickly and accurately detect changes in the elderly person's cognitive function and emotions, to promptly notify if an abnormality is found, and to provide a detailed report.

[2334] An "elderly terminal" is a device that can carry out voice conversations with the elderly, record audio, convert text, recognize emotions, and transmit data.

[2335] "Predetermined time" refers to a specific date, time or time period for which the system is set.

[2336] "Start speaking" means that the terminal automatically starts a voice conversation.

[2337] The "predetermined question" refers to a question that is set in advance in the system.

[2338] "Recording a speech response" means that the terminal records the voice of the elderly person's response.

[2339] "Natural language processing technology" is a general term for computer technology for converting voice data into text data.

[2340] "Text data" is recorded voice data converted into text information.

[2341] "Emotion data" refers to information about emotions recognized from voice data.

[2342] "Central Processing Unit" is the main computer that processes and analyzes the received data and notifies the results.

[2343] "Analyzing" means analyzing data in detail and extracting meaning and patterns.

[2344] "Memory ability" refers to a subject's ability to accurately recall past information.

[2345] "Cognitive function" is a general term for intellectual functions such as thinking, understanding, learning, and memory.

[2346] "Emotional state" refers to the subject's current emotional state.

[2347] "Generating an alert" means creating a notification mechanism when an anomaly is detected.

[2348] "Notify" refers to sending the generated alert to a family member or caregiver.

[2349] "Generating a report" means creating a report based on the analysis results.

[2350] The present invention is a system for evaluating cognitive function and emotional state through voice conversation with elderly people and detecting abnormalities at an early stage. Specific embodiments of the present invention will be described below.

[2351] System configuration

[2352] The system of the present invention includes an elderly terminal, a central processing unit, and related notification means, and also incorporates an emotion engine for emotion recognition.

[2353] Devices for the elderly

[2354] The elderly terminal has a function to automatically start the voice conversation system at a predetermined time. For example, the system starts up at 9:00 AM every day and starts a conversation with the elderly. The terminal asks everyday questions such as:

[2355] For example: "What did you do today?"

[2356] For example: "What is your grandchild's name?"

[2357] The user (elderly person) answers these questions verbally, and the device records the voice. The recorded voice data is passed through an emotion engine built into the device to recognize emotions in real time. For example, if the user answers, "I went to the park with my friends today," the emotion of joy is recognized from the tone of the voice.

[2358] Audio data conversion and transmission

[2359] The recorded voice data is converted into text data using natural language processing (NLP) technology, using software such as the Google Speech-to-Text API and DeepSpeech (an open-source library developed by Mozilla). Emotion data is generated along with the text data, and this data is sent to a central processing unit (server).

[2360] Central Processing Unit (Server)

[2361] The server receives the text and emotion data sent from the device. The received data is securely stored in a database (e.g., MySQL database) for each elderly person. The server then retrieves the stored data for analysis and uses natural language processing algorithms (e.g., SpaCy) to evaluate memory ability and cognitive function. The analysis includes the following points:

[2362] Conflicting answers

[2363] Memory deficiencies (e.g., forgetting grandchildren's names)

[2364] Confused conversation flow

[2365] Furthermore, the emotional data recognized by the emotion engine is also analyzed to evaluate abnormalities in the emotional state.

[2366] Alerts and Notifications

[2367] If the server detects any abnormalities in cognitive function or emotional state based on the analysis results, it will generate a warning message and notify the family or caregiver, such as, "The elderly person's recent conversations have shown inconsistencies and feelings of anxiety, so we recommend that you seek medical attention immediately."

[2368] Report generation and delivery

[2369] Periodically (e.g., weekly or monthly), the server generates detailed reports based on the elderly person's conversation and emotion data. A report generation tool (e.g., Tableau) is used to generate the reports. The generated reports are sent to family members or caregivers via email or a dedicated portal. For example, a report such as, "Conversation analysis results for the past month. Inconsistent answers, memory lapses, and feelings of anxiety were observed in multiple situations."

[2370] Examples of prompt statements

[2371] Using a generative AI model, you can input prompts like this:

[2372] "Describe a natural language processing algorithm for analyzing speech data from older adults and assessing their emotional state."

[2373] The system of the present invention combines an automated voice conversation system and an emotion engine to assess the cognitive function and emotional state of elderly people, detect abnormalities early, and send appropriate alerts and reports to family members and caregivers, enabling them to take necessary measures promptly.

[2374] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2375] Step 1: Voice conversation execution and emotion recognition (device)

[2376] The terminal activates the automated voice conversation system at a predetermined time.

[2377] Input: A specific time (e.g., 9:00 AM every day)

[2378] What it does: Your device will speak everyday questions like, "What did you do today?"

[2379] Output: The spoken question

[2380] The user verbally answers questions posed by the terminal.

[2381] Input: Terminal question

[2382] Action: The user responds, "Today I went to the park with my friends."

[2383] Output: Audio data (user's response)

[2384] The terminal records the user's response in audio format.

[2385] Input: User's voice response

[2386] How it works: The device's microphone captures audio data and saves it as a recording.

[2387] Output: Recorded audio data

[2388] The device's emotion engine analyzes the recorded voice data in real time and recognizes emotions.

[2389] Input: Recorded audio data

[2390] How it works: Emotion recognition algorithms analyze voice tone, pitch, speed, etc.

[2391] Output: Emotion data (e.g., "joy")

[2392] Step 2: Convert and send audio data (terminal)

[2393] The device converts the recorded voice data into text data using natural language processing (NLP) technology.

[2394] Input: Recorded audio data

[2395] What it does: Converts audio data into text using the Google Speech-to-Text API.

[2396] Output: Text data (e.g., "Today I went to the park with my friends.")

[2397] The terminal transmits the converted text data and emotion data to the server.

[2398] Input: Text data, emotion data

[2399] Operation: The terminal packets the data and sends it to the server over the network.

[2400] Output: Data sent to the server

[2401] Step 3: Receiving and storing data (server)

[2402] The server receives the text data and emotion data sent from the terminal.

[2403] Input: Text data, emotion data

[2404] How it works: The server captures packets on the network and analyzes the data.

[2405] Output: Received data

[2406] The server securely stores the received data in a database for each elderly person.

[2407] Input: Received data

[2408] What it does: Saves data to a MySQL database.

[2409] Output: Securely stored data

[2410] Step 4: Data analysis (server)

[2411] The server retrieves the stored text data and emotion data for analysis.

[2412] Input: saved text data, emotion data

[2413] What it does: Reads data from a database.

[2414] Output: Data to be analyzed

[2415] The server analyzes the text data using natural language processing algorithms (e.g., SpaCy) to assess memory ability and cognitive function.

[2416] Input: Data to be analyzed

[2417] How it works: It applies natural language processing algorithms to detect inconsistencies, memory lapses, and confused conversation flow.

[2418] Output: Analysis results (e.g., "Inconsistencies detected," "Memory gaps observed")

[2419] The server also analyzes the emotion data recognized by the emotion engine and evaluates abnormalities in the emotional state.

[2420] Input: Emotion data

[2421] How it works: Analyzes emotion data and detects abnormal emotional states (e.g., "persistent anxiety").

[2422] Output: Emotion analysis results

[2423] Step 5: Alert Generation and Notification (Server)

[2424] If an abnormality in cognitive function or emotional state is detected based on the analysis results, the server generates a warning message.

[2425] Input: Analysis results, emotion analysis results

[2426] Operation: Generates a warning message and determines the content of the message depending on the severity of the abnormality.

[2427] Output: Generated warning message

[2428] The server sends the generated alert message to the family member or caregiver.

[2429] Input: warning message

[2430] Action: Sends notifications via email and SMS.

[2431] Output: Warning message sent

[2432] Step 6: Generate and send report (server)

[2433] The server periodically generates detailed reports based on the elderly person's speech and emotion data.

[2434] Input: Saved conversation data, emotion data

[2435] Behavior: Use a report generation tool (e.g., Tableau) to visualize data and create reports.

[2436] Output: Generated report

[2437] The server sends the generated report to the family member or caregiver.

[2438] Input: Generated report

[2439] How it works: Reports are delivered via email and a dedicated portal.

[2440] Output: Report sent

[2441] (Application example 2)

[2442] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2443] Systems that assess the cognitive function and emotional state of elderly people need to be able to detect abnormalities in real time and quickly notify family members or caregivers, ensuring the safety of the elderly. Conventional systems have difficulty assessing the elderly's emotional state in detail and notifying them in a timely manner. Furthermore, there is a need for integration into a home security system to monitor the safety of elderly people at home.

[2444] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing received text data and evaluating memory ability and cognitive function, means for analyzing emotion data and evaluating emotional abnormalities, and means for generating and notifying an alert when an abnormality is detected based on the analysis results. This makes it possible to detect abnormalities in the cognitive function and emotional state of an elderly person in real time and quickly notify family members or security service operators.

[2445] An "elderly terminal" is an information processing terminal designed for use by elderly people, and is a device that includes voice recognition and emotion recognition functions.

[2446] The "predetermined time" refers to a specific time period that has been set in advance, during which the system will automatically operate.

[2447] "Means for initiating speech and asking specific questions" refers to the function of the elderly terminal to automatically initiate a voice dialogue and ask specific questions to the elderly.

[2448] "Means for recording spoken responses" refers to the function of the elderly person terminal to record the elderly person's responses in audio format.

[2449] "Means for converting recorded voice data into text data" refers to a function that converts recorded voice into text data using voice recognition technology.

[2450] "Means for recognizing emotions" refers to the function of analyzing and recognizing the speaker's emotional state from voice data.

[2451] "Means for transmitting text data and emotion data to a central processing unit" refers to a function for transmitting converted text and analyzed emotion information to a remote server.

[2452] The "central processing unit" refers to a server or the like that receives data sent from the elderly person's terminal and performs analytical processing.

[2453] "Means for analyzing text data to assess memory ability and cognitive function" refers to an algorithm that analyzes received text data to assess the memory ability and cognitive function of elderly people.

[2454] "Means for analyzing emotional data and evaluating emotional abnormalities" refers to a function for analyzing received emotional data and detecting abnormalities in the emotional state.

[2455] "Means for generating and notifying an alert when an abnormality is detected based on the analysis results" refers to the function of generating an alert and notifying family members or caregivers when an abnormality is detected based on the results of data analysis.

[2456] "Means for sending as a report" refers to the function of periodically sending a report summarizing the analysis results to family members or caregivers.

[2457] System configuration

[2458] The system required to implement this invention includes an "elderly terminal," a "central processing unit (server)," and a "notification means." Furthermore, it incorporates engines for speech recognition, natural language processing (NLP), and emotion recognition. Specifically, it uses the Google Cloud Speech-to-Text API for speech recognition, the Microsoft Emotion API or Amazon Rekognition for emotion recognition, and Firebase or Amazon DynamoDB for the database.

[2459] Process Overview

[2460] 1. Start a voice interaction:

[2461] The elderly device automatically starts up at a specified time and asks the elderly specific questions such as "What did you do today?" or "What is your grandchild's name?" These questions are spoken using voice synthesis technology.

[2462] 2. Audio data collection and text conversion:

[2463] When the elderly responds verbally to these questions, the device records their responses and then converts the recorded audio data into text using the Google Cloud Speech-to-Text API.

[2464] 3. Emotion recognition:

[2465] At the same time, an emotion engine (e.g., Amazon Rekognition) is used to recognize the elderly person's emotions from the voice data in real time, and the recognition results are saved as emotion data.

[2466] 4. Data transmission and storage:

[2467] The converted text data and emotion data are sent from the elderly person's device to a central processing unit (server), which receives the data and stores it securely in a database such as Firebase.

[2468] 5. Data Analysis:

[2469] The central processing unit retrieves the stored data and analyzes the text data to assess memory and cognitive function, specifically for inconsistent responses and memory lapses, and analyzes the emotional data to assess abnormal emotional states.

[2470] 6. Alert Generation and Notification:

[2471] If an abnormality is detected based on the analysis results, the system automatically generates an alert and notifies family members, caregivers, and in some cases security service operators via an SMS API (such as Twilio) or Firebase Cloud Messaging.

[2472] 7. Report generation and sending:

[2473] Detailed reports based on the elderly person's conversation and emotion data are generated periodically (e.g., weekly or monthly) and sent to family members or caregivers via email or a dedicated portal.

[2474] Specific examples

[2475] For example, if an elderly person says, "Today I went to the park with my friends," the system records the audio data and then uses the Google Cloud Speech-to-Text API to convert it into text data: "Today I went to the park with my friends." At the same time, Amazon Rekognition is used to identify the emotion as "joy" and send this data to the server. The server analyzes the text data and emotion data and evaluates whether there are any anomalies. If an anomaly is detected, for example, if the emotion "anxiety" persists, the system uses Twilio's SMS API to send an alert to the family member stating, "Anxiety has been detected in the elderly person's recent conversations, and immediate attention is required."

[2476] Prompt Sentence Examples

[2477] "What did you do today?"

[2478] "What's your grandson's name?"

[2479] "How are you feeling today?"

[2480] Through these example prompts, the system of the invention can efficiently assess the cognitive function and emotional state of elderly people and quickly provide this information to their families and caregivers.

[2481] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2482] Step 1:

[2483] The elderly terminal begins speaking at a predetermined time. Using speech synthesis technology, the elderly terminal speaks prompt sentences such as "What did you do today?" or "What is your grandchild's name?" The input is the prompt sentence, and the output is a voice response from the elderly person.

[2484] Step 2:

[2485] The elderly person's speech response is recorded on the elderly person terminal. The input is the elderly person's voice data, and the output is the recorded voice file.

[2486] Step 3:

[2487] The recorded voice data is converted into text data. The elderly device converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is an audio file, and the output is text data.

[2488] Step 4:

[2489] At the same time, the elderly device uses an emotion engine (e.g., Amazon Rekognition) to recognize emotions from voice data. The input is the voice file, and the output is emotion data.

[2490] Step 5:

[2491] The elderly person's device sends the converted text data and emotion data to the server. The input is the text data and emotion data, and the output is the data sent to the server.

[2492] Step 6:

[2493] The server analyzes the received text data and evaluates memory ability and cognitive function. Specifically, it checks for inconsistent answers and memory deficiencies. The input is text data, and the output is the evaluation results of memory ability and cognitive function.

[2494] Step 7:

[2495] The server analyzes the emotional data and evaluates the abnormality of the emotional state. The input is the emotional data, and the output is the evaluation result of the emotional state.

[2496] Step 8:

[2497] If the server detects an abnormality based on the analysis results, it generates and notifies a warning. When an abnormality is detected, it notifies family members or security service operators using Twilio's SMS API or Firebase Cloud Messaging. The input is the evaluation result, and the output is the notified warning message.

[2498] Step 9:

[2499] The server periodically generates detailed reports based on the elderly person's conversation and emotion data and sends them to family members or caregivers via email or a dedicated portal. The input is the analyzed data, and the output is the sent report.

[2500] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2501] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2502] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2503] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2504] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2505] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2506] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2507] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2508] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2509] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2510] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2511] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2512] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2513] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2514] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2515] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2516] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2517] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2518] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2519] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it g...

Claims

1. a means for the elderly person terminal to start speaking at a predetermined time and ask a predetermined question; A means for recording the elderly person's speech response in the elderly person terminal; A means for converting the recorded voice data into text data in the elderly person's device; A means for transmitting the converted text data to the central processing unit in the elderly person's terminal; A means for the central processing unit to analyze the received text data and evaluate memory ability and cognitive function; A system including a means for generating and notifying an alert when a central processing unit detects an abnormality based on the analysis results.

2. The system according to claim 1 , wherein the elderly person terminal further comprises means for periodically transmitting the analysis results as a report to a family member or a caregiver.

3. 2. The system according to claim 1, wherein the central processing unit further comprises means for generating and notifying different levels of warning depending on the type and degree of the abnormality.

Images