System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving input data from a user; means for analyzing the received data to identify past memories; means for generating content to stimulate memories based on the analysis; and means for providing reminders or notifications to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] This invention aims to provide an effective means of alleviating symptoms such as forgetfulness, confusion, anxiety, and associated irritability experienced by elderly people with dementia, thereby improving the quality of life for elderly people and reducing the burden on their families and caregivers. It also aims to address the growing needs of the dementia care market by providing personalized memory support. [Means for solving the problem]

[0005] The memory support system of the present invention includes a means for receiving input data from a user, a means for analyzing the received data to identify past memories, a means for generating content for stimulating memory based on the analysis results, and a means for providing reminders and notifications to the user.These problems can be effectively solved by further including a means for recording daily events, a means for providing information when specific information cannot be remembered during a conversation, a means for stimulating memory using past photographs and audio data, and a means for reminding the user of routines and events at specific times.

[0006] "Input data" refers to information or data provided by a user, and specifically includes voice, text, photographs, and the like.

[0007] "Means for receiving" refers to the function by which the terminal receives data from the user and transmits it to the server.

[0008] "Means of analysis" refers to a function that uses AI algorithms to process the data received by the server and identify the user's past memories and important events.

[0009] "Content for stimulating memory" refers to materials that activate memories when presented to the user, such as photographs, voice memos, and text information related to the user's past memories or important events.

[0010] "Means for providing reminders and notifications" refers to a function that notifies the user of reminders and notifications generated by the server via the terminal.

[0011] "Means for recording" is a function for saving the user's daily happenings and events so that they can be referenced later.

[0012] "Means for providing specific information during a conversation" refers to a function that allows the terminal to instantly present information that the user needs or cannot remember during a conversation.

[0013] "Means to stimulate memory using photographs and audio data" is a function that activates the user's memory and helps them recall past events by displaying or playing the user's past photographs and audio data.

[0014] "Timely reminders" refers to reminders that are sent at pre-set times to help users remember things, such as notifications about regular routines or important events. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to an integrated memory support system for supporting the memory of elderly people with dementia, specifically, a system that receives and analyzes data from users, generates memory-stimulating content, and provides timely reminders and notifications.

[0037] Overall system structure

[0038] The system mainly consists of three components: a server, a terminal, and a user. The server processes and stores all data, and the terminal provides an interface for the user, through which the user interacts with the system.

[0039] Server Processing

[0040] 1. Data collection and analysis

[0041] The server receives data provided by the user through the terminal, including voice, text, and photos.

[0042] The server then analyzes the received data using AI algorithms, which identify important memories and events from the user's past.

[0043] 2. Recording New Events

[0044] The server stores the data in real time as users enter their daily events, recording their daily routines and new events.

[0045] 3. Providing memory support

[0046] The server generates appropriate content based on the user's current situation, which stimulates the user's memory based on past photos and voice memos.

[0047] For example, a user who is in a particular location may be sent content that reminds them of a past event related to that location.

[0048] Terminal handling

[0049] 1. Data input

[0050] The device accepts the user's voice and text input and prepares it for transmission to the server. The device temporarily stores the input data and manages transmission to the server.

[0051] 2. Reminders and notifications

[0052] The device will display reminders and notifications from the server to the user at the appropriate time, such as reminders to help users remember their daily routines.

[0053] 3. Conversation support

[0054] The device can provide users with information if they cannot remember specific information during a conversation, such as the name of a specific person or supplementary information about a past event.

[0055] User operations

[0056] 1. Entering historical data

[0057] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[0058] 2. Recording everyday events

[0059] Users input their daily events into the terminal, and this input data is sent to the server and recorded.

[0060] 3. Check your reminders

[0061] Users can check reminders from their devices and carry out necessary events and activities in a timely manner.

[0062] Specific examples

[0063] 1. Helping people recall past memories

[0064] A user uploads photos of past trips to the device, and the server analyzes the photos and identifies them as trips taken in 2010. If the user then asks a question related to the travel destination, the device will notify them, "This is a trip taken in 2010."

[0065] 2. Recording New Events

[0066] When a user types "I went for a walk with a friend today" into the device, that data is sent to the server and recorded. Later, when the user asks "When was the day I went for a walk with a friend?", the device provides that information.

[0067] 3. Providing reminders

[0068] The system records the user's daily medication intake times, and when the time approaches, the server sends a reminder to the device, which then notifies the user, "It's time to take your medicine."

[0069] As described above, the system of the present invention provides an effective means of supporting the memory of elderly people with dementia and improving the quality of their daily lives.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] Users input past photos, voice memos, and text data into the device, select data related to past events and memories, and click the upload button.

[0073] Step 2:

[0074] The device temporarily stores the data received from the user and prepares it for transmission to the server. The device checks the data format and performs error checking to ensure smooth transmission.

[0075] Step 3:

[0076] The device sends the prepared data to the server. The server receives the data and stores it in a database. The server checks the integrity of the data and checks for incomplete data.

[0077] Step 4:

[0078] The server then runs the data through an AI algorithm to analyze it, identifying key events and memories from the user's past and extracting relevant information.

[0079] Step 5:

[0080] The server uses the analysis results to generate content to stimulate the user's memory, such as photos of past travel destinations or text related to events.

[0081] Step 6:

[0082] The user inputs new events and daily events into the device, inputting the activities they performed that day and their feelings in text, and then clicks the send button.

[0083] Step 7:

[0084] The device receives new data, temporarily stores it, and then sends it to the server. The server receives the new data and records it in the database. This updates the user's daily life history.

[0085] Step 8:

[0086] To utilize the system's reminder function, users set specific events or routines on their devices, such as setting a daily medication intake time or scheduling a regular health checkup.

[0087] Step 9:

[0088] The server sets a timer based on the user's settings, generates a reminder at the appropriate time, and sends the reminder content to the device, causing a notification to be displayed.

[0089] Step 10:

[0090] The device displays a reminder notification to the user at the specified time. The user checks the notification on the device and takes the necessary action, such as taking medicine.

[0091] Step 11:

[0092] If a user cannot remember a particular piece of information during a conversation, they can ask for help from the terminal by typing a question into the terminal and sending it.

[0093] Step 12:

[0094] The server receives the user's query and searches the database for the relevant information. The server responds quickly and identifies the information the user is looking for.

[0095] Step 13:

[0096] The server sends the identified information to the terminal, which displays it to the user. The user confirms the displayed information and continues the conversation.

[0097] Through the above steps, the system of the present invention can effectively support the memory of elderly people with dementia and improve the quality of their daily lives.

[0098] Example 1

[0099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] The present invention aims to provide a system that supports elderly people with dementia in recalling past memories in their daily lives and in carrying out daily events and routines without forgetting them. In particular, there is a need for an effective means for users to easily recall their past memories and appropriately manage their daily actions and events, but existing technologies have not been able to completely solve this problem.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0102] In this invention, the server includes a means for receiving input data from a user, a means for analyzing the received data using multiple analysis techniques, and a means for recording the user's behavioral patterns and routines in real time based on the analysis results, thereby enabling appropriate support for the user's memory and improving the quality of daily life.

[0103] "User input data" refers to information such as voice, text, and photos provided by the user through the terminal.

[0104] "Analyzing received data" means that the server analyzes the data received from the user using natural language processing, image recognition, etc.

[0105] "Identifying past memories" refers to revealing important events or memories from the user's past based on the data analyzed by the server.

[0106] "Generating content to stimulate memories" means that the server creates content such as photos, text, and audio that are generated for the purpose of stimulating the user's memories based on past memories.

[0107] "Providing reminders and notifications to the user" means that the server generates reminders and notifications, sends them to the user at the appropriate time, and the terminal displays them.

[0108] "Transmitting to a server via a network" means that the terminal transmits the user's input data to a server using a communication means such as the Internet.

[0109] "Analysis using multiple analysis technologies" means that the server analyzes data by combining different technologies such as natural language processing (NLP), image recognition, and voice analysis.

[0110] "Recording user behavior patterns and routines in real time" means that the server immediately saves and manages the user's daily behavior and routines based on the analysis results.

[0111] The present invention relates to an integrated memory support system for supporting the memory of elderly people with dementia. The system receives input data from a user, analyzes the data using multiple analytical techniques, identifies the user's past memories, generates content to stimulate memory based on the analysis results, and provides reminders and notifications at appropriate times.

[0112] Overall system structure

[0113] The system mainly consists of three components: a server, a terminal, and a user. The server processes and stores all data, and the terminal provides an interface for the user, through which the user interacts with the system.

[0114] Server Processing

[0115] The server processes the input data using multiple analysis technologies, including natural language processing (NLP), image recognition, and voice analysis. For example, it can analyze photo data using Google Cloud Vision API or Amazon Rekognition, and analyze text data using Google Natural Language API.

[0116] Data collection and analysis

[0117] The server receives data provided by the user through the device. This data includes voice, text, photos, etc. For example, if a user uploads photos of their trip, the server receives the photo data and analyzes it using image recognition technology. This analysis can identify important memories and events from the user's past.

[0118] Recording new events

[0119] The server stores the data in real time as the user inputs their daily events. For example, if a user inputs "I went for a walk with a friend today," the server analyzes the data and stores it in a MySQL database or similar.

[0120] Providing memory support

[0121] Based on the analysis results, the server generates content to stimulate the user's memory. This content can include past photos and voice memos, and is sent to the user via email or push notification. For example, when a user is in a specific location, they will receive a notification about past events related to that location.

[0122] Terminal handling

[0123] The device accepts voice and text input from the user, temporarily stores it in local storage, and then transmits the input data to a server over the network, for example, using Google Firebase or a REST API.

[0124] Data Input

[0125] The device receives the user's voice and text input and prepares it to be sent to the server. For example, if the user says, "Set the time to take my medicine," the device sends that data to the server.

[0126] Reminders and notifications

[0127] The device will display reminders and notifications from the server to the user at appropriate times, for example, a pop-up notification saying "It's time to take your medicine" at a specific time.

[0128] User operations

[0129] Users can use the device to upload past photos and voice memos. This data is automatically sent to the server, where it is analyzed and saved. Users can also enter daily events into the device, and the data is sent to the server and recorded in real time.

[0130] Specific examples

[0131] 1. Helping people recall past memories

[0132] A user uploads photos of a trip from 2010 to their device. The server analyzes the photos and identifies them as trips from 2010. Then, when the user asks about the trip, the device will say, "This is a trip from 2010."

[0133] Prompt Sentence Examples

[0134] Here are some photos from a trip we took in 2010. Please provide a suitable description when users ask about the destination.

[0135] 2. Recording New Events

[0136] A user enters "I went for a walk with a friend today" into a device. This information is sent to a server and recorded. Later, when the user asks "When was the day I went for a walk with a friend?", the device provides that information.

[0137] Prompt Sentence Examples

[0138] Record today's date and the information that you "went for a walk with a friend." When you ask for this information later, please let us know when you went for a walk.

[0139] 3. Providing reminders

[0140] The user sets the time to take their medicine daily, and when that time approaches, the server sends a reminder to the device, which then notifies them that it is time to take their medicine.

[0141] Prompt Sentence Examples

[0142] Set a time for the user to take their medicine, and when that time approaches, display a reminder saying "It's time to take your medicine."

[0143] As described above, the system of the present invention provides an effective means for supporting the memory of elderly people with dementia and improving their quality of daily life.

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1: User enters data into the terminal

[0146] A user opens an application on their device and enters "I went for a walk with a friend today" into a text box. The input data (text or voice) is temporarily stored in the device's local storage. The user provides the text "I went for a walk with a friend today" as input data, and the data is temporarily stored in the device.

[0147] Step 2: The device sends the data to the server

[0148] The device sends the text data entered by the user to the server via the network. The device uses an HTTPS request to send the entered text data, "I went for a walk with a friend today," in JSON format to the server. Specifically, the device calls a REST API to send the data to a specific endpoint on the server.

[0149] Step 3: The server receives and analyzes the data

[0150] The server stores the received text data in Google Cloud Storage or Amazon S3. The server uses a Python script to analyze the text data using the Google Natural Language API. The server receives the text data "I went for a walk with a friend today" as input for analysis, analyzes the text, and extracts event information containing the keywords "friend" and "walk."

[0151] Step 4: The server generates notifications and reminders based on the analysis results.

[0152] Based on the analysis results, the server records the user's behavioral patterns and saves them as new events. For example, the event information "went for a walk with a friend" is saved in a MySQL database. The server generates a reminder for the next event, allowing the user to look back on that day at a later date. The server receives the event information from the analysis results as input, generates reminder information as output, and saves it in the database.

[0153] Step 5: The server sends notifications and reminders to the device

[0154] The server generates reminder information and sends it to the device via the network. A push notification is sent using a dedicated notification API (such as Firebase Cloud Messaging). The server receives reminder information as input and generates and sends a notification message to the device as output.

[0155] Step 6: The device displays notifications and reminders to the user

[0156] The device displays notification messages received from the server to the user as popups or audio alerts. For example, it displays a reminder such as "It's time to take your medicine" on the screen to provide an alert to the user. It receives notification messages received from the server as input and displays visual and audio notifications to the user as output.

[0157] The above are the specific operations at each processing step of this system, which effectively provides memory support for elderly people with dementia.

[0158] (Application example 1)

[0159] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] Elderly people with dementia often experience difficulty in daily life due to memory loss and confusion. Furthermore, inability to recall past events or people can lead to problems with communication. This can lead to a decline in quality of life and increased psychological stress. Therefore, a system that can solve these problems and support the memory of elderly people with dementia is needed.

[0161] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0162] In this invention, the server includes means for receiving input data from a user, means for analyzing the received data and identifying past memories, means for displaying information about specific people using a face recognition function, means for recording daily events using a voice recognition function, means for generating content to stimulate memories based on the analysis results, means for displaying past events based on the user's location and time, and means for providing reminders and notifications to the user. This makes it easier for elderly people with dementia to remember daily events and past events and people, thereby improving their quality of life.

[0163] The "means for receiving input data from a user" is a device that has the function of receiving data such as voice, text, and photographs provided by a user via a terminal.

[0164] "Means for analyzing received data and identifying past memories" refers to a mechanism for analyzing received data using an AI algorithm to identify important memories and events from the user's past.

[0165] "Means for displaying information about a specific person using a facial recognition function" refers to technology that uses a camera to detect the face of a specific person and displays information related to that person.

[0166] The "means for recording daily events using a voice recognition function" is a system that uses a microphone to recognize a user's speech, converts it into text data, and records daily events.

[0167] "Means for generating content to stimulate memory based on analysis results" is a function that automatically creates content such as photos and voice memos to evoke the user's memories based on the analysis results.

[0168] The "means for displaying past events based on the user's location and time" is a system that presents past events related to a location or time point according to the user's current location or a specific time.

[0169] "Means for providing reminders and notifications to users" refers to a mechanism that has the function of sending visual and audio notifications to users at appropriate times according to pre-set schedules and routines.

[0170] "Means for providing visual and audio reminders via smart devices" refers to technology for providing both visual and audio reminders using devices such as smartphones or smart glasses.

[0171] "Means for providing specific information during a conversation using a voice output function" refers to a technology that uses a speaker to audibly convey information that a user needs during a conversation.

[0172] This invention is an integrated memory support system for supporting the memory of elderly people with dementia, and is realized by combining three main elements: a server, a terminal, and a user.

[0173] Overall system structure

[0174] This system consists of the following hardware and software:

[0175] Hardware: Smart glasses (with camera, microphone, and speaker), server (data processing and storage), smartphone

[0176] Software: Python, OpenCV, face_recognition, Vosk (voice recognition)

[0177] Server Processing

[0178] 1. Data Receipt and Analysis:

[0179] The server receives data such as voice, text, and photos provided by the user via the device, which is then analyzed using AI algorithms to identify important memories and events from the user's past.

[0180] 2. New Events Recorded:

[0181] The server uses a voice recognition function to record the user's daily activities in real time, analyzes the voice data, and saves it as text data.

[0182] 3. Providing memory support:

[0183] Based on the analysis results, the server generates content (such as photos and voice memos) to stimulate the user's memory, and also displays past events based on the user's location or a specific time.

[0184] Terminal handling

[0185] 1. Data input and transmission:

[0186] The device (smart glasses or smartphone) receives the user's voice and text input and sends it to the server. The input data is temporarily stored on the device, and transmission to the server is managed.

[0187] 2. Facial Recognition and Information Display:

[0188] Using the device's camera, it recognizes the face of a specific person and displays information related to that person, helping users remember the name of the person in front of them and past events.

[0189] 3. Reminders and notifications:

[0190] The device will display reminders and notifications sent from the server to the user visually and audibly at the appropriate time, for example, a reminder to take medicine at a specific time.

[0191] User operations

[0192] 1. Enter historical data:

[0193] Users can upload past photos and voice memos to the device, and this data is sent to a server and stored for analysis.

[0194] 2. Recording daily events:

[0195] Users can record daily events by voice input into the device, and this data is sent to the server and stored in a database.

[0196] 3. Check your reminders:

[0197] Users can check reminder notifications from their devices and perform necessary events and activities in a timely manner.

[0198] Specific examples

[0199] 1. Helping people recall past memories:

[0200] When a user wears the smart glasses and walks around town, photos and events from previous visits to the same places are automatically displayed, evoke memories of those times by showing photos of past trips.

[0201] 2. New Events Recorded:

[0202] When a user says, "I went for a walk with a friend today," the data is sent to the server and recorded. Later, when the user asks, "When was the day I went for a walk with a friend?", the device provides that information.

[0203] 3. Providing Reminders:

[0204] At a specific time, the smart glasses' voice function will remind the user, saying, "It's time to take your medicine," to help them remember.

[0205] Prompt Sentence Examples

[0206] Project: Memory Spot

[0207] Objective: To develop a system that uses smart glasses to support memory in elderly people with dementia.

[0208] procedure:

[0209] Uses face recognition library (face_recognition) and speech recognition library (Vosk).

[0210] Faces are recognized in the camera frame and the recognition results are notified via voice.

[0211] Record daily events using voice input from a microphone.

[0212] Regular voice reminders.

[0213] In this way, the system of the present invention is designed to enable elderly people with dementia to live their daily lives more efficiently and safely.

[0214] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0215] Program processing flow

[0216] Step 1:

[0217] The user inputs data into the device, such as voice input or photo upload. The device then uses the microphone to capture the voice and the camera to take a photo. The input data is temporarily stored on the device.

[0218] Input: User's voice and photo data

[0219] Output: Temporarily saved audio and image files

[0220] Step 2:

[0221] The device then sends the temporarily stored data to the server using a communication protocol via the Internet.

[0222] Input: Temporarily saved audio and image files

[0223] Output: Audio and image files sent to the server

[0224] Step 3:

[0225] The server converts the received voice data into text using the Vosk speech recognition library, and the converted text data is stored in a database.

[0226] Input: Audio file

[0227] Output: Text data

[0228] Step 4:

[0229] The server analyzes the received photo data using AI algorithms (e.g., image recognition models) to identify past events, and the analysis results are stored in a database.

[0230] Input: Image file

[0231] Output: Analysis results (past events)

[0232] Step 5:

[0233] The device's integrated facial recognition function is used to detect and identify faces from the user's current camera image. Based on the detection results, the server sends relevant information to the device.

[0234] Input: Real-time camera footage

[0235] Output: Detected person information

[0236] Step 6:

[0237] The device displays personal information and past events sent from the server to the user through the smart glasses' display or voice notification.

[0238] Input: Related information (personal information, past events)

[0239] Output: Visual and audio notification to the user

[0240] Step 7:

[0241] The server generates reminders based on schedules and routines and sends them to the device, where the reminder content is displayed to the user as text or audio data.

[0242] Input: Schedule data, routine data

[0243] Output: Reminder text data, audio data

[0244] Step 8:

[0245] Users receive reminders and notifications from their devices and act on them, such as remembering to take their medication or participate in a specific activity.

[0246] Input: Reminder text data, voice data

[0247] Output: User action execution

[0248] In this way, the system can provide comprehensive support for dementia care for the elderly and improve the quality of their daily lives.

[0249] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0250] The present invention relates to an integrated memory support system for supporting the memory of elderly people with dementia, and provides more effective support by combining an emotion engine that recognizes the user's emotions.

[0251] Overall system structure

[0252] This system is mainly composed of three components: a server, a terminal, and a user, as well as an emotion engine. The server processes and stores all data, while the terminal provides an interface for the user, who interacts with the system through the terminal. The emotion engine recognizes the user's emotional state and provides feedback to the system.

[0253] Server Processing

[0254] 1. Data collection and analysis

[0255] The server receives data provided by the user through the terminal, including voice, text, and photos.

[0256] The server analyzes the received data using AI algorithms to identify important memories and events from the user's past.

[0257] The user's emotional data recognized by the emotion engine is also analyzed at the same time.

[0258] 2. Emotional Recognition

[0259] The server receives emotion data from the emotion engine and grasps the user's current emotional state. The emotion engine recognizes the user's emotions in real time from changes in voice and facial expressions.

[0260] 3. Providing memory support

[0261] The server generates content to stimulate memories at appropriate times according to the user's situation and emotional state. For example, if the server recognizes that the user is feeling anxious, it selects content related to past happy memories.

[0262] The content and timing of reminders can also be adjusted based on emotional data, for example, if a user is feeling irritated, a reminder will be sent suggesting actions to calm them down.

[0263] Terminal handling

[0264] 1. Data input

[0265] The device accepts the user's voice and text input and prepares it for transmission to the server. The device temporarily stores the input data and manages transmission to the server.

[0266] 2. Emotion Detection

[0267] The device uses an emotion engine to detect emotions from the user's voice and facial expressions in real time and transmits the data to the server.

[0268] 3. Reminders and notifications

[0269] The device will display reminders and notifications from the server to the user at the appropriate time, such as reminders to help users remember their daily routines.

[0270] 4. Conversation support

[0271] The device can provide users with information if they cannot remember specific information during a conversation, such as the name of a specific person or supplementary information about a past event.

[0272] User operations

[0273] 1. Entering historical data

[0274] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[0275] 2. Recording everyday events

[0276] Users input their daily events into the terminal, and this input data is sent to the server and recorded.

[0277] 3. Check your reminders

[0278] Users can check reminders from their devices and carry out necessary events and activities in a timely manner.

[0279] 4. Emotional Feedback

[0280] When a user interacts with the device, the emotion engine detects the user's emotional state from their facial expressions and tone of voice and sends that information to the server, allowing the system to grasp the user's emotional state in real time.

[0281] Specific examples

[0282] 1. Emotional memory stimulation

[0283] When a user uploads photos of past trips to their device, the server analyzes the photos and identifies them as travel destinations from, say, 2010. If the emotion engine recognizes the user's emotion of joy, the server highlights and presents happy memories related to that trip.

[0284] 2. Recording new events and connecting emotions

[0285] When a user types "I went for a walk with a friend today" into the device, that data is sent to the server. The emotion engine records the happy emotion sensed from the user's voice and uses the emotion data when reminding the user of the event later.

[0286] 3. Providing Emotion-Based Reminders

[0287] If the emotion engine on the server recognizes that the user is frustrated, it generates a reminder to relax, such as "take a deep breath," and sends it to the device. The device then notifies the user.

[0288] As described above, by combining the system of the present invention with an emotion engine, it is possible to more effectively support the memory of elderly people with dementia and improve the quality of their daily lives.

[0289] The processing flow will be explained below.

[0290] Step 1:

[0291] The user inputs past photos, voice memos, and text data into the device. The user selects the data using the device's input interface and clicks the upload button.

[0292] Step 2:

[0293] The terminal temporarily stores the data received from the user and prepares it for transmission to the server. The terminal checks the data format and performs error checks.

[0294] Step 3:

[0295] The device sends the prepared data to the server. The server receives the data and stores it in a database. The server checks the integrity of the received data and checks for incomplete data.

[0296] Step 4:

[0297] The server uses AI algorithms to analyze the data it receives and identify key events and memories from the user's past. The server then organizes the information and prepares it for the next processing step.

[0298] Step 5:

[0299] The server analyzes the emotion data provided by the emotion engine, which detects the user's emotion in real time from voice tone and facial expressions, and sends the data to the server.

[0300] Step 6:

[0301] The server generates content to stimulate memories based on the analysis results and emotional data. If the user is in an anxious state, it selects and provides content related to past happy memories.

[0302] Step 7:

[0303] The device displays the generated content to the user, who can then review past photos and voice memos to stimulate their memories. For example, a user might view photos from a trip in 2010 and recall fond memories from that time.

[0304] Step 8:

[0305] The user inputs new events and daily events into the device, inputting the activities they performed that day and their feelings in text, and then clicks the send button.

[0306] Step 9:

[0307] The device receives new data, temporarily stores it, and then sends it to the server. The server receives the new data and records it in the database. This updates the user's daily life history.

[0308] Step 10:

[0309] The user sets the time to take daily medicine or schedule regular health checkups on the device. The user specifies the content and time of the reminder and clicks the setting button.

[0310] Step 11:

[0311] The server generates reminders based on the user's settings, and adjusts the content and timing of the reminders as needed, taking into account emotion data from the emotion engine.

[0312] Step 12:

[0313] The device will display a reminder notification to the user at the specified time. For example, it could display a notification saying "It's time to take your medicine," prompting the user to take action.

[0314] Step 13:

[0315] If a user cannot remember certain information during a conversation, they can ask for help from the device by entering a question using the device's question interface and clicking the send button.

[0316] Step 14:

[0317] The server receives the user's query and searches the database for the relevant information. The server responds quickly and identifies the information the user is looking for.

[0318] Step 15:

[0319] The server sends the identified information to the terminal, which displays it to the user. The user confirms the displayed information and continues the conversation.

[0320] Step 16:

[0321] The emotion engine runs even while the user is interacting with the system, continuously monitoring the user's emotional changes. Emotional data is fed back to the server in real time, and appropriate action is taken as necessary.

[0322] Through these steps, the system of the present invention can effectively support the memory of elderly people with dementia and improve the quality of their daily lives. By combining it with an emotion engine, it can provide more flexible and user-friendly support.

[0323] Example 2

[0324] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0325] To support the memory of elderly people with dementia, simply providing reminders and notifications is not enough. Systems that take the user's emotional state into account and appropriately stimulate past memories to improve the quality of daily life are needed. In particular, it is important to have a means to provide information when the user cannot remember specific information during a conversation, and a means to record the user's daily events and provide reminders based on that information.

[0326] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0327] In this invention, the server includes means for receiving and storing input data from a user, means for normalizing and analyzing the received data to identify past memories, means for receiving and analyzing emotional data to recognize the user's emotional state, means for generating content to stimulate memory based on the analysis and emotional analysis results, means for notifying the user of the generated content and reminders at an appropriate time, and means for providing information if the user cannot remember specific information during a conversation. This makes it possible to provide appropriate reminders and information based on the user's emotional state, thereby more effectively supporting memory in elderly people with dementia.

[0328] "Input data" refers to information such as voice, text, and photos provided by the user through the terminal.

[0329] "Normalization" refers to the process of converting received data into a format that is easier to analyze.

[0330] "Analysis" refers to the process of using AI algorithms to detect patterns and features in normalized data and identify past memories and events.

[0331] "Emotional data" refers to information about a user's emotional state obtained from their tone of voice and facial expressions.

[0332] "Emotion engine" refers to software or algorithms that analyze a user's tone of voice and facial expressions in real time to recognize their emotional state.

[0333] "Content" refers to information and media files (e.g., photos, videos, text) generated to stimulate a user's memory.

[0334] A "reminder" refers to a notification that reminds a user of a particular action or event.

[0335] "Notification" refers to a message from the system to inform the user of a reminder.

[0336] "Means" refers to a method or apparatus for accomplishing a particular function.

[0337] "Server" refers to a central device for data processing, analysis, and storage.

[0338] "Terminal" refers to the device (e.g., smartphone, tablet, or PC) that a user uses to interact with the system.

[0339] "Past memories" refer to important events and memories that the user has experienced in the past.

[0340] "Memory stimulation" refers to actions or content provided to evoke a user's past memories.

[0341] "Right time" refers to providing notifications and reminders at the most appropriate time based on the user's situation and emotional state.

[0342] This invention is an integrated memory support system for supporting the memory of elderly people with dementia. This system is composed of three main components: a server, a terminal, and a user, as well as an emotion engine. The functions of each component and the specific processing contents of the program are described in detail below.

[0343] Server Processing

[0344] The server is a central device that processes, analyzes, and stores various data. The server performs the following main processes:

[0345] 1. Data collection and analysis

[0346] The server receives data such as voice, text, and photos provided by the user through the device, and uses an AI algorithm such as TensorFlow.

[0347] The received data is normalized, and if it is audio data, it is converted into text data using a speech recognition engine such as Google Speech-to-Text.

[0348] The normalized data is analyzed to identify significant memories and events from the user's past.

[0349] 2. Emotional Recognition

[0350] The server receives emotional data from the emotion engine, which analyzes the user's voice tone and facial expressions, and determines their current emotional state. The emotion engine uses Microsoft Azure's emotion recognition API, among other things.

[0351] 3. Providing memory support

[0352] The server generates content to stimulate memories based on the analysis results and the user's emotional state. For example, if the user is anxious, the server generates content related to past happy memories.

[0353] Adjust the content and timing of reminders based on emotional data.

[0354] Terminal handling

[0355] A terminal is a device that functions as an interface between a user and a system. The main processes of a terminal are as follows:

[0356] 1. Data input

[0357] The device accepts voice and text input from the user. When the user records a diary entry, the device temporarily stores the data and prepares it for transmission to the server.

[0358] 2. Emotion Detection

[0359] The emotion engine detects emotions in real time from the user's voice and facial expressions, using IBM Watson's emotion analysis API.

[0360] 3. Reminders and notifications

[0361] Displaying reminders and notifications from the server to the user at appropriate times, for example, a reminder to "Don't forget to take your morning walk."

[0362] 4. Conversation support

[0363] If the user cannot remember a particular piece of information during a conversation, the system will provide that information to the user. For example, if the user asks, "Who is this person?", the system will search past memories and display the relevant information.

[0364] User operations

[0365] The user can perform the following operations through the terminal.

[0366] 1. Entering historical data

[0367] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[0368] 2. Recording everyday events

[0369] Users input daily events into the device using voice or text, and the input data is sent to the server and recorded.

[0370] 3. Check your reminders

[0371] The user checks the reminder from the device and performs the necessary tasks or events, for example, checking and performing the reminder to "take a deep breath."

[0372] 4. Emotional Feedback

[0373] When a user interacts with the device, the emotion engine detects the user's emotional state from facial expressions and tone of voice and sends this information to the server.

[0374] Specific examples

[0375] 1. Emotional memory stimulation

[0376] When a user uploads photos from past trips to their device, the photos are sent to the server, which analyzes the photos to identify the travel destination and time period. If the emotion engine detects the user's joy, the server highlights and displays happy memories related to that trip.

[0377] Prompt: "Analyze a travel photo. Identify when and where the photo was taken and provide the user with relevant memories."

[0378] 2. Recording new events and connecting emotions

[0379] When a user types "I went for a walk with a friend today" into the device, that data is sent to the server. If the emotion engine senses the user's joy, it also records that emotion data and uses it to remind the user later.

[0380] Prompt: "Analyze the user's diary and record the events and emotions of the day."

[0381] 3. Providing Emotion-Based Reminders

[0382] When the emotion engine detects that the user is frustrated, it sends this data to the server, which then generates a reminder to "take a deep breath" and sends it to the device, which then notifies the user of the reminder.

[0383] Prompt: "If the user is frustrated, generate a reminder and notify them."

[0384] By utilizing data from the emotion engine, this system will more effectively support the memory of elderly people with dementia and contribute to improving the quality of their daily lives.

[0385] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0386] Server Processing

[0387] Step 1: Receiving Data

[0388] The server receives data such as the user's voice, text, and photos from the device. For example, when a user records a voice diary entry, the voice file is transferred from the device to the server.

[0389] Input: Voice, text, and photo data sent from your device.

[0390] Output: Saving received data to storage.

[0391] Step 2: Normalize and preprocess the data

[0392] The server normalizes the received data and uses a speech recognition engine (e.g., Google Speech-to-Text) to convert the audio data into text data.

[0393] Input: Received data (voice, text, photos).

[0394] Output: Normalized data, speech data converted to text.

[0395] Step 3: Analyze the data

[0396] The server analyzes the normalized data using AI algorithms (e.g., TensorFlow) to identify important memories and events from the user's past.

[0397] Input: Normalized data.

[0398] Output: Identified memory or event information.

[0399] Step 4: Receiving emotion data

[0400] The server receives the user's emotional data (voice tone and facial expression analysis results) from the emotion engine.

[0401] Input: Emotion data sent from the emotion engine.

[0402] Output: Save emotion data to storage.

[0403] Step 5: Analyze the sentiment data

[0404] The server analyzes the received emotion data to understand the user's current emotional state, for example, determining whether the user is happy or sad from the tone of their voice.

[0405] Input: Emotion data.

[0406] Output: Parsed emotional state information.

[0407] Step 6: Generate memory support content

[0408] The server generates content to stimulate memories based on the analysis results and the user's emotional state. For example, if the user is anxious, the server generates content related to past happy memories.

[0409] Input: Identified memory or event information, analyzed emotional state information.

[0410] Output: The generated content.

[0411] Step 7: Adjust reminders and notifications

[0412] The server adjusts the content and timing of reminders based on emotional data. For example, if a user is feeling stressed, it generates a reminder to take a deep breath to relax.

[0413] Input: Parsed emotional state information.

[0414] Output: Coordinated reminders and notifications.

[0415] Step 8: Sending Data

[0416] The server sends the generated content and adjusted reminders to the device.

[0417] Input: Generated content, tailored reminders.

[0418] Output: Data sent to the terminal.

[0419] Terminal handling

[0420] Step 1: Data Input

[0421] The device accepts the user's voice and text input and prepares it to be sent to the server. For example, when a user records a diary entry, the device temporarily stores the voice data.

[0422] Input: User voice and text input.

[0423] Output: Temporarily saved data.

[0424] Step 2: Sending data

[0425] The terminal transmits the temporarily stored data to the server.

[0426] Input: Temporarily saved data.

[0427] Output: Data sent to the server.

[0428] Step 3: Emotion detection

[0429] The device uses an emotion engine to detect emotions from the user's voice and facial expressions in real time, for example, by analyzing voice and identifying emotions.

[0430] Input: Real-time user voice and facial expression data.

[0431] Output: Parsed emotion data.

[0432] Step 4: Sending emotion data

[0433] The terminal transmits the analysis results to the server.

[0434] Input: Parsed emotion data.

[0435] Output: Data sent to the server.

[0436] Step 5: Receive reminders

[0437] The device receives reminders and notifications from the server, such as a reminder to "don't forget to take a morning walk."

[0438] Input: Reminder and notification data from the server.

[0439] Output: Received data to the terminal.

[0440] Step 6: View Reminders

[0441] The device will display reminders and notifications to the user at appropriate times, using pop-up notifications and sound notifications.

[0442] Input: Received reminder, notification data.

[0443] Output: What is displayed to the user.

[0444] Step 7: Conversation support

[0445] If the user cannot remember a particular piece of information during a conversation, the device will provide that information to the user. For example, if the user asks, "Who is this person?", the device will search past memories and display relevant information.

[0446] Input: User question.

[0447] Output: Display of relevant information.

[0448] User operations

[0449] Step 1: Enter historical data

[0450] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[0451] Input: Past photos, voice memos.

[0452] Output: Upload data to the device.

[0453] Step 2: Record your daily events

[0454] Users input daily events into the device using voice or text, and the input data is sent to the server and stored as a record.

[0455] Input: Audio and text data of the user's daily life.

[0456] Output: Record data to terminal.

[0457] Step 3: Check your reminders

[0458] The user can check the reminders received from the device and perform the necessary tasks or events, for example, checking and performing the reminder to "take a deep breath."

[0459] Enter: reminders, notifications.

[0460] Output: The user's action.

[0461] Step 4: Emotional Feedback

[0462] When a user interacts with the device, the emotion engine detects the user's emotional state from facial expressions and tone of voice and sends this information to the server.

[0463] Input: User's facial expression, tone of voice.

[0464] Output: Data sent to the server.

[0465] (Application example 2)

[0466] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0467] The problem that this invention aims to solve is to prevent elderly people with dementia from experiencing difficulties in their daily lives due to memory confusion and forgetting important information, thereby improving their quality of life. Furthermore, by providing appropriate support according to the elderly's emotional state, it aims to reduce stress and anxiety, allowing them to live their daily lives with greater peace of mind.

[0468] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0469] In this invention, the server includes means for receiving input data from a user, means for analyzing the received data and identifying past memories, means for generating content for stimulating memory based on the analysis results, means for providing reminders and notifications to the user, and means for recognizing the user's emotions in real time using an emotion engine and reflecting that data in the analysis, thereby providing appropriate and timely support based on the user's emotional state, enabling memory improvement and stress reduction.

[0470] "Means for receiving input data from a user" is a general term for hardware and software that allows a user to input data such as voice, text, or images into the system and receive it.

[0471] "Means for analyzing received data and identifying past memories" refers to the algorithms and processes by which the system analyzes received data and identifies the user's past events and important memories.

[0472] The "means for generating content to stimulate memory based on the analysis results" is a mechanism for automatically generating content such as text, images, and audio to stimulate the user's memory based on identified past memories.

[0473] "Means for providing user reminders and notifications" are mechanisms for informing users of system-generated content and important information in a timely manner.

[0474] "Means of recognizing user emotions in real time using an emotion engine and reflecting that data in analysis" refers to technology that detects the user's emotional state in real time from their voice and facial expressions, and incorporates that information into the system's analysis process.

[0475] MODE FOR CARRYING OUT THE INVENTION

[0476] The present invention provides a smart shopping support system for supporting the shopping experience of elderly people with dementia, allowing them to live their lives with greater peace of mind. The following are embodiments of the invention.

[0477] Overall system structure

[0478] The system mainly consists of three components: a server, a terminal, and a user. The server is responsible for the main data processing and storage, and the terminal provides an interface for the user, through which the user interacts with the system.

[0479] Server Processing

[0480] The server does the following:

[0481] 1. Data collection and analysis

[0482] The server receives and analyzes data such as voice, text, and images provided by the device, and uses a generative AI model to identify the user's past memories and important events.

[0483] Emotion engine data will also be analyzed at the same time.

[0484] 2. Emotional Recognition

[0485] The server uses an emotion engine to recognize the user's emotions in real time and incorporates that data into its analysis, identifying emotions based on changes in voice and facial expressions.

[0486] 3. Providing memory support

[0487] The server generates content to stimulate memories according to the user's situation and emotional state. For example, if the user is feeling anxious, it selects content related to past happy memories.

[0488] 4. Reminders and notifications

[0489] The server generates reminders and notifications for users at appropriate times and sends them to their devices, helping them to remember their daily routines and important events.

[0490] Terminal handling

[0491] The terminal does the following:

[0492] 1. Data input

[0493] The device accepts the user's voice and text input and prepares it for transmission to the server. The input data is temporarily stored and managed for transmission to the server.

[0494] 2. Emotion Detection

[0495] The device uses an emotion engine to detect emotions from the user's voice and facial expressions in real time and transmits the data to the server.

[0496] 3. Reminders and notifications

[0497] The device will display reminders and notifications from the server to the user at the appropriate time, such as reminders to help users remember their daily routines.

[0498] 4. Conversation support

[0499] The device can provide users with information if they cannot remember specific information during a conversation, such as the name of a specific person or supplementary information about a past event.

[0500] User operations

[0501] The user does the following:

[0502] 1. Create a shopping list

[0503] The user adds an item to the shopping list, for example, by entering a voice command such as "add milk to the shopping list" into the terminal, which is then sent to the server.

[0504] 2. Recording everyday events

[0505] Users input daily events into the device by voice or text, for example, "Today I went for a walk with a friend."

[0506] 3. Emotional Feedback

[0507] When a user interacts with the device, the emotion engine detects the user's emotional state from their facial expressions and tone of voice, and sends that information to the server.

[0508] Specific examples

[0509] For example:

[0510] 1. Create a shopping list

[0511] When the user types "Add milk to the shopping list," data is sent from the device to the server, and milk is added to the shopping list.

[0512] 2. Emotional awareness and reminders

[0513] When a user types "I'm feeling happy today," the emotion engine recognizes the emotion as "happy" and sends it to the server, which then generates a reminder such as "Stay happy and enjoy shopping," which is then sent to the device.

[0514] Prompt Sentence Examples

[0515] Prompts to identify the user's emotional state:

[0516] Identify the emotional state based on user input. Input text: "Today was a very happy day!"

[0517] If the keyword "happy" is included, the emotional state will be set to "happy".

[0518] Input: "Today was a very happy day!", Identified emotion: "happy"

[0519] Prompt to record past events:

[0520] The user enters a past event. Event: "I went on a fun picnic with my family."

[0521] Emotion: Record the associated emotion "happy."

[0522] Input: "I went on a fun picnic with my family", emotion: "happy"

[0523] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0524] Step 1:

[0525] A user creates a shopping list. The user inputs a voice command into the terminal, such as "Please add milk to the shopping list." The terminal converts this voice command into text data and sends it to the server. The server receives this data and adds "milk" to the shopping list.

[0526] Input: User speaks "Add milk to my shopping list"

[0527] Data processing: Converting voice data into text data

[0528] Output: "Milk" is added to the shopping list

[0529] Step 2:

[0530] The user records a past event. The user types, "Today I went for a walk with a friend" into the device. The device converts this voice data into text data and sends it to the server. The server identifies the received text data and adds its contents to a list of past events.

[0531] Input: User's voice input: "I went for a walk with a friend today"

[0532] Data processing: Converting voice data into text data

[0533] Output: "I went for a walk with a friend today" is added to the list of past events

[0534] Step 3:

[0535] The device detects the user's emotions. The user speaks, "I'm feeling happy today." The device converts the voice data into text data and analyzes it with the emotion engine. The emotion engine recognizes "happy" from the text and sends the information to the server.

[0536] Input: User's voice input: "I'm feeling happy today"

[0537] Data processing: Converts voice data into text data and analyzes emotions with an emotion engine

[0538] Output: Emotional state "happy" is sent to the server

[0539] Step 4:

[0540] The server generates appropriate reminders based on the user's emotional state. For example, if the server receives the emotion data "happy," it generates a reminder such as "Stay happy and enjoy shopping." Using this generative AI model, the server inputs prompts to create reminders.

[0541] Input: Emotional state "happy"

[0542] Data processing: Generative AI models are used to generate reminders

[0543] Output: Reminder: "Stay happy and enjoy your shopping"

[0544] Step 5:

[0545] The device notifies the user of the reminder. The device receives the reminder "Keep your shopping fun and enjoy yourself" sent from the server and displays it as a notification on the user's device.

[0546] Input: Reminder sent from the server

[0547] Data processing: Convert reminders into notifications

[0548] Output: The reminder appears on the user's device

[0549] Through these steps, the system supports the shopping experience of elderly people with dementia, providing users with appropriate feedback based on their emotional state, allowing them to carry out their daily shopping with greater peace of mind.

[0550] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0551] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0552] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0553] [Second embodiment]

[0554] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0555] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0556] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0557] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0558] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0559] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0560] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0561] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0562] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0563] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0564] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0565] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0566] The present invention relates to an integrated memory support system for supporting the memory of elderly people with dementia, specifically, a system that receives and analyzes data from users, generates memory-stimulating content, and provides timely reminders and notifications.

[0567] Overall system structure

[0568] The system mainly consists of three components: a server, a terminal, and a user. The server processes and stores all data, and the terminal provides an interface for the user, through which the user interacts with the system.

[0569] Server Processing

[0570] 1. Data collection and analysis

[0571] The server receives data provided by the user through the terminal, including voice, text, and photos.

[0572] The server then analyzes the received data using AI algorithms, which identify important memories and events from the user's past.

[0573] 2. Recording New Events

[0574] The server stores the data in real time as users enter their daily events, recording their daily routines and new events.

[0575] 3. Providing memory support

[0576] The server generates appropriate content based on the user's current situation, which stimulates the user's memory based on past photos and voice memos.

[0577] For example, a user who is in a particular location may be sent content that reminds them of a past event related to that location.

[0578] Terminal handling

[0579] 1. Data input

[0580] The device accepts the user's voice and text input and prepares it for transmission to the server. The device temporarily stores the input data and manages transmission to the server.

[0581] 2. Reminders and notifications

[0582] The device will display reminders and notifications from the server to the user at the appropriate time, such as reminders to help users remember their daily routines.

[0583] 3. Conversation support

[0584] The device can provide users with information if they cannot remember specific information during a conversation, such as the name of a specific person or supplementary information about a past event.

[0585] User operations

[0586] 1. Entering historical data

[0587] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[0588] 2. Recording everyday events

[0589] Users input their daily events into the terminal, and this input data is sent to the server and recorded.

[0590] 3. Check your reminders

[0591] Users can check reminders from their devices and carry out necessary events and activities in a timely manner.

[0592] Specific examples

[0593] 1. Helping people recall past memories

[0594] A user uploads photos of past trips to the device, and the server analyzes the photos and identifies them as trips taken in 2010. If the user then asks a question related to the travel destination, the device will notify them, "This is a trip taken in 2010."

[0595] 2. Recording New Events

[0596] When a user types "I went for a walk with a friend today" into the device, that data is sent to the server and recorded. Later, when the user asks "When was the day I went for a walk with a friend?", the device provides that information.

[0597] 3. Providing reminders

[0598] The system records the user's daily medication intake times, and when the time approaches, the server sends a reminder to the device, which then notifies the user, "It's time to take your medicine."

[0599] As described above, the system of the present invention provides an effective means of supporting the memory of elderly people with dementia and improving the quality of their daily lives.

[0600] The processing flow will be explained below.

[0601] Step 1:

[0602] Users input past photos, voice memos, and text data into the device, select data related to past events and memories, and click the upload button.

[0603] Step 2:

[0604] The device temporarily stores the data received from the user and prepares it for transmission to the server. The device checks the data format and performs error checking to ensure smooth transmission.

[0605] Step 3:

[0606] The device sends the prepared data to the server. The server receives the data and stores it in a database. The server checks the integrity of the data and checks for incomplete data.

[0607] Step 4:

[0608] The server then runs the data through an AI algorithm to analyze it, identifying key events and memories from the user's past and extracting relevant information.

[0609] Step 5:

[0610] The server uses the analysis results to generate content to stimulate the user's memory, such as photos of past travel destinations or text related to events.

[0611] Step 6:

[0612] The user inputs new events and daily events into the device, inputting the activities they performed that day and their feelings in text, and then clicks the send button.

[0613] Step 7:

[0614] The device receives new data, temporarily stores it, and then sends it to the server. The server receives the new data and records it in the database. This updates the user's daily life history.

[0615] Step 8:

[0616] To utilize the system's reminder function, users set specific events or routines on their devices, such as setting a daily medication intake time or scheduling a regular health checkup.

[0617] Step 9:

[0618] The server sets a timer based on the user's settings, generates a reminder at the appropriate time, and sends the reminder content to the device, causing a notification to be displayed.

[0619] Step 10:

[0620] The device displays a reminder notification to the user at the specified time. The user checks the notification on the device and takes the necessary action, such as taking medicine.

[0621] Step 11:

[0622] If a user cannot remember a particular piece of information during a conversation, they can ask for help from the terminal by typing a question into the terminal and sending it.

[0623] Step 12:

[0624] The server receives the user's query and searches the database for the relevant information. The server responds quickly and identifies the information the user is looking for.

[0625] Step 13:

[0626] The server sends the identified information to the terminal, which displays it to the user. The user confirms the displayed information and continues the conversation.

[0627] Through the above steps, the system of the present invention can effectively support the memory of elderly people with dementia and improve the quality of their daily lives.

[0628] Example 1

[0629] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0630] The present invention aims to provide a system that supports elderly people with dementia in recalling past memories in their daily lives and in carrying out daily events and routines without forgetting them. In particular, there is a need for an effective means for users to easily recall their past memories and appropriately manage their daily actions and events, but existing technologies have not been able to completely solve this problem.

[0631] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0632] In this invention, the server includes a means for receiving input data from a user, a means for analyzing the received data using multiple analysis techniques, and a means for recording the user's behavioral patterns and routines in real time based on the analysis results, thereby enabling appropriate support for the user's memory and improving the quality of daily life.

[0633] "User input data" refers to information such as voice, text, and photos provided by the user through the terminal.

[0634] "Analyzing received data" means that the server analyzes the data received from the user using natural language processing, image recognition, etc.

[0635] "Identifying past memories" refers to revealing important events or memories from the user's past based on the data analyzed by the server.

[0636] "Generating content to stimulate memories" means that the server creates content such as photos, text, and audio that are generated for the purpose of stimulating the user's memories based on past memories.

[0637] "Providing reminders and notifications to the user" means that the server generates reminders and notifications, sends them to the user at the appropriate time, and the terminal displays them.

[0638] "Transmitting to a server via a network" means that the terminal transmits the user's input data to a server using a communication means such as the Internet.

[0639] "Analysis using multiple analysis technologies" means that the server analyzes data by combining different technologies such as natural language processing (NLP), image recognition, and voice analysis.

[0640] "Recording user behavior patterns and routines in real time" means that the server immediately saves and manages the user's daily behavior and routines based on the analysis results.

[0641] The present invention relates to an integrated memory support system for supporting the memory of elderly people with dementia. The system receives input data from a user, analyzes the data using multiple analytical techniques, identifies the user's past memories, generates content to stimulate memory based on the analysis results, and provides reminders and notifications at appropriate times.

[0642] Overall system structure

[0643] The system mainly consists of three components: a server, a terminal, and a user. The server processes and stores all data, and the terminal provides an interface for the user, through which the user interacts with the system.

[0644] Server Processing

[0645] The server processes the input data using multiple analysis technologies, including natural language processing (NLP), image recognition, and voice analysis. For example, it can analyze photo data using Google Cloud Vision API or Amazon Rekognition, and analyze text data using Google Natural Language API.

[0646] Data collection and analysis

[0647] The server receives data provided by the user through the device. This data includes voice, text, photos, etc. For example, if a user uploads photos of their trip, the server receives the photo data and analyzes it using image recognition technology. This analysis can identify important memories and events from the user's past.

[0648] Recording new events

[0649] The server stores the data in real time as the user inputs their daily events. For example, if a user inputs "I went for a walk with a friend today," the server analyzes the data and stores it in a MySQL database or similar.

[0650] Providing memory support

[0651] Based on the analysis results, the server generates content to stimulate the user's memory. This content can include past photos and voice memos, and is sent to the user via email or push notification. For example, when a user is in a specific location, they will receive a notification about past events related to that location.

[0652] Terminal handling

[0653] The device accepts voice and text input from the user, temporarily stores it in local storage, and then transmits the input data to a server over the network, for example, using Google Firebase or a REST API.

[0654] Data Input

[0655] The device receives the user's voice and text input and prepares it to be sent to the server. For example, if the user says, "Set the time to take my medicine," the device sends that data to the server.

[0656] Reminders and notifications

[0657] The device will display reminders and notifications from the server to the user at appropriate times, for example, a pop-up notification saying "It's time to take your medicine" at a specific time.

[0658] User operations

[0659] Users can use the device to upload past photos and voice memos. This data is automatically sent to the server, where it is analyzed and saved. Users can also enter daily events into the device, and the data is sent to the server and recorded in real time.

[0660] Specific examples

[0661] 1. Helping people recall past memories

[0662] A user uploads photos of a trip from 2010 to their device. The server analyzes the photos and identifies them as trips from 2010. Then, when the user asks about the trip, the device will say, "This is a trip from 2010."

[0663] Prompt Sentence Examples

[0664] Here are some photos from a trip we took in 2010. Please provide a suitable description when users ask about the destination.

[0665] 2. Recording New Events

[0666] A user enters "I went for a walk with a friend today" into a device. This information is sent to a server and recorded. Later, when the user asks "When was the day I went for a walk with a friend?", the device provides that information.

[0667] Prompt Sentence Examples

[0668] Record today's date and the information that you "went for a walk with a friend." When you ask for this information later, please let us know when you went for a walk.

[0669] 3. Providing reminders

[0670] The user sets the time to take their medicine daily, and when that time approaches, the server sends a reminder to the device, which then notifies them that it is time to take their medicine.

[0671] Prompt Sentence Examples

[0672] Set a time for the user to take their medicine, and when that time approaches, display a reminder saying "It's time to take your medicine."

[0673] As described above, the system of the present invention provides an effective means for supporting the memory of elderly people with dementia and improving their quality of daily life.

[0674] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0675] Step 1: User enters data into the terminal

[0676] A user opens an application on their device and enters "I went for a walk with a friend today" into a text box. The input data (text or voice) is temporarily stored in the device's local storage. The user provides the text "I went for a walk with a friend today" as input data, and the data is temporarily stored in the device.

[0677] Step 2: The device sends the data to the server

[0678] The device sends the text data entered by the user to the server via the network. The device uses an HTTPS request to send the entered text data, "I went for a walk with a friend today," in JSON format to the server. Specifically, the device calls a REST API to send the data to a specific endpoint on the server.

[0679] Step 3: The server receives and analyzes the data

[0680] The server stores the received text data in Google Cloud Storage or Amazon S3. The server uses a Python script to analyze the text data using the Google Natural Language API. The server receives the text data "I went for a walk with a friend today" as input for analysis, analyzes the text, and extracts event information containing the keywords "friend" and "walk."

[0681] Step 4: The server generates notifications and reminders based on the analysis results.

[0682] Based on the analysis results, the server records the user's behavioral patterns and saves them as new events. For example, the event information "went for a walk with a friend" is saved in a MySQL database. The server generates a reminder for the next event, allowing the user to look back on that day at a later date. The server receives the event information from the analysis results as input, generates reminder information as output, and saves it in the database.

[0683] Step 5: The server sends notifications and reminders to the device

[0684] The server generates reminder information and sends it to the device via the network. A push notification is sent using a dedicated notification API (such as Firebase Cloud Messaging). The server receives reminder information as input and generates and sends a notification message to the device as output.

[0685] Step 6: The device displays notifications and reminders to the user

[0686] The device displays notification messages received from the server to the user as popups or audio alerts. For example, it displays a reminder such as "It's time to take your medicine" on the screen to provide an alert to the user. It receives notification messages received from the server as input and displays visual and audio notifications to the user as output.

[0687] The above are the specific operations at each processing step of this system, which effectively provides memory support for elderly people with dementia.

[0688] (Application example 1)

[0689] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0690] Elderly people with dementia often experience difficulty in daily life due to memory loss and confusion. Furthermore, inability to recall past events or people can lead to problems with communication. This can lead to a decline in quality of life and increased psychological stress. Therefore, a system that can solve these problems and support the memory of elderly people with dementia is needed.

[0691] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0692] In this invention, the server includes means for receiving input data from a user, means for analyzing the received data and identifying past memories, means for displaying information about specific people using a face recognition function, means for recording daily events using a voice recognition function, means for generating content to stimulate memories based on the analysis results, means for displaying past events based on the user's location and time, and means for providing reminders and notifications to the user. This makes it easier for elderly people with dementia to remember daily events and past events and people, thereby improving their quality of life.

[0693] The "means for receiving input data from a user" is a device that has the function of receiving data such as voice, text, and photographs provided by a user via a terminal.

[0694] "Means for analyzing received data and identifying past memories" refers to a mechanism for analyzing received data using an AI algorithm to identify important memories and events from the user's past.

[0695] "Means for displaying information about a specific person using a facial recognition function" refers to technology that uses a camera to detect the face of a specific person and displays information related to that person.

[0696] The "means for recording daily events using a voice recognition function" is a system that uses a microphone to recognize a user's speech, converts it into text data, and records daily events.

[0697] "Means for generating content to stimulate memory based on analysis results" is a function that automatically creates content such as photos and voice memos to evoke the user's memories based on the analysis results.

[0698] The "means for displaying past events based on the user's location and time" is a system that presents past events related to a location or time point according to the user's current location or a specific time.

[0699] "Means for providing reminders and notifications to users" refers to a mechanism that has the function of sending visual and audio notifications to users at appropriate times according to pre-set schedules and routines.

[0700] "Means for providing visual and audio reminders via smart devices" refers to technology for providing both visual and audio reminders using devices such as smartphones or smart glasses.

[0701] "Means for providing specific information during a conversation using a voice output function" refers to a technology that uses a speaker to audibly convey information that a user needs during a conversation.

[0702] This invention is an integrated memory support system for supporting the memory of elderly people with dementia, and is realized by combining three main elements: a server, a terminal, and a user.

[0703] Overall system structure

[0704] This system consists of the following hardware and software:

[0705] Hardware: Smart glasses (with camera, microphone, and speaker), server (data processing and storage), smartphone

[0706] Software: Python, OpenCV, face_recognition, Vosk (voice recognition)

[0707] Server Processing

[0708] 1. Data Receipt and Analysis:

[0709] The server receives data such as voice, text, and photos provided by the user via the device, which is then analyzed using AI algorithms to identify important memories and events from the user's past.

[0710] 2. New Events Recorded:

[0711] The server uses a voice recognition function to record the user's daily activities in real time, analyzes the voice data, and saves it as text data.

[0712] 3. Providing memory support:

[0713] Based on the analysis results, the server generates content (such as photos and voice memos) to stimulate the user's memory, and also displays past events based on the user's location or a specific time.

[0714] Terminal handling

[0715] 1. Data input and transmission:

[0716] The device (smart glasses or smartphone) receives the user's voice and text input and sends it to the server. The input data is temporarily stored on the device, and transmission to the server is managed.

[0717] 2. Facial Recognition and Information Display:

[0718] Using the device's camera, it recognizes the face of a specific person and displays information related to that person, helping users remember the name of the person in front of them and past events.

[0719] 3. Reminders and notifications:

[0720] The device will display reminders and notifications sent from the server to the user visually and audibly at the appropriate time, for example, a reminder to take medicine at a specific time.

[0721] User operations

[0722] 1. Enter historical data:

[0723] Users can upload past photos and voice memos to the device, and this data is sent to a server and stored for analysis.

[0724] 2. Recording daily events:

[0725] Users can record daily events by voice input into the device, and this data is sent to the server and stored in a database.

[0726] 3. Check your reminders:

[0727] Users can check reminder notifications from their devices and perform necessary events and activities in a timely manner.

[0728] Specific examples

[0729] 1. Helping people recall past memories:

[0730] When a user wears the smart glasses and walks around town, photos and events from previous visits to the same places are automatically displayed, evoke memories of those times by showing photos of past trips.

[0731] 2. New Events Recorded:

[0732] When a user says, "I went for a walk with a friend today," the data is sent to the server and recorded. Later, when the user asks, "When was the day I went for a walk with a friend?", the device provides that information.

[0733] 3. Providing Reminders:

[0734] At a specific time, the smart glasses' voice function will remind the user, saying, "It's time to take your medicine," to help them remember.

[0735] Prompt Sentence Examples

[0736] Project: Memory Spot

[0737] Objective: To develop a system that uses smart glasses to support memory in elderly people with dementia.

[0738] procedure:

[0739] Uses face recognition library (face_recognition) and speech recognition library (Vosk).

[0740] Faces are recognized in the camera frame and the recognition results are notified via voice.

[0741] Record daily events using voice input from a microphone.

[0742] Regular voice reminders.

[0743] In this way, the system of the present invention is designed to enable elderly people with dementia to live their daily lives more efficiently and safely.

[0744] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0745] Program processing flow

[0746] Step 1:

[0747] The user inputs data into the device, such as voice input or photo upload. The device then uses the microphone to capture the voice and the camera to take a photo. The input data is temporarily stored on the device.

[0748] Input: User's voice and photo data

[0749] Output: Temporarily saved audio and image files

[0750] Step 2:

[0751] The device then sends the temporarily stored data to the server using a communication protocol via the Internet.

[0752] Input: Temporarily saved audio and image files

[0753] Output: Audio and image files sent to the server

[0754] Step 3:

[0755] The server converts the received voice data into text using the Vosk speech recognition library, and the converted text data is stored in a database.

[0756] Input: Audio file

[0757] Output: Text data

[0758] Step 4:

[0759] The server analyzes the received photo data using AI algorithms (e.g., image recognition models) to identify past events, and the analysis results are stored in a database.

[0760] Input: Image file

[0761] Output: Analysis results (past events)

[0762] Step 5:

[0763] The device's integrated facial recognition function is used to detect and identify faces from the user's current camera image. Based on the detection results, the server sends relevant information to the device.

[0764] Input: Real-time camera footage

[0765] Output: Detected person information

[0766] Step 6:

[0767] The device displays personal information and past events sent from the server to the user through the smart glasses' display or voice notification.

[0768] Input: Related information (personal information, past events)

[0769] Output: Visual and audio notification to the user

[0770] Step 7:

[0771] The server generates reminders based on schedules and routines and sends them to the device, where the reminder content is displayed to the user as text or audio data.

[0772] Input: Schedule data, routine data

[0773] Output: Reminder text data, audio data

[0774] Step 8:

[0775] Users receive reminders and notifications from their devices and act on them, such as remembering to take their medication or participate in a specific activity.

[0776] Input: Reminder text data, voice data

[0777] Output: User action execution

[0778] In this way, the system can provide comprehensive support for dementia care for the elderly and improve the quality of their daily lives.

[0779] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0780] The present invention relates to an integrated memory support system for supporting the memory of elderly people with dementia, and provides more effective support by combining an emotion engine that recognizes the user's emotions.

[0781] Overall system structure

[0782] This system is mainly composed of three components: a server, a terminal, and a user, as well as an emotion engine. The server processes and stores all data, while the terminal provides an interface for the user, who interacts with the system through the terminal. The emotion engine recognizes the user's emotional state and provides feedback to the system.

[0783] Server Processing

[0784] 1. Data collection and analysis

[0785] The server receives data provided by the user through the terminal, including voice, text, and photos.

[0786] The server analyzes the received data using AI algorithms to identify important memories and events from the user's past.

[0787] The user's emotional data recognized by the emotion engine is also analyzed at the same time.

[0788] 2. Emotional Recognition

[0789] The server receives emotion data from the emotion engine and grasps the user's current emotional state. The emotion engine recognizes the user's emotions in real time from changes in voice and facial expressions.

[0790] 3. Providing memory support

[0791] The server generates content to stimulate memories at appropriate times according to the user's situation and emotional state. For example, if the server recognizes that the user is feeling anxious, it selects content related to past happy memories.

[0792] The content and timing of reminders can also be adjusted based on emotional data, for example, if a user is feeling irritated, a reminder will be sent suggesting actions to calm them down.

[0793] Terminal handling

[0794] 1. Data input

[0795] The device accepts the user's voice and text input and prepares it for transmission to the server. The device temporarily stores the input data and manages transmission to the server.

[0796] 2. Emotion Detection

[0797] The device uses an emotion engine to detect emotions from the user's voice and facial expressions in real time and transmits the data to the server.

[0798] 3. Reminders and notifications

[0799] The device will display reminders and notifications from the server to the user at the appropriate time, such as reminders to help users remember their daily routines.

[0800] 4. Conversation support

[0801] The device can provide users with information if they cannot remember specific information during a conversation, such as the name of a specific person or supplementary information about a past event.

[0802] User operations

[0803] 1. Entering historical data

[0804] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[0805] 2. Recording everyday events

[0806] Users input their daily events into the terminal, and this input data is sent to the server and recorded.

[0807] 3. Check your reminders

[0808] Users can check reminders from their devices and carry out necessary events and activities in a timely manner.

[0809] 4. Emotional Feedback

[0810] When a user interacts with the device, the emotion engine detects the user's emotional state from their facial expressions and tone of voice and sends that information to the server, allowing the system to grasp the user's emotional state in real time.

[0811] Specific examples

[0812] 1. Emotional memory stimulation

[0813] When a user uploads photos of past trips to their device, the server analyzes the photos and identifies them as travel destinations from, say, 2010. If the emotion engine recognizes the user's emotion of joy, the server highlights and presents happy memories related to that trip.

[0814] 2. Recording new events and connecting emotions

[0815] When a user types "I went for a walk with a friend today" into the device, that data is sent to the server. The emotion engine records the happy emotion sensed from the user's voice and uses the emotion data when reminding the user of the event later.

[0816] 3. Providing Emotion-Based Reminders

[0817] If the emotion engine on the server recognizes that the user is frustrated, it generates a reminder to relax, such as "take a deep breath," and sends it to the device. The device then notifies the user.

[0818] As described above, by combining the system of the present invention with an emotion engine, it is possible to more effectively support the memory of elderly people with dementia and improve the quality of their daily lives.

[0819] The processing flow will be explained below.

[0820] Step 1:

[0821] The user inputs past photos, voice memos, and text data into the device. The user selects the data using the device's input interface and clicks the upload button.

[0822] Step 2:

[0823] The terminal temporarily stores the data received from the user and prepares it for transmission to the server. The terminal checks the data format and performs error checks.

[0824] Step 3:

[0825] The device sends the prepared data to the server. The server receives the data and stores it in a database. The server checks the integrity of the received data and checks for incomplete data.

[0826] Step 4:

[0827] The server uses AI algorithms to analyze the data it receives and identify key events and memories from the user's past. The server then organizes the information and prepares it for the next processing step.

[0828] Step 5:

[0829] The server analyzes the emotion data provided by the emotion engine, which detects the user's emotion in real time from voice tone and facial expressions, and sends the data to the server.

[0830] Step 6:

[0831] The server generates content to stimulate memories based on the analysis results and emotional data. If the user is in an anxious state, it selects and provides content related to past happy memories.

[0832] Step 7:

[0833] The device displays the generated content to the user, who can then review past photos and voice memos to stimulate their memories. For example, a user might view photos from a trip in 2010 and recall fond memories from that time.

[0834] Step 8:

[0835] The user inputs new events and daily events into the device, inputting the activities they performed that day and their feelings in text, and then clicks the send button.

[0836] Step 9:

[0837] The device receives new data, temporarily stores it, and then sends it to the server. The server receives the new data and records it in the database. This updates the user's daily life history.

[0838] Step 10:

[0839] The user sets the time to take daily medicine or schedule regular health checkups on the device. The user specifies the content and time of the reminder and clicks the setting button.

[0840] Step 11:

[0841] The server generates reminders based on the user's settings, and adjusts the content and timing of the reminders as needed, taking into account emotion data from the emotion engine.

[0842] Step 12:

[0843] The device will display a reminder notification to the user at the specified time. For example, it could display a notification saying "It's time to take your medicine," prompting the user to take action.

[0844] Step 13:

[0845] If a user cannot remember certain information during a conversation, they can ask for help from the device by entering a question using the device's question interface and clicking the send button.

[0846] Step 14:

[0847] The server receives the user's query and searches the database for the relevant information. The server responds quickly and identifies the information the user is looking for.

[0848] Step 15:

[0849] The server sends the identified information to the terminal, which displays it to the user. The user confirms the displayed information and continues the conversation.

[0850] Step 16:

[0851] The emotion engine runs even while the user is interacting with the system, continuously monitoring the user's emotional changes. Emotional data is fed back to the server in real time, and appropriate action is taken as necessary.

[0852] Through these steps, the system of the present invention can effectively support the memory of elderly people with dementia and improve the quality of their daily lives. By combining it with an emotion engine, it can provide more flexible and user-friendly support.

[0853] Example 2

[0854] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0855] To support the memory of elderly people with dementia, simply providing reminders and notifications is not enough. Systems that take the user's emotional state into account and appropriately stimulate past memories to improve the quality of daily life are needed. In particular, it is important to have a means to provide information when the user cannot remember specific information during a conversation, and a means to record the user's daily events and provide reminders based on that information.

[0856] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0857] In this invention, the server includes means for receiving and storing input data from a user, means for normalizing and analyzing the received data to identify past memories, means for receiving and analyzing emotional data to recognize the user's emotional state, means for generating content to stimulate memory based on the analysis and emotional analysis results, means for notifying the user of the generated content and reminders at an appropriate time, and means for providing information if the user cannot remember specific information during a conversation. This makes it possible to provide appropriate reminders and information based on the user's emotional state, thereby more effectively supporting memory in elderly people with dementia.

[0858] "Input data" refers to information such as voice, text, and photos provided by the user through the terminal.

[0859] "Normalization" refers to the process of converting received data into a format that is easier to analyze.

[0860] "Analysis" refers to the process of using AI algorithms to detect patterns and features in normalized data and identify past memories and events.

[0861] "Emotional data" refers to information about a user's emotional state obtained from their tone of voice and facial expressions.

[0862] "Emotion engine" refers to software or algorithms that analyze a user's tone of voice and facial expressions in real time to recognize their emotional state.

[0863] "Content" refers to information and media files (e.g., photos, videos, text) generated to stimulate a user's memory.

[0864] A "reminder" refers to a notification that reminds a user of a particular action or event.

[0865] "Notification" refers to a message from the system to inform the user of a reminder.

[0866] "Means" refers to a method or apparatus for accomplishing a particular function.

[0867] "Server" refers to a central device for data processing, analysis, and storage.

[0868] "Terminal" refers to the device (e.g., smartphone, tablet, or PC) that a user uses to interact with the system.

[0869] "Past memories" refer to important events and memories that the user has experienced in the past.

[0870] "Memory stimulation" refers to actions or content provided to evoke a user's past memories.

[0871] "Right time" refers to providing notifications and reminders at the most appropriate time based on the user's situation and emotional state.

[0872] This invention is an integrated memory support system for supporting the memory of elderly people with dementia. This system is composed of three main components: a server, a terminal, and a user, as well as an emotion engine. The functions of each component and the specific processing contents of the program are described in detail below.

[0873] Server Processing

[0874] The server is a central device that processes, analyzes, and stores various data. The server performs the following main processes:

[0875] 1. Data collection and analysis

[0876] The server receives data such as voice, text, and photos provided by the user through the device, and uses an AI algorithm such as TensorFlow.

[0877] The received data is normalized, and if it is audio data, it is converted into text data using a speech recognition engine such as Google Speech-to-Text.

[0878] The normalized data is analyzed to identify significant memories and events from the user's past.

[0879] 2. Emotional Recognition

[0880] The server receives emotional data from the emotion engine, which analyzes the user's voice tone and facial expressions, and determines their current emotional state. The emotion engine uses Microsoft Azure's emotion recognition API, among other things.

[0881] 3. Providing memory support

[0882] The server generates content to stimulate memories based on the analysis results and the user's emotional state. For example, if the user is anxious, the server generates content related to past happy memories.

[0883] Adjust the content and timing of reminders based on emotional data.

[0884] Terminal handling

[0885] A terminal is a device that functions as an interface between a user and a system. The main processes of a terminal are as follows:

[0886] 1. Data input

[0887] The device accepts voice and text input from the user. When the user records a diary entry, the device temporarily stores the data and prepares it for transmission to the server.

[0888] 2. Emotion Detection

[0889] The emotion engine detects emotions in real time from the user's voice and facial expressions, using IBM Watson's emotion analysis API.

[0890] 3. Reminders and notifications

[0891] Displaying reminders and notifications from the server to the user at appropriate times, for example, a reminder to "Don't forget to take your morning walk."

[0892] 4. Conversation support

[0893] If the user cannot remember a particular piece of information during a conversation, the system will provide that information to the user. For example, if the user asks, "Who is this person?", the system will search past memories and display the relevant information.

[0894] User operations

[0895] The user can perform the following operations through the terminal.

[0896] 1. Entering historical data

[0897] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[0898] 2. Recording everyday events

[0899] Users input daily events into the device using voice or text, and the input data is sent to the server and recorded.

[0900] 3. Check your reminders

[0901] The user checks the reminder from the device and performs the necessary tasks or events, for example, checking and performing the reminder to "take a deep breath."

[0902] 4. Emotional Feedback

[0903] When a user interacts with the device, the emotion engine detects the user's emotional state from facial expressions and tone of voice and sends this information to the server.

[0904] Specific examples

[0905] 1. Emotional memory stimulation

[0906] When a user uploads photos from past trips to their device, the photos are sent to the server, which analyzes the photos to identify the travel destination and time period. If the emotion engine detects the user's joy, the server highlights and displays happy memories related to that trip.

[0907] Prompt: "Analyze a travel photo. Identify when and where the photo was taken and provide the user with relevant memories."

[0908] 2. Recording new events and connecting emotions

[0909] When a user types "I went for a walk with a friend today" into the device, that data is sent to the server. If the emotion engine senses the user's joy, it also records that emotion data and uses it to remind the user later.

[0910] Prompt: "Analyze the user's diary and record the events and emotions of the day."

[0911] 3. Providing Emotion-Based Reminders

[0912] When the emotion engine detects that the user is frustrated, it sends this data to the server, which then generates a reminder to "take a deep breath" and sends it to the device, which then notifies the user of the reminder.

[0913] Prompt: "If the user is frustrated, generate a reminder and notify them."

[0914] By utilizing data from the emotion engine, this system will more effectively support the memory of elderly people with dementia and contribute to improving the quality of their daily lives.

[0915] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0916] Server Processing

[0917] Step 1: Receiving Data

[0918] The server receives data such as the user's voice, text, and photos from the device. For example, when a user records a voice diary entry, the voice file is transferred from the device to the server.

[0919] Input: Voice, text, and photo data sent from your device.

[0920] Output: Saving received data to storage.

[0921] Step 2: Normalize and preprocess the data

[0922] The server normalizes the received data and uses a speech recognition engine (e.g., Google Speech-to-Text) to convert the audio data into text data.

[0923] Input: Received data (voice, text, photos).

[0924] Output: Normalized data, speech data converted to text.

[0925] Step 3: Analyze the data

[0926] The server analyzes the normalized data using AI algorithms (e.g., TensorFlow) to identify important memories and events from the user's past.

[0927] Input: Normalized data.

[0928] Output: Identified memory or event information.

[0929] Step 4: Receiving emotion data

[0930] The server receives the user's emotional data (voice tone and facial expression analysis results) from the emotion engine.

[0931] Input: Emotion data sent from the emotion engine.

[0932] Output: Save emotion data to storage.

[0933] Step 5: Analyze the sentiment data

[0934] The server analyzes the received emotion data to understand the user's current emotional state, for example, determining whether the user is happy or sad from the tone of their voice.

[0935] Input: Emotion data.

[0936] Output: Parsed emotional state information.

[0937] Step 6: Generate memory support content

[0938] The server generates content to stimulate memories based on the analysis results and the user's emotional state. For example, if the user is anxious, the server generates content related to past happy memories.

[0939] Input: Identified memory or event information, analyzed emotional state information.

[0940] Output: The generated content.

[0941] Step 7: Adjust reminders and notifications

[0942] The server adjusts the content and timing of reminders based on emotional data. For example, if a user is feeling stressed, it generates a reminder to take a deep breath to relax.

[0943] Input: Parsed emotional state information.

[0944] Output: Coordinated reminders and notifications.

[0945] Step 8: Sending Data

[0946] The server sends the generated content and adjusted reminders to the device.

[0947] Input: Generated content, tailored reminders.

[0948] Output: Data sent to the terminal.

[0949] Terminal handling

[0950] Step 1: Data Input

[0951] The device accepts the user's voice and text input and prepares it to be sent to the server. For example, when a user records a diary entry, the device temporarily stores the voice data.

[0952] Input: User voice and text input.

[0953] Output: Temporarily saved data.

[0954] Step 2: Sending data

[0955] The terminal transmits the temporarily stored data to the server.

[0956] Input: Temporarily saved data.

[0957] Output: Data sent to the server.

[0958] Step 3: Emotion detection

[0959] The device uses an emotion engine to detect emotions from the user's voice and facial expressions in real time, for example, by analyzing voice and identifying emotions.

[0960] Input: Real-time user voice and facial expression data.

[0961] Output: Parsed emotion data.

[0962] Step 4: Sending emotion data

[0963] The terminal transmits the analysis results to the server.

[0964] Input: Parsed emotion data.

[0965] Output: Data sent to the server.

[0966] Step 5: Receive reminders

[0967] The device receives reminders and notifications from the server, such as a reminder to "don't forget to take a morning walk."

[0968] Input: Reminder and notification data from the server.

[0969] Output: Received data to the terminal.

[0970] Step 6: View Reminders

[0971] The device will display reminders and notifications to the user at appropriate times, using pop-up notifications and sound notifications.

[0972] Input: Received reminder, notification data.

[0973] Output: What is displayed to the user.

[0974] Step 7: Conversation support

[0975] If the user cannot remember a particular piece of information during a conversation, the device will provide that information to the user. For example, if the user asks, "Who is this person?", the device will search past memories and display relevant information.

[0976] Input: User question.

[0977] Output: Display of relevant information.

[0978] User operations

[0979] Step 1: Enter historical data

[0980] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[0981] Input: Past photos, voice memos.

[0982] Output: Upload data to the device.

[0983] Step 2: Record your daily events

[0984] Users input daily events into the device using voice or text, and the input data is sent to the server and stored as a record.

[0985] Input: Audio and text data of the user's daily life.

[0986] Output: Record data to terminal.

[0987] Step 3: Check your reminders

[0988] The user can check the reminders received from the device and perform the necessary tasks or events, for example, checking and performing the reminder to "take a deep breath."

[0989] Enter: reminders, notifications.

[0990] Output: The user's action.

[0991] Step 4: Emotional Feedback

[0992] When a user interacts with the device, the emotion engine detects the user's emotional state from facial expressions and tone of voice and sends this information to the server.

[0993] Input: User's facial expression, tone of voice.

[0994] Output: Data sent to the server.

[0995] (Application example 2)

[0996] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0997] The problem that this invention aims to solve is to prevent elderly people with dementia from experiencing difficulties in their daily lives due to memory confusion and forgetting important information, thereby improving their quality of life. Furthermore, by providing appropriate support according to the elderly's emotional state, it aims to reduce stress and anxiety, allowing them to live their daily lives with greater peace of mind.

[0998] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0999] In this invention, the server includes means for receiving input data from a user, means for analyzing the received data and identifying past memories, means for generating content for stimulating memory based on the analysis results, means for providing reminders and notifications to the user, and means for recognizing the user's emotions in real time using an emotion engine and reflecting that data in the analysis, thereby providing appropriate and timely support based on the user's emotional state, enabling memory improvement and stress reduction.

[1000] "Means for receiving input data from a user" is a general term for hardware and software that allows a user to input data such as voice, text, or images into the system and receive it.

[1001] "Means for analyzing received data and identifying past memories" refers to the algorithms and processes by which the system analyzes received data and identifies the user's past events and important memories.

[1002] The "means for generating content to stimulate memory based on the analysis results" is a mechanism for automatically generating content such as text, images, and audio to stimulate the user's memory based on identified past memories.

[1003] "Means for providing user reminders and notifications" are mechanisms for informing users of system-generated content and important information in a timely manner.

[1004] "Means of recognizing user emotions in real time using an emotion engine and reflecting that data in analysis" refers to technology that detects the user's emotional state in real time from their voice and facial expressions, and incorporates that information into the system's analysis process.

[1005] MODE FOR CARRYING OUT THE INVENTION

[1006] The present invention provides a smart shopping support system for supporting the shopping experience of elderly people with dementia, allowing them to live their lives with greater peace of mind. The following are embodiments of the invention.

[1007] Overall system structure

[1008] The system mainly consists of three components: a server, a terminal, and a user. The server is responsible for the main data processing and storage, and the terminal provides an interface for the user, through which the user interacts with the system.

[1009] Server Processing

[1010] The server does the following:

[1011] 1. Data collection and analysis

[1012] The server receives and analyzes data such as voice, text, and images provided by the device, and uses a generative AI model to identify the user's past memories and important events.

[1013] Emotion engine data will also be analyzed at the same time.

[1014] 2. Emotional Recognition

[1015] The server uses an emotion engine to recognize the user's emotions in real time and incorporates that data into its analysis, identifying emotions based on changes in voice and facial expressions.

[1016] 3. Providing memory support

[1017] The server generates content to stimulate memories according to the user's situation and emotional state. For example, if the user is feeling anxious, it selects content related to past happy memories.

[1018] 4. Reminders and notifications

[1019] The server generates reminders and notifications for users at appropriate times and sends them to their devices, helping them to remember their daily routines and important events.

[1020] Terminal handling

[1021] The terminal does the following:

[1022] 1. Data input

[1023] The device accepts the user's voice and text input and prepares it for transmission to the server. The input data is temporarily stored and managed for transmission to the server.

[1024] 2. Emotion Detection

[1025] The device uses an emotion engine to detect emotions from the user's voice and facial expressions in real time and transmits the data to the server.

[1026] 3. Reminders and notifications

[1027] The device will display reminders and notifications from the server to the user at the appropriate time, such as reminders to help users remember their daily routines.

[1028] 4. Conversation support

[1029] The device can provide users with information if they cannot remember specific information during a conversation, such as the name of a specific person or supplementary information about a past event.

[1030] User operations

[1031] The user does the following:

[1032] 1. Create a shopping list

[1033] The user adds an item to the shopping list, for example, by entering a voice command such as "add milk to the shopping list" into the terminal, which is then sent to the server.

[1034] 2. Recording everyday events

[1035] Users input daily events into the device by voice or text, for example, "Today I went for a walk with a friend."

[1036] 3. Emotional Feedback

[1037] When a user interacts with the device, the emotion engine detects the user's emotional state from their facial expressions and tone of voice, and sends that information to the server.

[1038] Specific examples

[1039] For example:

[1040] 1. Create a shopping list

[1041] When the user types "Add milk to the shopping list," data is sent from the device to the server, and milk is added to the shopping list.

[1042] 2. Emotional awareness and reminders

[1043] When a user types "I'm feeling happy today," the emotion engine recognizes the emotion as "happy" and sends it to the server, which then generates a reminder such as "Stay happy and enjoy shopping," which is then sent to the device.

[1044] Prompt Sentence Examples

[1045] Prompts to identify the user's emotional state:

[1046] Identify the emotional state based on user input. Input text: "Today was a very happy day!"

[1047] If the keyword "happy" is included, the emotional state will be set to "happy".

[1048] Input: "Today was a very happy day!", Identified emotion: "happy"

[1049] Prompt to record past events:

[1050] The user enters a past event. Event: "I went on a fun picnic with my family."

[1051] Emotion: Record the associated emotion "happy."

[1052] Input: "I went on a fun picnic with my family", emotion: "happy"

[1053] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1054] Step 1:

[1055] A user creates a shopping list. The user inputs a voice command into the terminal, such as "Please add milk to the shopping list." The terminal converts this voice command into text data and sends it to the server. The server receives this data and adds "milk" to the shopping list.

[1056] Input: User speaks "Add milk to my shopping list"

[1057] Data processing: Converting voice data into text data

[1058] Output: "Milk" is added to the shopping list

[1059] Step 2:

[1060] The user records a past event. The user types, "Today I went for a walk with a friend" into the device. The device converts this voice data into text data and sends it to the server. The server identifies the received text data and adds its contents to a list of past events.

[1061] Input: User's voice input: "I went for a walk with a friend today"

[1062] Data processing: Converting voice data into text data

[1063] Output: "I went for a walk with a friend today" is added to the list of past events

[1064] Step 3:

[1065] The device detects the user's emotions. The user speaks, "I'm feeling happy today." The device converts the voice data into text data and analyzes it with the emotion engine. The emotion engine recognizes "happy" from the text and sends the information to the server.

[1066] Input: User's voice input: "I'm feeling happy today"

[1067] Data processing: Converts voice data into text data and analyzes emotions with an emotion engine

[1068] Output: Emotional state "happy" is sent to the server

[1069] Step 4:

[1070] The server generates appropriate reminders based on the user's emotional state. For example, if the server receives the emotion data "happy," it generates a reminder such as "Stay happy and enjoy shopping." Using this generative AI model, the server inputs prompts to create reminders.

[1071] Input: Emotional state "happy"

[1072] Data processing: Generative AI models are used to generate reminders

[1073] Output: Reminder: "Stay happy and enjoy your shopping"

[1074] Step 5:

[1075] The device notifies the user of the reminder. The device receives the reminder "Keep your shopping fun and enjoy yourself" sent from the server and displays it as a notification on the user's device.

[1076] Input: Reminder sent from the server

[1077] Data processing: Convert reminders into notifications

[1078] Output: The reminder appears on the user's device

[1079] Through these steps, the system supports the shopping experience of elderly people with dementia, providing users with appropriate feedback based on their emotional state, allowing them to carry out their daily shopping with greater peace of mind.

[1080] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1081] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1082] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1083] [Third embodiment]

[1084] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1085] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1086] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1087] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1088] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1089] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1090] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1091] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1092] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1093] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1094] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1095] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1096] The present invention relates to an integrated memory support system for supporting the memory of elderly people with dementia, specifically, a system that receives and analyzes data from users, generates memory-stimulating content, and provides timely reminders and notifications.

[1097] Overall system structure

[1098] The system mainly consists of three components: a server, a terminal, and a user. The server processes and stores all data, and the terminal provides an interface for the user, through which the user interacts with the system.

[1099] Server Processing

[1100] 1. Data collection and analysis

[1101] The server receives data provided by the user through the terminal, including voice, text, and photos.

[1102] The server then analyzes the received data using AI algorithms, which identify important memories and events from the user's past.

[1103] 2. Recording New Events

[1104] The server stores the data in real time as users enter their daily events, recording their daily routines and new events.

[1105] 3. Providing memory support

[1106] The server generates appropriate content based on the user's current situation, which stimulates the user's memory based on past photos and voice memos.

[1107] For example, a user who is in a particular location may be sent content that reminds them of a past event related to that location.

[1108] Terminal handling

[1109] 1. Data input

[1110] The device accepts the user's voice and text input and prepares it for transmission to the server. The device temporarily stores the input data and manages transmission to the server.

[1111] 2. Reminders and notifications

[1112] The device will display reminders and notifications from the server to the user at the appropriate time, such as reminders to help users remember their daily routines.

[1113] 3. Conversation support

[1114] The device can provide users with information if they cannot remember specific information during a conversation, such as the name of a specific person or supplementary information about a past event.

[1115] User operations

[1116] 1. Entering historical data

[1117] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[1118] 2. Recording everyday events

[1119] Users input their daily events into the terminal, and this input data is sent to the server and recorded.

[1120] 3. Check your reminders

[1121] Users can check reminders from their devices and carry out necessary events and activities in a timely manner.

[1122] Specific examples

[1123] 1. Helping people recall past memories

[1124] A user uploads photos of past trips to the device, and the server analyzes the photos and identifies them as trips taken in 2010. If the user then asks a question related to the travel destination, the device will notify them, "This is a trip taken in 2010."

[1125] 2. Recording New Events

[1126] When a user types "I went for a walk with a friend today" into the device, that data is sent to the server and recorded. Later, when the user asks "When was the day I went for a walk with a friend?", the device provides that information.

[1127] 3. Providing reminders

[1128] The system records the user's daily medication intake times, and when the time approaches, the server sends a reminder to the device, which then notifies the user, "It's time to take your medicine."

[1129] As described above, the system of the present invention provides an effective means of supporting the memory of elderly people with dementia and improving the quality of their daily lives.

[1130] The processing flow will be explained below.

[1131] Step 1:

[1132] Users input past photos, voice memos, and text data into the device, select data related to past events and memories, and click the upload button.

[1133] Step 2:

[1134] The device temporarily stores the data received from the user and prepares it for transmission to the server. The device checks the data format and performs error checking to ensure smooth transmission.

[1135] Step 3:

[1136] The device sends the prepared data to the server. The server receives the data and stores it in a database. The server checks the integrity of the data and checks for incomplete data.

[1137] Step 4:

[1138] The server then runs the data through an AI algorithm to analyze it, identifying key events and memories from the user's past and extracting relevant information.

[1139] Step 5:

[1140] The server uses the analysis results to generate content to stimulate the user's memory, such as photos of past travel destinations or text related to events.

[1141] Step 6:

[1142] The user inputs new events and daily events into the device, inputting the activities they performed that day and their feelings in text, and then clicks the send button.

[1143] Step 7:

[1144] The device receives new data, temporarily stores it, and then sends it to the server. The server receives the new data and records it in the database. This updates the user's daily life history.

[1145] Step 8:

[1146] To utilize the system's reminder function, users set specific events or routines on their devices, such as setting a daily medication intake time or scheduling a regular health checkup.

[1147] Step 9:

[1148] The server sets a timer based on the user's settings, generates a reminder at the appropriate time, and sends the reminder content to the device, causing a notification to be displayed.

[1149] Step 10:

[1150] The device displays a reminder notification to the user at the specified time. The user checks the notification on the device and takes the necessary action, such as taking medicine.

[1151] Step 11:

[1152] If a user cannot remember a particular piece of information during a conversation, they can ask for help from the terminal by typing a question into the terminal and sending it.

[1153] Step 12:

[1154] The server receives the user's query and searches the database for the relevant information. The server responds quickly and identifies the information the user is looking for.

[1155] Step 13:

[1156] The server sends the identified information to the terminal, which displays it to the user. The user confirms the displayed information and continues the conversation.

[1157] Through the above steps, the system of the present invention can effectively support the memory of elderly people with dementia and improve the quality of their daily lives.

[1158] Example 1

[1159] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1160] The present invention aims to provide a system that supports elderly people with dementia in recalling past memories in their daily lives and in carrying out daily events and routines without forgetting them. In particular, there is a need for an effective means for users to easily recall their past memories and appropriately manage their daily actions and events, but existing technologies have not been able to completely solve this problem.

[1161] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1162] In this invention, the server includes a means for receiving input data from a user, a means for analyzing the received data using multiple analysis techniques, and a means for recording the user's behavioral patterns and routines in real time based on the analysis results, thereby enabling appropriate support for the user's memory and improving the quality of daily life.

[1163] "User input data" refers to information such as voice, text, and photos provided by the user through the terminal.

[1164] "Analyzing received data" means that the server analyzes the data received from the user using natural language processing, image recognition, etc.

[1165] "Identifying past memories" refers to revealing important events or memories from the user's past based on the data analyzed by the server.

[1166] "Generating content to stimulate memories" means that the server creates content such as photos, text, and audio that are generated for the purpose of stimulating the user's memories based on past memories.

[1167] "Providing reminders and notifications to the user" means that the server generates reminders and notifications, sends them to the user at the appropriate time, and the terminal displays them.

[1168] "Transmitting to a server via a network" means that the terminal transmits the user's input data to a server using a communication means such as the Internet.

[1169] "Analysis using multiple analysis technologies" means that the server analyzes data by combining different technologies such as natural language processing (NLP), image recognition, and voice analysis.

[1170] "Recording user behavior patterns and routines in real time" means that the server immediately saves and manages the user's daily behavior and routines based on the analysis results.

[1171] The present invention relates to an integrated memory support system for supporting the memory of elderly people with dementia. The system receives input data from a user, analyzes the data using multiple analytical techniques, identifies the user's past memories, generates content to stimulate memory based on the analysis results, and provides reminders and notifications at appropriate times.

[1172] Overall system structure

[1173] The system mainly consists of three components: a server, a terminal, and a user. The server processes and stores all data, and the terminal provides an interface for the user, through which the user interacts with the system.

[1174] Server Processing

[1175] The server processes the input data using multiple analysis technologies, including natural language processing (NLP), image recognition, and voice analysis. For example, it can analyze photo data using Google Cloud Vision API or Amazon Rekognition, and analyze text data using Google Natural Language API.

[1176] Data collection and analysis

[1177] The server receives data provided by the user through the device. This data includes voice, text, photos, etc. For example, if a user uploads photos of their trip, the server receives the photo data and analyzes it using image recognition technology. This analysis can identify important memories and events from the user's past.

[1178] Recording new events

[1179] The server stores the data in real time as the user inputs their daily events. For example, if a user inputs "I went for a walk with a friend today," the server analyzes the data and stores it in a MySQL database or similar.

[1180] Providing memory support

[1181] Based on the analysis results, the server generates content to stimulate the user's memory. This content can include past photos and voice memos, and is sent to the user via email or push notification. For example, when a user is in a specific location, they will receive a notification about past events related to that location.

[1182] Terminal handling

[1183] The device accepts voice and text input from the user, temporarily stores it in local storage, and then transmits the input data to a server over the network, for example, using Google Firebase or a REST API.

[1184] Data Input

[1185] The device receives the user's voice and text input and prepares it to be sent to the server. For example, if the user says, "Set the time to take my medicine," the device sends that data to the server.

[1186] Reminders and notifications

[1187] The device will display reminders and notifications from the server to the user at appropriate times, for example, a pop-up notification saying "It's time to take your medicine" at a specific time.

[1188] User operations

[1189] Users can use the device to upload past photos and voice memos. This data is automatically sent to the server, where it is analyzed and saved. Users can also enter daily events into the device, and the data is sent to the server and recorded in real time.

[1190] Specific examples

[1191] 1. Helping people recall past memories

[1192] A user uploads photos of a trip from 2010 to their device. The server analyzes the photos and identifies them as trips from 2010. Then, when the user asks about the trip, the device will say, "This is a trip from 2010."

[1193] Prompt Sentence Examples

[1194] Here are some photos from a trip we took in 2010. Please provide a suitable description when users ask about the destination.

[1195] 2. Recording New Events

[1196] A user enters "I went for a walk with a friend today" into a device. This information is sent to a server and recorded. Later, when the user asks "When was the day I went for a walk with a friend?", the device provides that information.

[1197] Prompt Sentence Examples

[1198] Record today's date and the information that you "went for a walk with a friend." When you ask for this information later, please let us know when you went for a walk.

[1199] 3. Providing reminders

[1200] The user sets the time to take their medicine daily, and when that time approaches, the server sends a reminder to the device, which then notifies them that it is time to take their medicine.

[1201] Prompt Sentence Examples

[1202] Set a time for the user to take their medicine, and when that time approaches, display a reminder saying "It's time to take your medicine."

[1203] As described above, the system of the present invention provides an effective means for supporting the memory of elderly people with dementia and improving their quality of daily life.

[1204] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1205] Step 1: User enters data into the terminal

[1206] A user opens an application on their device and enters "I went for a walk with a friend today" into a text box. The input data (text or voice) is temporarily stored in the device's local storage. The user provides the text "I went for a walk with a friend today" as input data, and the data is temporarily stored in the device.

[1207] Step 2: The device sends the data to the server

[1208] The device sends the text data entered by the user to the server via the network. The device uses an HTTPS request to send the entered text data, "I went for a walk with a friend today," in JSON format to the server. Specifically, the device calls a REST API to send the data to a specific endpoint on the server.

[1209] Step 3: The server receives and analyzes the data

[1210] The server stores the received text data in Google Cloud Storage or Amazon S3. The server uses a Python script to analyze the text data using the Google Natural Language API. The server receives the text data "I went for a walk with a friend today" as input for analysis, analyzes the text, and extracts event information containing the keywords "friend" and "walk."

[1211] Step 4: The server generates notifications and reminders based on the analysis results.

[1212] Based on the analysis results, the server records the user's behavioral patterns and saves them as new events. For example, the event information "went for a walk with a friend" is saved in a MySQL database. The server generates a reminder for the next event, allowing the user to look back on that day at a later date. The server receives the event information from the analysis results as input, generates reminder information as output, and saves it in the database.

[1213] Step 5: The server sends notifications and reminders to the device

[1214] The server generates reminder information and sends it to the device via the network. A push notification is sent using a dedicated notification API (such as Firebase Cloud Messaging). The server receives reminder information as input and generates and sends a notification message to the device as output.

[1215] Step 6: The device displays notifications and reminders to the user

[1216] The device displays notification messages received from the server to the user as popups or audio alerts. For example, it displays a reminder such as "It's time to take your medicine" on the screen to provide an alert to the user. It receives notification messages received from the server as input and displays visual and audio notifications to the user as output.

[1217] The above are the specific operations at each processing step of this system, which effectively provides memory support for elderly people with dementia.

[1218] (Application example 1)

[1219] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1220] Elderly people with dementia often experience difficulty in daily life due to memory loss and confusion. Furthermore, inability to recall past events or people can lead to problems with communication. This can lead to a decline in quality of life and increased psychological stress. Therefore, a system that can solve these problems and support the memory of elderly people with dementia is needed.

[1221] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1222] In this invention, the server includes means for receiving input data from a user, means for analyzing the received data and identifying past memories, means for displaying information about specific people using a face recognition function, means for recording daily events using a voice recognition function, means for generating content to stimulate memories based on the analysis results, means for displaying past events based on the user's location and time, and means for providing reminders and notifications to the user. This makes it easier for elderly people with dementia to remember daily events and past events and people, thereby improving their quality of life.

[1223] The "means for receiving input data from a user" is a device that has the function of receiving data such as voice, text, and photographs provided by a user via a terminal.

[1224] "Means for analyzing received data and identifying past memories" refers to a mechanism for analyzing received data using an AI algorithm to identify important memories and events from the user's past.

[1225] "Means for displaying information about a specific person using a facial recognition function" refers to technology that uses a camera to detect the face of a specific person and displays information related to that person.

[1226] The "means for recording daily events using a voice recognition function" is a system that uses a microphone to recognize a user's speech, converts it into text data, and records daily events.

[1227] "Means for generating content to stimulate memory based on analysis results" is a function that automatically creates content such as photos and voice memos to evoke the user's memories based on the analysis results.

[1228] The "means for displaying past events based on the user's location and time" is a system that presents past events related to a location or time point according to the user's current location or a specific time.

[1229] "Means for providing reminders and notifications to users" refers to a mechanism that has the function of sending visual and audio notifications to users at appropriate times according to pre-set schedules and routines.

[1230] "Means for providing visual and audio reminders via smart devices" refers to technology for providing both visual and audio reminders using devices such as smartphones or smart glasses.

[1231] "Means for providing specific information during a conversation using a voice output function" refers to a technology that uses a speaker to audibly convey information that a user needs during a conversation.

[1232] This invention is an integrated memory support system for supporting the memory of elderly people with dementia, and is realized by combining three main elements: a server, a terminal, and a user.

[1233] Overall system structure

[1234] This system consists of the following hardware and software:

[1235] Hardware: Smart glasses (with camera, microphone, and speaker), server (data processing and storage), smartphone

[1236] Software: Python, OpenCV, face_recognition, Vosk (voice recognition)

[1237] Server Processing

[1238] 1. Data Receipt and Analysis:

[1239] The server receives data such as voice, text, and photos provided by the user via the device, which is then analyzed using AI algorithms to identify important memories and events from the user's past.

[1240] 2. New Events Recorded:

[1241] The server uses a voice recognition function to record the user's daily activities in real time, analyzes the voice data, and saves it as text data.

[1242] 3. Providing memory support:

[1243] Based on the analysis results, the server generates content (such as photos and voice memos) to stimulate the user's memory, and also displays past events based on the user's location or a specific time.

[1244] Terminal handling

[1245] 1. Data input and transmission:

[1246] The device (smart glasses or smartphone) receives the user's voice and text input and sends it to the server. The input data is temporarily stored on the device, and transmission to the server is managed.

[1247] 2. Facial Recognition and Information Display:

[1248] Using the device's camera, it recognizes the face of a specific person and displays information related to that person, helping users remember the name of the person in front of them and past events.

[1249] 3. Reminders and notifications:

[1250] The device will display reminders and notifications sent from the server to the user visually and audibly at the appropriate time, for example, a reminder to take medicine at a specific time.

[1251] User operations

[1252] 1. Enter historical data:

[1253] Users can upload past photos and voice memos to the device, and this data is sent to a server and stored for analysis.

[1254] 2. Recording daily events:

[1255] Users can record daily events by voice input into the device, and this data is sent to the server and stored in a database.

[1256] 3. Check your reminders:

[1257] Users can check reminder notifications from their devices and perform necessary events and activities in a timely manner.

[1258] Specific examples

[1259] 1. Helping people recall past memories:

[1260] When a user wears the smart glasses and walks around town, photos and events from previous visits to the same places are automatically displayed, evoke memories of those times by showing photos of past trips.

[1261] 2. New Events Recorded:

[1262] When a user says, "I went for a walk with a friend today," the data is sent to the server and recorded. Later, when the user asks, "When was the day I went for a walk with a friend?", the device provides that information.

[1263] 3. Providing Reminders:

[1264] At a specific time, the smart glasses' voice function will remind the user, saying, "It's time to take your medicine," to help them remember.

[1265] Prompt Sentence Examples

[1266] Project: Memory Spot

[1267] Objective: To develop a system that uses smart glasses to support memory in elderly people with dementia.

[1268] procedure:

[1269] Uses face recognition library (face_recognition) and speech recognition library (Vosk).

[1270] Faces are recognized in the camera frame and the recognition results are notified via voice.

[1271] Record daily events using voice input from a microphone.

[1272] Regular voice reminders.

[1273] In this way, the system of the present invention is designed to enable elderly people with dementia to live their daily lives more efficiently and safely.

[1274] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1275] Program processing flow

[1276] Step 1:

[1277] The user inputs data into the device, such as voice input or photo upload. The device then uses the microphone to capture the voice and the camera to take a photo. The input data is temporarily stored on the device.

[1278] Input: User's voice and photo data

[1279] Output: Temporarily saved audio and image files

[1280] Step 2:

[1281] The device then sends the temporarily stored data to the server using a communication protocol via the Internet.

[1282] Input: Temporarily saved audio and image files

[1283] Output: Audio and image files sent to the server

[1284] Step 3:

[1285] The server converts the received voice data into text using the Vosk speech recognition library, and the converted text data is stored in a database.

[1286] Input: Audio file

[1287] Output: Text data

[1288] Step 4:

[1289] The server analyzes the received photo data using AI algorithms (e.g., image recognition models) to identify past events, and the analysis results are stored in a database.

[1290] Input: Image file

[1291] Output: Analysis results (past events)

[1292] Step 5:

[1293] The device's integrated facial recognition function is used to detect and identify faces from the user's current camera image. Based on the detection results, the server sends relevant information to the device.

[1294] Input: Real-time camera footage

[1295] Output: Detected person information

[1296] Step 6:

[1297] The device displays personal information and past events sent from the server to the user through the smart glasses' display or voice notification.

[1298] Input: Related information (personal information, past events)

[1299] Output: Visual and audio notification to the user

[1300] Step 7:

[1301] The server generates reminders based on schedules and routines and sends them to the device, where the reminder content is displayed to the user as text or audio data.

[1302] Input: Schedule data, routine data

[1303] Output: Reminder text data, audio data

[1304] Step 8:

[1305] Users receive reminders and notifications from their devices and act on them, such as remembering to take their medication or participate in a specific activity.

[1306] Input: Reminder text data, voice data

[1307] Output: User action execution

[1308] In this way, the system can provide comprehensive support for dementia care for the elderly and improve the quality of their daily lives.

[1309] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1310] The present invention relates to an integrated memory support system for supporting the memory of elderly people with dementia, and provides more effective support by combining an emotion engine that recognizes the user's emotions.

[1311] Overall system structure

[1312] This system is mainly composed of three components: a server, a terminal, and a user, as well as an emotion engine. The server processes and stores all data, while the terminal provides an interface for the user, who interacts with the system through the terminal. The emotion engine recognizes the user's emotional state and provides feedback to the system.

[1313] Server Processing

[1314] 1. Data collection and analysis

[1315] The server receives data provided by the user through the terminal, including voice, text, and photos.

[1316] The server analyzes the received data using AI algorithms to identify important memories and events from the user's past.

[1317] The user's emotional data recognized by the emotion engine is also analyzed at the same time.

[1318] 2. Emotional Recognition

[1319] The server receives emotion data from the emotion engine and grasps the user's current emotional state. The emotion engine recognizes the user's emotions in real time from changes in voice and facial expressions.

[1320] 3. Providing memory support

[1321] The server generates content to stimulate memories at appropriate times according to the user's situation and emotional state. For example, if the server recognizes that the user is feeling anxious, it selects content related to past happy memories.

[1322] The content and timing of reminders can also be adjusted based on emotional data, for example, if a user is feeling irritated, a reminder will be sent suggesting actions to calm them down.

[1323] Terminal handling

[1324] 1. Data input

[1325] The device accepts the user's voice and text input and prepares it for transmission to the server. The device temporarily stores the input data and manages transmission to the server.

[1326] 2. Emotion Detection

[1327] The device uses an emotion engine to detect emotions from the user's voice and facial expressions in real time and transmits the data to the server.

[1328] 3. Reminders and notifications

[1329] The device will display reminders and notifications from the server to the user at the appropriate time, such as reminders to help users remember their daily routines.

[1330] 4. Conversation support

[1331] The device can provide users with information if they cannot remember specific information during a conversation, such as the name of a specific person or supplementary information about a past event.

[1332] User operations

[1333] 1. Entering historical data

[1334] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[1335] 2. Recording everyday events

[1336] Users input their daily events into the terminal, and this input data is sent to the server and recorded.

[1337] 3. Check your reminders

[1338] Users can check reminders from their devices and carry out necessary events and activities in a timely manner.

[1339] 4. Emotional Feedback

[1340] When a user interacts with the device, the emotion engine detects the user's emotional state from their facial expressions and tone of voice and sends that information to the server, allowing the system to grasp the user's emotional state in real time.

[1341] Specific examples

[1342] 1. Emotional memory stimulation

[1343] When a user uploads photos of past trips to their device, the server analyzes the photos and identifies them as travel destinations from, say, 2010. If the emotion engine recognizes the user's emotion of joy, the server highlights and presents happy memories related to that trip.

[1344] 2. Recording new events and connecting emotions

[1345] When a user types "I went for a walk with a friend today" into the device, that data is sent to the server. The emotion engine records the happy emotion sensed from the user's voice and uses the emotion data when reminding the user of the event later.

[1346] 3. Providing Emotion-Based Reminders

[1347] If the emotion engine on the server recognizes that the user is frustrated, it generates a reminder to relax, such as "take a deep breath," and sends it to the device. The device then notifies the user.

[1348] As described above, by combining the system of the present invention with an emotion engine, it is possible to more effectively support the memory of elderly people with dementia and improve the quality of their daily lives.

[1349] The processing flow will be explained below.

[1350] Step 1:

[1351] The user inputs past photos, voice memos, and text data into the device. The user selects the data using the device's input interface and clicks the upload button.

[1352] Step 2:

[1353] The terminal temporarily stores the data received from the user and prepares it for transmission to the server. The terminal checks the data format and performs error checks.

[1354] Step 3:

[1355] The device sends the prepared data to the server. The server receives the data and stores it in a database. The server checks the integrity of the received data and checks for incomplete data.

[1356] Step 4:

[1357] The server uses AI algorithms to analyze the data it receives and identify key events and memories from the user's past. The server then organizes the information and prepares it for the next processing step.

[1358] Step 5:

[1359] The server analyzes the emotion data provided by the emotion engine, which detects the user's emotion in real time from voice tone and facial expressions, and sends the data to the server.

[1360] Step 6:

[1361] The server generates content to stimulate memories based on the analysis results and emotional data. If the user is in an anxious state, it selects and provides content related to past happy memories.

[1362] Step 7:

[1363] The device displays the generated content to the user, who can then review past photos and voice memos to stimulate their memories. For example, a user might view photos from a trip in 2010 and recall fond memories from that time.

[1364] Step 8:

[1365] The user inputs new events and daily events into the device, inputting the activities they performed that day and their feelings in text, and then clicks the send button.

[1366] Step 9:

[1367] The device receives new data, temporarily stores it, and then sends it to the server. The server receives the new data and records it in the database. This updates the user's daily life history.

[1368] Step 10:

[1369] The user sets the time to take daily medicine or schedule regular health checkups on the device. The user specifies the content and time of the reminder and clicks the setting button.

[1370] Step 11:

[1371] The server generates reminders based on the user's settings, and adjusts the content and timing of the reminders as needed, taking into account emotion data from the emotion engine.

[1372] Step 12:

[1373] The device will display a reminder notification to the user at the specified time. For example, it could display a notification saying "It's time to take your medicine," prompting the user to take action.

[1374] Step 13:

[1375] If a user cannot remember certain information during a conversation, they can ask for help from the device by entering a question using the device's question interface and clicking the send button.

[1376] Step 14:

[1377] The server receives the user's query and searches the database for the relevant information. The server responds quickly and identifies the information the user is looking for.

[1378] Step 15:

[1379] The server sends the identified information to the terminal, which displays it to the user. The user confirms the displayed information and continues the conversation.

[1380] Step 16:

[1381] The emotion engine runs even while the user is interacting with the system, continuously monitoring the user's emotional changes. Emotional data is fed back to the server in real time, and appropriate action is taken as necessary.

[1382] Through these steps, the system of the present invention can effectively support the memory of elderly people with dementia and improve the quality of their daily lives. By combining it with an emotion engine, it can provide more flexible and user-friendly support.

[1383] Example 2

[1384] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1385] To support the memory of elderly people with dementia, simply providing reminders and notifications is not enough. Systems that take the user's emotional state into account and appropriately stimulate past memories to improve the quality of daily life are needed. In particular, it is important to have a means to provide information when the user cannot remember specific information during a conversation, and a means to record the user's daily events and provide reminders based on that information.

[1386] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1387] In this invention, the server includes means for receiving and storing input data from a user, means for normalizing and analyzing the received data to identify past memories, means for receiving and analyzing emotional data to recognize the user's emotional state, means for generating content to stimulate memory based on the analysis and emotional analysis results, means for notifying the user of the generated content and reminders at an appropriate time, and means for providing information if the user cannot remember specific information during a conversation. This makes it possible to provide appropriate reminders and information based on the user's emotional state, thereby more effectively supporting memory in elderly people with dementia.

[1388] "Input data" refers to information such as voice, text, and photos provided by the user through the terminal.

[1389] "Normalization" refers to the process of converting received data into a format that is easier to analyze.

[1390] "Analysis" refers to the process of using AI algorithms to detect patterns and features in normalized data and identify past memories and events.

[1391] "Emotional data" refers to information about a user's emotional state obtained from their tone of voice and facial expressions.

[1392] "Emotion engine" refers to software or algorithms that analyze a user's tone of voice and facial expressions in real time to recognize their emotional state.

[1393] "Content" refers to information and media files (e.g., photos, videos, text) generated to stimulate a user's memory.

[1394] A "reminder" refers to a notification that reminds a user of a particular action or event.

[1395] "Notification" refers to a message from the system to inform the user of a reminder.

[1396] "Means" refers to a method or apparatus for accomplishing a particular function.

[1397] "Server" refers to a central device for data processing, analysis, and storage.

[1398] "Terminal" refers to the device (e.g., smartphone, tablet, or PC) that a user uses to interact with the system.

[1399] "Past memories" refer to important events and memories that the user has experienced in the past.

[1400] "Memory stimulation" refers to actions or content provided to evoke a user's past memories.

[1401] "Right time" refers to providing notifications and reminders at the most appropriate time based on the user's situation and emotional state.

[1402] This invention is an integrated memory support system for supporting the memory of elderly people with dementia. This system is composed of three main components: a server, a terminal, and a user, as well as an emotion engine. The functions of each component and the specific processing contents of the program are described in detail below.

[1403] Server Processing

[1404] The server is a central device that processes, analyzes, and stores various data. The server performs the following main processes:

[1405] 1. Data collection and analysis

[1406] The server receives data such as voice, text, and photos provided by the user through the device, and uses an AI algorithm such as TensorFlow.

[1407] The received data is normalized, and if it is audio data, it is converted into text data using a speech recognition engine such as Google Speech-to-Text.

[1408] The normalized data is analyzed to identify significant memories and events from the user's past.

[1409] 2. Emotional Recognition

[1410] The server receives emotional data from the emotion engine, which analyzes the user's voice tone and facial expressions, and determines their current emotional state. The emotion engine uses Microsoft Azure's emotion recognition API, among other things.

[1411] 3. Providing memory support

[1412] The server generates content to stimulate memories based on the analysis results and the user's emotional state. For example, if the user is anxious, the server generates content related to past happy memories.

[1413] Adjust the content and timing of reminders based on emotional data.

[1414] Terminal handling

[1415] A terminal is a device that functions as an interface between a user and a system. The main processes of a terminal are as follows:

[1416] 1. Data input

[1417] The device accepts voice and text input from the user. When the user records a diary entry, the device temporarily stores the data and prepares it for transmission to the server.

[1418] 2. Emotion Detection

[1419] The emotion engine detects emotions in real time from the user's voice and facial expressions, using IBM Watson's emotion analysis API.

[1420] 3. Reminders and notifications

[1421] Displaying reminders and notifications from the server to the user at appropriate times, for example, a reminder to "Don't forget to take your morning walk."

[1422] 4. Conversation support

[1423] If the user cannot remember a particular piece of information during a conversation, the system will provide that information to the user. For example, if the user asks, "Who is this person?", the system will search past memories and display the relevant information.

[1424] User operations

[1425] The user can perform the following operations through the terminal.

[1426] 1. Entering historical data

[1427] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[1428] 2. Recording everyday events

[1429] Users input daily events into the device using voice or text, and the input data is sent to the server and recorded.

[1430] 3. Check your reminders

[1431] The user checks the reminder from the device and performs the necessary tasks or events, for example, checking and performing the reminder to "take a deep breath."

[1432] 4. Emotional Feedback

[1433] When a user interacts with the device, the emotion engine detects the user's emotional state from facial expressions and tone of voice and sends this information to the server.

[1434] Specific examples

[1435] 1. Emotional memory stimulation

[1436] When a user uploads photos from past trips to their device, the photos are sent to the server, which analyzes the photos to identify the travel destination and time period. If the emotion engine detects the user's joy, the server highlights and displays happy memories related to that trip.

[1437] Prompt: "Analyze a travel photo. Identify when and where the photo was taken and provide the user with relevant memories."

[1438] 2. Recording new events and connecting emotions

[1439] When a user types "I went for a walk with a friend today" into the device, that data is sent to the server. If the emotion engine senses the user's joy, it also records that emotion data and uses it to remind the user later.

[1440] Prompt: "Analyze the user's diary and record the events and emotions of the day."

[1441] 3. Providing Emotion-Based Reminders

[1442] When the emotion engine detects that the user is frustrated, it sends this data to the server, which then generates a reminder to "take a deep breath" and sends it to the device, which then notifies the user of the reminder.

[1443] Prompt: "If the user is frustrated, generate a reminder and notify them."

[1444] By utilizing data from the emotion engine, this system will more effectively support the memory of elderly people with dementia and contribute to improving the quality of their daily lives.

[1445] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1446] Server Processing

[1447] Step 1: Receiving Data

[1448] The server receives data such as the user's voice, text, and photos from the device. For example, when a user records a voice diary entry, the voice file is transferred from the device to the server.

[1449] Input: Voice, text, and photo data sent from your device.

[1450] Output: Saving received data to storage.

[1451] Step 2: Normalize and preprocess the data

[1452] The server normalizes the received data and uses a speech recognition engine (e.g., Google Speech-to-Text) to convert the audio data into text data.

[1453] Input: Received data (voice, text, photos).

[1454] Output: Normalized data, speech data converted to text.

[1455] Step 3: Analyze the data

[1456] The server analyzes the normalized data using AI algorithms (e.g., TensorFlow) to identify important memories and events from the user's past.

[1457] Input: Normalized data.

[1458] Output: Identified memory or event information.

[1459] Step 4: Receiving emotion data

[1460] The server receives the user's emotional data (voice tone and facial expression analysis results) from the emotion engine.

[1461] Input: Emotion data sent from the emotion engine.

[1462] Output: Save emotion data to storage.

[1463] Step 5: Analyze the sentiment data

[1464] The server analyzes the received emotion data to understand the user's current emotional state, for example, determining whether the user is happy or sad from the tone of their voice.

[1465] Input: Emotion data.

[1466] Output: Parsed emotional state information.

[1467] Step 6: Generate memory support content

[1468] The server generates content to stimulate memories based on the analysis results and the user's emotional state. For example, if the user is anxious, the server generates content related to past happy memories.

[1469] Input: Identified memory or event information, analyzed emotional state information.

[1470] Output: The generated content.

[1471] Step 7: Adjust reminders and notifications

[1472] The server adjusts the content and timing of reminders based on emotional data. For example, if a user is feeling stressed, it generates a reminder to take a deep breath to relax.

[1473] Input: Parsed emotional state information.

[1474] Output: Coordinated reminders and notifications.

[1475] Step 8: Sending Data

[1476] The server sends the generated content and adjusted reminders to the device.

[1477] Input: Generated content, tailored reminders.

[1478] Output: Data sent to the terminal.

[1479] Terminal handling

[1480] Step 1: Data Input

[1481] The device accepts the user's voice and text input and prepares it to be sent to the server. For example, when a user records a diary entry, the device temporarily stores the voice data.

[1482] Input: User voice and text input.

[1483] Output: Temporarily saved data.

[1484] Step 2: Sending data

[1485] The terminal transmits the temporarily stored data to the server.

[1486] Input: Temporarily saved data.

[1487] Output: Data sent to the server.

[1488] Step 3: Emotion detection

[1489] The device uses an emotion engine to detect emotions from the user's voice and facial expressions in real time, for example, by analyzing voice and identifying emotions.

[1490] Input: Real-time user voice and facial expression data.

[1491] Output: Parsed emotion data.

[1492] Step 4: Sending emotion data

[1493] The terminal transmits the analysis results to the server.

[1494] Input: Parsed emotion data.

[1495] Output: Data sent to the server.

[1496] Step 5: Receive reminders

[1497] The device receives reminders and notifications from the server, such as a reminder to "don't forget to take a morning walk."

[1498] Input: Reminder and notification data from the server.

[1499] Output: Received data to the terminal.

[1500] Step 6: View Reminders

[1501] The device will display reminders and notifications to the user at appropriate times, using pop-up notifications and sound notifications.

[1502] Input: Received reminder, notification data.

[1503] Output: What is displayed to the user.

[1504] Step 7: Conversation support

[1505] If the user cannot remember a particular piece of information during a conversation, the device will provide that information to the user. For example, if the user asks, "Who is this person?", the device will search past memories and display relevant information.

[1506] Input: User question.

[1507] Output: Display of relevant information.

[1508] User operations

[1509] Step 1: Enter historical data

[1510] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[1511] Input: Past photos, voice memos.

[1512] Output: Upload data to the device.

[1513] Step 2: Record your daily events

[1514] Users input daily events into the device using voice or text, and the input data is sent to the server and stored as a record.

[1515] Input: Audio and text data of the user's daily life.

[1516] Output: Record data to terminal.

[1517] Step 3: Check your reminders

[1518] The user can check the reminders received from the device and perform the necessary tasks or events, for example, checking and performing the reminder to "take a deep breath."

[1519] Enter: reminders, notifications.

[1520] Output: The user's action.

[1521] Step 4: Emotional Feedback

[1522] When a user interacts with the device, the emotion engine detects the user's emotional state from facial expressions and tone of voice and sends this information to the server.

[1523] Input: User's facial expression, tone of voice.

[1524] Output: Data sent to the server.

[1525] (Application example 2)

[1526] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1527] The problem that this invention aims to solve is to prevent elderly people with dementia from experiencing difficulties in their daily lives due to memory confusion and forgetting important information, thereby improving their quality of life. Furthermore, by providing appropriate support according to the elderly's emotional state, it aims to reduce stress and anxiety, allowing them to live their daily lives with greater peace of mind.

[1528] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1529] In this invention, the server includes means for receiving input data from a user, means for analyzing the received data and identifying past memories, means for generating content for stimulating memory based on the analysis results, means for providing reminders and notifications to the user, and means for recognizing the user's emotions in real time using an emotion engine and reflecting that data in the analysis, thereby providing appropriate and timely support based on the user's emotional state, enabling memory improvement and stress reduction.

[1530] "Means for receiving input data from a user" is a general term for hardware and software that allows a user to input data such as voice, text, or images into the system and receive it.

[1531] "Means for analyzing received data and identifying past memories" refers to the algorithms and processes by which the system analyzes received data and identifies the user's past events and important memories.

[1532] The "means for generating content to stimulate memory based on the analysis results" is a mechanism for automatically generating content such as text, images, and audio to stimulate the user's memory based on identified past memories.

[1533] "Means for providing user reminders and notifications" are mechanisms for informing users of system-generated content and important information in a timely manner.

[1534] "Means of recognizing user emotions in real time using an emotion engine and reflecting that data in analysis" refers to technology that detects the user's emotional state in real time from their voice and facial expressions, and incorporates that information into the system's analysis process.

[1535] MODE FOR CARRYING OUT THE INVENTION

[1536] The present invention provides a smart shopping support system for supporting the shopping experience of elderly people with dementia, allowing them to live their lives with greater peace of mind. The following are embodiments of the invention.

[1537] Overall system structure

[1538] The system mainly consists of three components: a server, a terminal, and a user. The server is responsible for the main data processing and storage, and the terminal provides an interface for the user, through which the user interacts with the system.

[1539] Server Processing

[1540] The server does the following:

[1541] 1. Data collection and analysis

[1542] The server receives and analyzes data such as voice, text, and images provided by the device, and uses a generative AI model to identify the user's past memories and important events.

[1543] Emotion engine data will also be analyzed at the same time.

[1544] 2. Emotional Recognition

[1545] The server uses an emotion engine to recognize the user's emotions in real time and incorporates that data into its analysis, identifying emotions based on changes in voice and facial expressions.

[1546] 3. Providing memory support

[1547] The server generates content to stimulate memories according to the user's situation and emotional state. For example, if the user is feeling anxious, it selects content related to past happy memories.

[1548] 4. Reminders and notifications

[1549] The server generates reminders and notifications for users at appropriate times and sends them to their devices, helping them to remember their daily routines and important events.

[1550] Terminal handling

[1551] The terminal does the following:

[1552] 1. Data input

[1553] The device accepts the user's voice and text input and prepares it for transmission to the server. The input data is temporarily stored and managed for transmission to the server.

[1554] 2. Emotion Detection

[1555] The device uses an emotion engine to detect emotions from the user's voice and facial expressions in real time and transmits the data to the server.

[1556] 3. Reminders and notifications

[1557] The device will display reminders and notifications from the server to the user at the appropriate time, such as reminders to help users remember their daily routines.

[1558] 4. Conversation support

[1559] The device can provide users with information if they cannot remember specific information during a conversation, such as the name of a specific person or supplementary information about a past event.

[1560] User operations

[1561] The user does the following:

[1562] 1. Create a shopping list

[1563] The user adds an item to the shopping list, for example, by entering a voice command such as "add milk to the shopping list" into the terminal, which is then sent to the server.

[1564] 2. Recording everyday events

[1565] Users input daily events into the device by voice or text, for example, "Today I went for a walk with a friend."

[1566] 3. Emotional Feedback

[1567] When a user interacts with the device, the emotion engine detects the user's emotional state from their facial expressions and tone of voice, and sends that information to the server.

[1568] Specific examples

[1569] For example:

[1570] 1. Create a shopping list

[1571] When the user types "Add milk to the shopping list," data is sent from the device to the server, and milk is added to the shopping list.

[1572] 2. Emotional awareness and reminders

[1573] When a user types "I'm feeling happy today," the emotion engine recognizes the emotion as "happy" and sends it to the server, which then generates a reminder such as "Stay happy and enjoy shopping," which is then sent to the device.

[1574] Prompt Sentence Examples

[1575] Prompts to identify the user's emotional state:

[1576] Identify the emotional state based on user input. Input text: "Today was a very happy day!"

[1577] If the keyword "happy" is included, the emotional state will be set to "happy".

[1578] Input: "Today was a very happy day!", Identified emotion: "happy"

[1579] Prompt to record past events:

[1580] The user enters a past event. Event: "I went on a fun picnic with my family."

[1581] Emotion: Record the associated emotion "happy."

[1582] Input: "I went on a fun picnic with my family", emotion: "happy"

[1583] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1584] Step 1:

[1585] A user creates a shopping list. The user inputs a voice command into the terminal, such as "Please add milk to the shopping list." The terminal converts this voice command into text data and sends it to the server. The server receives this data and adds "milk" to the shopping list.

[1586] Input: User speaks "Add milk to my shopping list"

[1587] Data processing: Converting voice data into text data

[1588] Output: "Milk" is added to the shopping list

[1589] Step 2:

[1590] The user records a past event. The user types, "Today I went for a walk with a friend" into the device. The device converts this voice data into text data and sends it to the server. The server identifies the received text data and adds its contents to a list of past events.

[1591] Input: User's voice input: "I went for a walk with a friend today"

[1592] Data processing: Converting voice data into text data

[1593] Output: "I went for a walk with a friend today" is added to the list of past events

[1594] Step 3:

[1595] The device detects the user's emotions. The user speaks, "I'm feeling happy today." The device converts the voice data into text data and analyzes it with the emotion engine. The emotion engine recognizes "happy" from the text and sends the information to the server.

[1596] Input: User's voice input: "I'm feeling happy today"

[1597] Data processing: Converts voice data into text data and analyzes emotions with an emotion engine

[1598] Output: Emotional state "happy" is sent to the server

[1599] Step 4:

[1600] The server generates appropriate reminders based on the user's emotional state. For example, if the server receives the emotion data "happy," it generates a reminder such as "Stay happy and enjoy shopping." Using this generative AI model, the server inputs prompts to create reminders.

[1601] Input: Emotional state "happy"

[1602] Data processing: Generative AI models are used to generate reminders

[1603] Output: Reminder: "Stay happy and enjoy your shopping"

[1604] Step 5:

[1605] The device notifies the user of the reminder. The device receives the reminder "Keep your shopping fun and enjoy yourself" sent from the server and displays it as a notification on the user's device.

[1606] Input: Reminder sent from the server

[1607] Data processing: Convert reminders into notifications

[1608] Output: The reminder appears on the user's device

[1609] Through these steps, the system supports the shopping experience of elderly people with dementia, providing users with appropriate feedback based on their emotional state, allowing them to carry out their daily shopping with greater peace of mind.

[1610] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1611] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1612] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1613] [Fourth embodiment]

[1614] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1615] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1616] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1617] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1618] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1619] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1620] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1621] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1622] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1623] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1624] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1625] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1626] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1627] The present invention relates to an integrated memory support system for supporting the memory of elderly people with dementia, specifically, a system that receives and analyzes data from users, generates memory-stimulating content, and provides timely reminders and notifications.

[1628] Overall system structure

[1629] The system mainly consists of three components: a server, a terminal, and a user. The server processes and stores all data, and the terminal provides an interface for the user, through which the user interacts with the system.

[1630] Server Processing

[1631] 1. Data collection and analysis

[1632] The server receives data provided by the user through the terminal, including voice, text, and photos.

[1633] The server then analyzes the received data using AI algorithms, which identify important memories and events from the user's past.

[1634] 2. Recording New Events

[1635] The server stores the data in real time as users enter their daily events, recording their daily routines and new events.

[1636] 3. Providing memory support

[1637] The server generates appropriate content based on the user's current situation, which stimulates the user's memory based on past photos and voice memos.

[1638] For example, a user who is in a particular location may be sent content that reminds them of a past event related to that location.

[1639] Terminal handling

[1640] 1. Data input

[1641] The device accepts the user's voice and text input and prepares it for transmission to the server. The device temporarily stores the input data and manages transmission to the server.

[1642] 2. Reminders and notifications

[1643] The device will display reminders and notifications from the server to the user at the appropriate time, such as reminders to help users remember their daily routines.

[1644] 3. Conversation support

[1645] The device can provide users with information if they cannot remember specific information during a conversation, such as the name of a specific person or supplementary information about a past event.

[1646] User operations

[1647] 1. Entering historical data

[1648] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[1649] 2. Recording everyday events

[1650] Users input their daily events into the terminal, and this input data is sent to the server and recorded.

[1651] 3. Check your reminders

[1652] Users can check reminders from their devices and carry out necessary events and activities in a timely manner.

[1653] Specific examples

[1654] 1. Helping people recall past memories

[1655] A user uploads photos of past trips to the device, and the server analyzes the photos and identifies them as trips taken in 2010. If the user then asks a question related to the travel destination, the device will notify them, "This is a trip taken in 2010."

[1656] 2. Recording New Events

[1657] When a user types "I went for a walk with a friend today" into the device, that data is sent to the server and recorded. Later, when the user asks "When was the day I went for a walk with a friend?", the device provides that information.

[1658] 3. Providing reminders

[1659] The system records the user's daily medication intake times, and when the time approaches, the server sends a reminder to the device, which then notifies the user, "It's time to take your medicine."

[1660] As described above, the system of the present invention provides an effective means of supporting the memory of elderly people with dementia and improving the quality of their daily lives.

[1661] The processing flow will be explained below.

[1662] Step 1:

[1663] Users input past photos, voice memos, and text data into the device, select data related to past events and memories, and click the upload button.

[1664] Step 2:

[1665] The device temporarily stores the data received from the user and prepares it for transmission to the server. The device checks the data format and performs error checking to ensure smooth transmission.

[1666] Step 3:

[1667] The device sends the prepared data to the server. The server receives the data and stores it in a database. The server checks the integrity of the data and checks for incomplete data.

[1668] Step 4:

[1669] The server then runs the data through an AI algorithm to analyze it, identifying key events and memories from the user's past and extracting relevant information.

[1670] Step 5:

[1671] The server uses the analysis results to generate content to stimulate the user's memory, such as photos of past travel destinations or text related to events.

[1672] Step 6:

[1673] The user inputs new events and daily events into the device, inputting the activities they performed that day and their feelings in text, and then clicks the send button.

[1674] Step 7:

[1675] The device receives new data, temporarily stores it, and then sends it to the server. The server receives the new data and records it in the database. This updates the user's daily life history.

[1676] Step 8:

[1677] To utilize the system's reminder function, users set specific events or routines on their devices, such as setting a daily medication intake time or scheduling a regular health checkup.

[1678] Step 9:

[1679] The server sets a timer based on the user's settings, generates a reminder at the appropriate time, and sends the reminder content to the device, causing a notification to be displayed.

[1680] Step 10:

[1681] The device displays a reminder notification to the user at the specified time. The user checks the notification on the device and takes the necessary action, such as taking medicine.

[1682] Step 11:

[1683] If a user cannot remember a particular piece of information during a conversation, they can ask for help from the terminal by typing a question into the terminal and sending it.

[1684] Step 12:

[1685] The server receives the user's query and searches the database for the relevant information. The server responds quickly and identifies the information the user is looking for.

[1686] Step 13:

[1687] The server sends the identified information to the terminal, which displays it to the user. The user confirms the displayed information and continues the conversation.

[1688] Through the above steps, the system of the present invention can effectively support the memory of elderly people with dementia and improve the quality of their daily lives.

[1689] Example 1

[1690] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1691] The present invention aims to provide a system that supports elderly people with dementia in recalling past memories in their daily lives and in carrying out daily events and routines without forgetting them. In particular, there is a need for an effective means for users to easily recall their past memories and appropriately manage their daily actions and events, but existing technologies have not been able to completely solve this problem.

[1692] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1693] In this invention, the server includes a means for receiving input data from a user, a means for analyzing the received data using multiple analysis techniques, and a means for recording the user's behavioral patterns and routines in real time based on the analysis results, thereby enabling appropriate support for the user's memory and improving the quality of daily life.

[1694] "User input data" refers to information such as voice, text, and photos provided by the user through the terminal.

[1695] "Analyzing received data" means that the server analyzes the data received from the user using natural language processing, image recognition, etc.

[1696] "Identifying past memories" refers to revealing important events or memories from the user's past based on the data analyzed by the server.

[1697] "Generating content to stimulate memories" means that the server creates content such as photos, text, and audio that are generated for the purpose of stimulating the user's memories based on past memories.

[1698] "Providing reminders and notifications to the user" means that the server generates reminders and notifications, sends them to the user at the appropriate time, and the terminal displays them.

[1699] "Transmitting to a server via a network" means that the terminal transmits the user's input data to a server using a communication means such as the Internet.

[1700] "Analysis using multiple analysis technologies" means that the server analyzes data by combining different technologies such as natural language processing (NLP), image recognition, and voice analysis.

[1701] "Recording user behavior patterns and routines in real time" means that the server immediately saves and manages the user's daily behavior and routines based on the analysis results.

[1702] The present invention relates to an integrated memory support system for supporting the memory of elderly people with dementia. The system receives input data from a user, analyzes the data using multiple analytical techniques, identifies the user's past memories, generates content to stimulate memory based on the analysis results, and provides reminders and notifications at appropriate times.

[1703] Overall system structure

[1704] The system mainly consists of three components: a server, a terminal, and a user. The server processes and stores all data, and the terminal provides an interface for the user, through which the user interacts with the system.

[1705] Server Processing

[1706] The server processes the input data using multiple analysis technologies, including natural language processing (NLP), image recognition, and voice analysis. For example, it can analyze photo data using Google Cloud Vision API or Amazon Rekognition, and analyze text data using Google Natural Language API.

[1707] Data collection and analysis

[1708] The server receives data provided by the user through the device. This data includes voice, text, photos, etc. For example, if a user uploads photos of their trip, the server receives the photo data and analyzes it using image recognition technology. This analysis can identify important memories and events from the user's past.

[1709] Recording new events

[1710] The server stores the data in real time as the user inputs their daily events. For example, if a user inputs "I went for a walk with a friend today," the server analyzes the data and stores it in a MySQL database or similar.

[1711] Providing memory support

[1712] Based on the analysis results, the server generates content to stimulate the user's memory. This content can include past photos and voice memos, and is sent to the user via email or push notification. For example, when a user is in a specific location, they will receive a notification about past events related to that location.

[1713] Terminal handling

[1714] The device accepts voice and text input from the user, temporarily stores it in local storage, and then transmits the input data to a server over the network, for example, using Google Firebase or a REST API.

[1715] Data Input

[1716] The device receives the user's voice and text input and prepares it to be sent to the server. For example, if the user says, "Set the time to take my medicine," the device sends that data to the server.

[1717] Reminders and notifications

[1718] The device will display reminders and notifications from the server to the user at appropriate times, for example, a pop-up notification saying "It's time to take your medicine" at a specific time.

[1719] User operations

[1720] Users can use the device to upload past photos and voice memos. This data is automatically sent to the server, where it is analyzed and saved. Users can also enter daily events into the device, and the data is sent to the server and recorded in real time.

[1721] Specific examples

[1722] 1. Helping people recall past memories

[1723] A user uploads photos of a trip from 2010 to their device. The server analyzes the photos and identifies them as trips from 2010. Then, when the user asks about the trip, the device will say, "This is a trip from 2010."

[1724] Prompt Sentence Examples

[1725] Here are some photos from a trip we took in 2010. Please provide a suitable description when users ask about the destination.

[1726] 2. Recording New Events

[1727] A user enters "I went for a walk with a friend today" into a device. This information is sent to a server and recorded. Later, when the user asks "When was the day I went for a walk with a friend?", the device provides that information.

[1728] Prompt Sentence Examples

[1729] Record today's date and the information that you "went for a walk with a friend." When you ask for this information later, please let us know when you went for a walk.

[1730] 3. Providing reminders

[1731] The user sets the time to take their medicine daily, and when that time approaches, the server sends a reminder to the device, which then notifies them that it is time to take their medicine.

[1732] Prompt Sentence Examples

[1733] Set a time for the user to take their medicine, and when that time approaches, display a reminder saying "It's time to take your medicine."

[1734] As described above, the system of the present invention provides an effective means for supporting the memory of elderly people with dementia and improving their quality of daily life.

[1735] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1736] Step 1: User enters data into the terminal

[1737] A user opens an application on their device and enters "I went for a walk with a friend today" into a text box. The input data (text or voice) is temporarily stored in the device's local storage. The user provides the text "I went for a walk with a friend today" as input data, and the data is temporarily stored in the device.

[1738] Step 2: The device sends the data to the server

[1739] The device sends the text data entered by the user to the server via the network. The device uses an HTTPS request to send the entered text data, "I went for a walk with a friend today," in JSON format to the server. Specifically, the device calls a REST API to send the data to a specific endpoint on the server.

[1740] Step 3: The server receives and analyzes the data

[1741] The server stores the received text data in Google Cloud Storage or Amazon S3. The server uses a Python script to analyze the text data using the Google Natural Language API. The server receives the text data "I went for a walk with a friend today" as input for analysis, analyzes the text, and extracts event information containing the keywords "friend" and "walk."

[1742] Step 4: The server generates notifications and reminders based on the analysis results.

[1743] Based on the analysis results, the server records the user's behavioral patterns and saves them as new events. For example, the event information "went for a walk with a friend" is saved in a MySQL database. The server generates a reminder for the next event, allowing the user to look back on that day at a later date. The server receives the event information from the analysis results as input, generates reminder information as output, and saves it in the database.

[1744] Step 5: The server sends notifications and reminders to the device

[1745] The server generates reminder information and sends it to the device via the network. A push notification is sent using a dedicated notification API (such as Firebase Cloud Messaging). The server receives reminder information as input and generates and sends a notification message to the device as output.

[1746] Step 6: The device displays notifications and reminders to the user

[1747] The device displays notification messages received from the server to the user as popups or audio alerts. For example, it displays a reminder such as "It's time to take your medicine" on the screen to provide an alert to the user. It receives notification messages received from the server as input and displays visual and audio notifications to the user as output.

[1748] The above are the specific operations at each processing step of this system, which effectively provides memory support for elderly people with dementia.

[1749] (Application example 1)

[1750] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1751] Elderly people with dementia often experience difficulty in daily life due to memory loss and confusion. Furthermore, inability to recall past events or people can lead to problems with communication. This can lead to a decline in quality of life and increased psychological stress. Therefore, a system that can solve these problems and support the memory of elderly people with dementia is needed.

[1752] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1753] In this invention, the server includes means for receiving input data from a user, means for analyzing the received data and identifying past memories, means for displaying information about specific people using a face recognition function, means for recording daily events using a voice recognition function, means for generating content to stimulate memories based on the analysis results, means for displaying past events based on the user's location and time, and means for providing reminders and notifications to the user. This makes it easier for elderly people with dementia to remember daily events and past events and people, thereby improving their quality of life.

[1754] The "means for receiving input data from a user" is a device that has the function of receiving data such as voice, text, and photographs provided by a user via a terminal.

[1755] "Means for analyzing received data and identifying past memories" refers to a mechanism for analyzing received data using an AI algorithm to identify important memories and events from the user's past.

[1756] "Means for displaying information about a specific person using a facial recognition function" refers to technology that uses a camera to detect the face of a specific person and displays information related to that person.

[1757] The "means for recording daily events using a voice recognition function" is a system that uses a microphone to recognize a user's speech, converts it into text data, and records daily events.

[1758] "Means for generating content to stimulate memory based on analysis results" is a function that automatically creates content such as photos and voice memos to evoke the user's memories based on the analysis results.

[1759] The "means for displaying past events based on the user's location and time" is a system that presents past events related to a location or time point according to the user's current location or a specific time.

[1760] "Means for providing reminders and notifications to users" refers to a mechanism that has the function of sending visual and audio notifications to users at appropriate times according to pre-set schedules and routines.

[1761] "Means for providing visual and audio reminders via smart devices" refers to technology for providing both visual and audio reminders using devices such as smartphones or smart glasses.

[1762] "Means for providing specific information during a conversation using a voice output function" refers to a technology that uses a speaker to audibly convey information that a user needs during a conversation.

[1763] This invention is an integrated memory support system for supporting the memory of elderly people with dementia, and is realized by combining three main elements: a server, a terminal, and a user.

[1764] Overall system structure

[1765] This system consists of the following hardware and software:

[1766] Hardware: Smart glasses (with camera, microphone, and speaker), server (data processing and storage), smartphone

[1767] Software: Python, OpenCV, face_recognition, Vosk (voice recognition)

[1768] Server Processing

[1769] 1. Data Receipt and Analysis:

[1770] The server receives data such as voice, text, and photos provided by the user via the device, which is then analyzed using AI algorithms to identify important memories and events from the user's past.

[1771] 2. New Events Recorded:

[1772] The server uses a voice recognition function to record the user's daily activities in real time, analyzes the voice data, and saves it as text data.

[1773] 3. Providing memory support:

[1774] Based on the analysis results, the server generates content (such as photos and voice memos) to stimulate the user's memory, and also displays past events based on the user's location or a specific time.

[1775] Terminal handling

[1776] 1. Data input and transmission:

[1777] The device (smart glasses or smartphone) receives the user's voice and text input and sends it to the server. The input data is temporarily stored on the device, and transmission to the server is managed.

[1778] 2. Facial Recognition and Information Display:

[1779] Using the device's camera, it recognizes the face of a specific person and displays information related to that person, helping users remember the name of the person in front of them and past events.

[1780] 3. Reminders and notifications:

[1781] The device will display reminders and notifications sent from the server to the user visually and audibly at the appropriate time, for example, a reminder to take medicine at a specific time.

[1782] User operations

[1783] 1. Enter historical data:

[1784] Users can upload past photos and voice memos to the device, and this data is sent to a server and stored for analysis.

[1785] 2. Recording daily events:

[1786] Users can record daily events by voice input into the device, and this data is sent to the server and stored in a database.

[1787] 3. Check your reminders:

[1788] Users can check reminder notifications from their devices and perform necessary events and activities in a timely manner.

[1789] Specific examples

[1790] 1. Helping people recall past memories:

[1791] When a user wears the smart glasses and walks around town, photos and events from previous visits to the same places are automatically displayed, evoke memories of those times by showing photos of past trips.

[1792] 2. New Events Recorded:

[1793] When a user says, "I went for a walk with a friend today," the data is sent to the server and recorded. Later, when the user asks, "When was the day I went for a walk with a friend?", the device provides that information.

[1794] 3. Providing Reminders:

[1795] At a specific time, the smart glasses' voice function will remind the user, saying, "It's time to take your medicine," to help them remember.

[1796] Prompt Sentence Examples

[1797] Project: Memory Spot

[1798] Objective: To develop a system that uses smart glasses to support memory in elderly people with dementia.

[1799] procedure:

[1800] Uses face recognition library (face_recognition) and speech recognition library (Vosk).

[1801] Faces are recognized in the camera frame and the recognition results are notified via voice.

[1802] Record daily events using voice input from a microphone.

[1803] Regular voice reminders.

[1804] In this way, the system of the present invention is designed to enable elderly people with dementia to live their daily lives more efficiently and safely.

[1805] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1806] Program processing flow

[1807] Step 1:

[1808] The user inputs data into the device, such as voice input or photo upload. The device then uses the microphone to capture the voice and the camera to take a photo. The input data is temporarily stored on the device.

[1809] Input: User's voice and photo data

[1810] Output: Temporarily saved audio and image files

[1811] Step 2:

[1812] The device then sends the temporarily stored data to the server using a communication protocol via the Internet.

[1813] Input: Temporarily saved audio and image files

[1814] Output: Audio and image files sent to the server

[1815] Step 3:

[1816] The server converts the received voice data into text using the Vosk speech recognition library, and the converted text data is stored in a database.

[1817] Input: Audio file

[1818] Output: Text data

[1819] Step 4:

[1820] The server analyzes the received photo data using AI algorithms (e.g., image recognition models) to identify past events, and the analysis results are stored in a database.

[1821] Input: Image file

[1822] Output: Analysis results (past events)

[1823] Step 5:

[1824] The device's integrated facial recognition function is used to detect and identify faces from the user's current camera image. Based on the detection results, the server sends relevant information to the device.

[1825] Input: Real-time camera footage

[1826] Output: Detected person information

[1827] Step 6:

[1828] The device displays personal information and past events sent from the server to the user through the smart glasses' display or voice notification.

[1829] Input: Related information (personal information, past events)

[1830] Output: Visual and audio notification to the user

[1831] Step 7:

[1832] The server generates reminders based on schedules and routines and sends them to the device, where the reminder content is displayed to the user as text or audio data.

[1833] Input: Schedule data, routine data

[1834] Output: Reminder text data, audio data

[1835] Step 8:

[1836] Users receive reminders and notifications from their devices and act on them, such as remembering to take their medication or participate in a specific activity.

[1837] Input: Reminder text data, voice data

[1838] Output: User action execution

[1839] In this way, the system can provide comprehensive support for dementia care for the elderly and improve the quality of their daily lives.

[1840] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1841] The present invention relates to an integrated memory support system for supporting the memory of elderly people with dementia, and provides more effective support by combining an emotion engine that recognizes the user's emotions.

[1842] Overall system structure

[1843] This system is mainly composed of three components: a server, a terminal, and a user, as well as an emotion engine. The server processes and stores all data, while the terminal provides an interface for the user, who interacts with the system through the terminal. The emotion engine recognizes the user's emotional state and provides feedback to the system.

[1844] Server Processing

[1845] 1. Data collection and analysis

[1846] The server receives data provided by the user through the terminal, including voice, text, and photos.

[1847] The server analyzes the received data using AI algorithms to identify important memories and events from the user's past.

[1848] The user's emotional data recognized by the emotion engine is also analyzed at the same time.

[1849] 2. Emotional Recognition

[1850] The server receives emotion data from the emotion engine and grasps the user's current emotional state. The emotion engine recognizes the user's emotions in real time from changes in voice and facial expressions.

[1851] 3. Providing memory support

[1852] The server generates content to stimulate memories at appropriate times according to the user's situation and emotional state. For example, if the server recognizes that the user is feeling anxious, it selects content related to past happy memories.

[1853] The content and timing of reminders can also be adjusted based on emotional data, for example, if a user is feeling irritated, a reminder will be sent suggesting actions to calm them down.

[1854] Terminal handling

[1855] 1. Data input

[1856] The device accepts the user's voice and text input and prepares it for transmission to the server. The device temporarily stores the input data and manages transmission to the server.

[1857] 2. Emotion Detection

[1858] The device uses an emotion engine to detect emotions from the user's voice and facial expressions in real time and transmits the data to the server.

[1859] 3. Reminders and notifications

[1860] The device will display reminders and notifications from the server to the user at the appropriate time, such as reminders to help users remember their daily routines.

[1861] 4. Conversation support

[1862] The device can provide users with information if they cannot remember specific information during a conversation, such as the name of a specific person or supplementary information about a past event.

[1863] User operations

[1864] 1. Entering historical data

[1865] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[1866] 2. Recording everyday events

[1867] Users input their daily events into the terminal, and this input data is sent to the server and recorded.

[1868] 3. Check your reminders

[1869] Users can check reminders from their devices and carry out necessary events and activities in a timely manner.

[1870] 4. Emotional Feedback

[1871] When a user interacts with the device, the emotion engine detects the user's emotional state from their facial expressions and tone of voice and sends that information to the server, allowing the system to grasp the user's emotional state in real time.

[1872] Specific examples

[1873] 1. Emotional memory stimulation

[1874] When a user uploads photos of past trips to their device, the server analyzes the photos and identifies them as travel destinations from, say, 2010. If the emotion engine recognizes the user's emotion of joy, the server highlights and presents happy memories related to that trip.

[1875] 2. Recording new events and connecting emotions

[1876] When a user types "I went for a walk with a friend today" into the device, that data is sent to the server. The emotion engine records the happy emotion sensed from the user's voice and uses the emotion data when reminding the user of the event later.

[1877] 3. Providing Emotion-Based Reminders

[1878] If the emotion engine on the server recognizes that the user is frustrated, it generates a reminder to relax, such as "take a deep breath," and sends it to the device. The device then notifies the user.

[1879] As described above, by combining the system of the present invention with an emotion engine, it is possible to more effectively support the memory of elderly people with dementia and improve the quality of their daily lives.

[1880] The processing flow will be explained below.

[1881] Step 1:

[1882] The user inputs past photos, voice memos, and text data into the device. The user selects the data using the device's input interface and clicks the upload button.

[1883] Step 2:

[1884] The terminal temporarily stores the data received from the user and prepares it for transmission to the server. The terminal checks the data format and performs error checks.

[1885] Step 3:

[1886] The device sends the prepared data to the server. The server receives the data and stores it in a database. The server checks the integrity of the received data and checks for incomplete data.

[1887] Step 4:

[1888] The server uses AI algorithms to analyze the data it receives and identify key events and memories from the user's past. The server then organizes the information and prepares it for the next processing step.

[1889] Step 5:

[1890] The server analyzes the emotion data provided by the emotion engine, which detects the user's emotion in real time from voice tone and facial expressions, and sends the data to the server.

[1891] Step 6:

[1892] The server generates content to stimulate memories based on the analysis results and emotional data. If the user is in an anxious state, it selects and provides content related to past happy memories.

[1893] Step 7:

[1894] The device displays the generated content to the user, who can then review past photos and voice memos to stimulate their memories. For example, a user might view photos from a trip in 2010 and recall fond memories from that time.

[1895] Step 8:

[1896] The user inputs new events and daily events into the device, inputting the activities they performed that day and their feelings in text, and then clicks the send button.

[1897] Step 9:

[1898] The device receives new data, temporarily stores it, and then sends it to the server. The server receives the new data and records it in the database. This updates the user's daily life history.

[1899] Step 10:

[1900] The user sets the time to take daily medicine or schedule regular health checkups on the device. The user specifies the content and time of the reminder and clicks the setting button.

[1901] Step 11:

[1902] The server generates reminders based on the user's settings, and adjusts the content and timing of the reminders as needed, taking into account emotion data from the emotion engine.

[1903] Step 12:

[1904] The device will display a reminder notification to the user at the specified time. For example, it could display a notification saying "It's time to take your medicine," prompting the user to take action.

[1905] Step 13:

[1906] If a user cannot remember certain information during a conversation, they can ask for help from the device by entering a question using the device's question interface and clicking the send button.

[1907] Step 14:

[1908] The server receives the user's query and searches the database for the relevant information. The server responds quickly and identifies the information the user is looking for.

[1909] Step 15:

[1910] The server sends the identified information to the terminal, which displays it to the user. The user confirms the displayed information and continues the conversation.

[1911] Step 16:

[1912] The emotion engine runs even while the user is interacting with the system, continuously monitoring the user's emotional changes. Emotional data is fed back to the server in real time, and appropriate action is taken as necessary.

[1913] Through these steps, the system of the present invention can effectively support the memory of elderly people with dementia and improve the quality of their daily lives. By combining it with an emotion engine, it can provide more flexible and user-friendly support.

[1914] Example 2

[1915] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1916] To support the memory of elderly people with dementia, simply providing reminders and notifications is not enough. Systems that take the user's emotional state into account and appropriately stimulate past memories to improve the quality of daily life are needed. In particular, it is important to have a means to provide information when the user cannot remember specific information during a conversation, and a means to record the user's daily events and provide reminders based on that information.

[1917] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1918] In this invention, the server includes means for receiving and storing input data from a user, means for normalizing and analyzing the received data to identify past memories, means for receiving and analyzing emotional data to recognize the user's emotional state, means for generating content to stimulate memory based on the analysis and emotional analysis results, means for notifying the user of the generated content and reminders at an appropriate time, and means for providing information if the user cannot remember specific information during a conversation. This makes it possible to provide appropriate reminders and information based on the user's emotional state, thereby more effectively supporting memory in elderly people with dementia.

[1919] "Input data" refers to information such as voice, text, and photos provided by the user through the terminal.

[1920] "Normalization" refers to the process of converting received data into a format that is easier to analyze.

[1921] "Analysis" refers to the process of using AI algorithms to detect patterns and features in normalized data and identify past memories and events.

[1922] "Emotional data" refers to information about a user's emotional state obtained from their tone of voice and facial expressions.

[1923] "Emotion engine" refers to software or algorithms that analyze a user's tone of voice and facial expressions in real time to recognize their emotional state.

[1924] "Content" refers to information and media files (e.g., photos, videos, text) generated to stimulate a user's memory.

[1925] A "reminder" refers to a notification that reminds a user of a particular action or event.

[1926] "Notification" refers to a message from the system to inform the user of a reminder.

[1927] "Means" refers to a method or apparatus for accomplishing a particular function.

[1928] "Server" refers to a central device for data processing, analysis, and storage.

[1929] "Terminal" refers to the device (e.g., smartphone, tablet, or PC) that a user uses to interact with the system.

[1930] "Past memories" refer to important events and memories that the user has experienced in the past.

[1931] "Memory stimulation" refers to actions or content provided to evoke a user's past memories.

[1932] "Right time" refers to providing notifications and reminders at the most appropriate time based on the user's situation and emotional state.

[1933] This invention is an integrated memory support system for supporting the memory of elderly people with dementia. This system is composed of three main components: a server, a terminal, and a user, as well as an emotion engine. The functions of each component and the specific processing contents of the program are described in detail below.

[1934] Server Processing

[1935] The server is a central device that processes, analyzes, and stores various data. The server performs the following main processes:

[1936] 1. Data collection and analysis

[1937] The server receives data such as voice, text, and photos provided by the user through the device, and uses an AI algorithm such as TensorFlow.

[1938] The received data is normalized, and if it is audio data, it is converted into text data using a speech recognition engine such as Google Speech-to-Text.

[1939] The normalized data is analyzed to identify significant memories and events from the user's past.

[1940] 2. Emotional Recognition

[1941] The server receives emotional data from the emotion engine, which analyzes the user's voice tone and facial expressions, and determines their current emotional state. The emotion engine uses Microsoft Azure's emotion recognition API, among other things.

[1942] 3. Providing memory support

[1943] The server generates content to stimulate memories based on the analysis results and the user's emotional state. For example, if the user is anxious, the server generates content related to past happy memories.

[1944] Adjust the content and timing of reminders based on emotional data.

[1945] Terminal handling

[1946] A terminal is a device that functions as an interface between a user and a system. The main processes of a terminal are as follows:

[1947] 1. Data input

[1948] The device accepts voice and text input from the user. When the user records a diary entry, the device temporarily stores the data and prepares it for transmission to the server.

[1949] 2. Emotion Detection

[1950] The emotion engine detects emotions in real time from the user's voice and facial expressions, using IBM Watson's emotion analysis API.

[1951] 3. Reminders and notifications

[1952] Displaying reminders and notifications from the server to the user at appropriate times, for example, a reminder to "Don't forget to take your morning walk."

[1953] 4. Conversation support

[1954] If the user cannot remember a particular piece of information during a conversation, the system will provide that information to the user. For example, if the user asks, "Who is this person?", the system will search past memories and display the relevant information.

[1955] User operations

[1956] The user can perform the following operations through the terminal.

[1957] 1. Entering historical data

[1958] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[1959] 2. Recording everyday events

[1960] Users input daily events into the device using voice or text, and the input data is sent to the server and recorded.

[1961] 3. Check your reminders

[1962] The user checks the reminder from the device and performs the necessary tasks or events, for example, checking and performing the reminder to "take a deep breath."

[1963] 4. Emotional Feedback

[1964] When a user interacts with the device, the emotion engine detects the user's emotional state from facial expressions and tone of voice and sends this information to the server.

[1965] Specific examples

[1966] 1. Emotional memory stimulation

[1967] When a user uploads photos from past trips to their device, the photos are sent to the server, which analyzes the photos to identify the travel destination and time period. If the emotion engine detects the user's joy, the server highlights and displays happy memories related to that trip.

[1968] Prompt: "Analyze a travel photo. Identify when and where the photo was taken and provide the user with relevant memories."

[1969] 2. Recording new events and connecting emotions

[1970] When a user types "I went for a walk with a friend today" into the device, that data is sent to the server. If the emotion engine senses the user's joy, it also records that emotion data and uses it to remind the user later.

[1971] Prompt: "Analyze the user's diary and record the events and emotions of the day."

[1972] 3. Providing Emotion-Based Reminders

[1973] When the emotion engine detects that the user is frustrated, it sends this data to the server, which then generates a reminder to "take a deep breath" and sends it to the device, which then notifies the user of the reminder.

[1974] Prompt: "If the user is frustrated, generate a reminder and notify them."

[1975] By utilizing data from the emotion engine, this system will more effectively support the memory of elderly people with dementia and contribute to improving the quality of their daily lives.

[1976] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1977] Server Processing

[1978] Step 1: Receiving Data

[1979] The server receives data such as the user's voice, text, and photos from the device. For example, when a user records a voice diary entry, the voice file is transferred from the device to the server.

[1980] Input: Voice, text, and photo data sent from your device.

[1981] Output: Saving received data to storage.

[1982] Step 2: Normalize and preprocess the data

[1983] The server normalizes the received data and uses a speech recognition engine (e.g., Google Speech-to-Text) to convert the audio data into text data.

[1984] Input: Received data (voice, text, photos).

[1985] Output: Normalized data, speech data converted to text.

[1986] Step 3: Analyze the data

[1987] The server analyzes the normalized data using AI algorithms (e.g., TensorFlow) to identify important memories and events from the user's past.

[1988] Input: Normalized data.

[1989] Output: Identified memory or event information.

[1990] Step 4: Receiving emotion data

[1991] The server receives the user's emotional data (voice tone and facial expression analysis results) from the emotion engine.

[1992] Input: Emotion data sent from the emotion engine.

[1993] Output: Save emotion data to storage.

[1994] Step 5: Analyze the sentiment data

[1995] The server analyzes the received emotion data to understand the user's current emotional state, for example, determining whether the user is happy or sad from the tone of their voice.

[1996] Input: Emotion data.

[1997] Output: Parsed emotional state information.

[1998] Step 6: Generate memory support content

[1999] The server generates content to stimulate memories based on the analysis results and the user's emotional state. For example, if the user is anxious, the server generates content related to past happy memories.

[2000] Input: Identified memory or event information, analyzed emotional state information.

[2001] Output: The generated content.

[2002] Step 7: Adjust reminders and notifications

[2003] The server adjusts the content and timing of reminders based on emotional data. For example, if a user is feeling stressed, it generates a reminder to take a deep breath to relax.

[2004] Input: Parsed emotional state information.

[2005] Output: Coordinated reminders and notifications.

[2006] Step 8: Sending Data

[2007] The server sends the generated content and adjusted reminders to the device.

[2008] Input: Generated content, tailored reminders.

[2009] Output: Data sent to the terminal.

[2010] Terminal handling

[2011] Step 1: Data Input

[2012] The device accepts the user's voice and text input and prepares it to be sent to the server. For example, when a user records a diary entry, the device temporarily stores the voice data.

[2013] Input: User voice and text input.

[2014] Output: Temporarily saved data.

[2015] Step 2: Sending data

[2016] The terminal transmits the temporarily stored data to the server.

[2017] Input: Temporarily saved data.

[2018] Output: Data sent to the server.

[2019] Step 3: Emotion detection

[2020] The device uses an emotion engine to detect emotions from the user's voice and facial expressions in real time, for example, by analyzing voice and identifying emotions.

[2021] Input: Real-time user voice and facial expression data.

[2022] Output: Parsed emotion data.

[2023] Step 4: Sending emotion data

[2024] The terminal transmits the analysis results to the server.

[2025] Input: Parsed emotion data.

[2026] Output: Data sent to the server.

[2027] Step 5: Receive reminders

[2028] The device receives reminders and notifications from the server, such as a reminder to "don't forget to take a morning walk."

[2029] Input: Reminder and notification data from the server.

[2030] Output: Received data to the terminal.

[2031] Step 6: View Reminders

[2032] The device will display reminders and notifications to the user at appropriate times, using pop-up notifications and sound notifications.

[2033] Input: Received reminder, notification data.

[2034] Output: What is displayed to the user.

[2035] Step 7: Conversation support

[2036] If the user cannot remember a particular piece of information during a conversation, the device will provide that information to the user. For example, if the user asks, "Who is this person?", the device will search past memories and display relevant information.

[2037] Input: User question.

[2038] Output: Display of relevant information.

[2039] User operations

[2040] Step 1: Enter historical data

[2041] Users upload past photos and voice memos to their devices, and this data is sent to a server where it is stored for analysis.

[2042] Input: Past photos, voice memos.

[2043] Output: Upload data to the device.

[2044] Step 2: Record your daily events

[2045] Users input daily events into the device using voice or text, and the input data is sent to the server and stored as a record.

[2046] Input: Audio and text data of the user's daily life.

[2047] Output: Record data to terminal.

[2048] Step 3: Check your reminders

[2049] The user can check the reminders received from the device and perform the necessary tasks or events, for example, checking and performing the reminder to "take a deep breath."

[2050] Enter: reminders, notifications.

[2051] Output: The user's action.

[2052] Step 4: Emotional Feedback

[2053] When a user interacts with the device, the emotion engine detects the user's emotional state from facial expressions and tone of voice and sends this information to the server.

[2054] Input: User's facial expression, tone of voice.

[2055] Output: Data sent to the server.

[2056] (Application example 2)

[2057] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2058] The problem that this invention aims to solve is to prevent elderly people with dementia from experiencing difficulties in their daily lives due to memory confusion and forgetting important information, thereby improving their quality of life. Furthermore, by providing appropriate support according to the elderly's emotional state, it aims to reduce stress and anxiety, allowing them to live their daily lives with greater peace of mind.

[2059] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2060] In this invention, the server includes means for receiving input data from a user, means for analyzing the received data and identifying past memories, means for generating content for stimulating memory based on the analysis results, means for providing reminders and notifications to the user, and means for recognizing the user's emotions in real time using an emotion engine and reflecting that data in the analysis, thereby providing appropriate and timely support based on the user's emotional state, enabling memory improvement and stress reduction.

[2061] "Means for receiving input data from a user" is a general term for hardware and software that allows a user to input data such as voice, text, or images into the system and receive it.

[2062] "Means for analyzing received data and identifying past memories" refers to the algorithms and processes by which the system analyzes received data and identifies the user's past events and important memories.

[2063] The "means for generating content to stimulate memory based on the analysis results" is a mechanism for automatically generating content such as text, images, and audio to stimulate the user's memory based on identified past memories.

[2064] "Means for providing user reminders and notifications" are mechanisms for informing users of system-generated content and important information in a timely manner.

[2065] "Means of recognizing user emotions in real time using an emotion engine and reflecting that data in analysis" refers to technology that detects the user's emotional state in real time from their voice and facial expressions, and incorporates that information into the system's analysis process.

[2066] MODE FOR CARRYING OUT THE INVENTION

[2067] The present invention provides a smart shopping support system for supporting the shopping experience of elderly people with dementia, allowing them to live their lives with greater peace of mind. The following are embodiments of the invention.

[2068] Overall system structure

[2069] The system mainly consists of three components: a server, a terminal, and a user. The server is responsible for the main data processing and storage, and the terminal provides an interface for the user, through which the user interacts with the system.

[2070] Server Processing

[2071] The server does the following:

[2072] 1. Data collection and analysis

[2073] The server receives and analyzes data such as voice, text, and images provided by the device, and uses a generative AI model to identify the user's past memories and important events.

[2074] Emotion engine data will also be analyzed at the same time.

[2075] 2. Emotional Recognition

[2076] The server uses an emotion engine to recognize the user's emotions in real time and incorporates that data into its analysis, identifying emotions based on changes in voice and facial expressions.

[2077] 3. Providing memory support

[2078] The server generates content to stimulate memories according to the user's situation and emotional state. For example, if the user is feeling anxious, it selects content related to past happy memories.

[2079] 4. Reminders and notifications

[2080] The server generates reminders and notifications for users at appropriate times and sends them to their devices, helping them to remember their daily routines and important events.

[2081] Terminal handling

[2082] The terminal does the following:

[2083] 1. Data input

[2084] The device accepts the user's voice and text input and prepares it for transmission to the server. The input data is temporarily stored and managed for transmission to the server.

[2085] 2. Emotion Detection

[2086] The device uses an emotion engine to detect emotions from the user's voice and facial expressions in real time and transmits the data to the server.

[2087] 3. Reminders and notifications

[2088] The device will display reminders and notifications from the server to the user at the appropriate time, such as reminders to help users remember their daily routines.

[2089] 4. Conversation support

[2090] The device can provide users with information if they cannot remember specific information during a conversation, such as the name of a specific person or supplementary information about a past event.

[2091] User operations

[2092] The user does the following:

[2093] 1. Create a shopping list

[2094] The user adds an item to the shopping list, for example, by entering a voice command such as "add milk to the shopping list" into the terminal, which is then sent to the server.

[2095] 2. Recording everyday events

[2096] Users input daily events into the device by voice or text, for example, "Today I went for a walk with a friend."

[2097] 3. Emotional Feedback

[2098] When a user interacts with the device, the emotion engine detects the user's emotional state from their facial expressions and tone of voice, and sends that information to the server.

[2099] Specific examples

[2100] For example:

[2101] 1. Create a shopping list

[2102] When the user types "Add milk to the shopping list," data is sent from the device to the server, and milk is added to the shopping list.

[2103] 2. Emotional awareness and reminders

[2104] When a user types "I'm feeling happy today," the emotion engine recognizes the emotion as "happy" and sends it to the server, which then generates a reminder such as "Stay happy and enjoy shopping," which is then sent to the device.

[2105] Prompt Sentence Examples

[2106] Prompts to identify the user's emotional state:

[2107] Identify the emotional state based on user input. Input text: "Today was a very happy day!"

[2108] If the keyword "happy" is included, the emotional state will be set to "happy".

[2109] Input: "Today was a very happy day!", Identified emotion: "happy"

[2110] Prompt to record past events:

[2111] The user enters a past event. Event: "I went on a fun picnic with my family."

[2112] Emotion: Record the associated emotion "happy."

[2113] Input: "I went on a fun picnic with my family", emotion: "happy"

[2114] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2115] Step 1:

[2116] A user creates a shopping list. The user inputs a voice command into the terminal, such as "Please add milk to the shopping list." The terminal converts this voice command into text data and sends it to the server. The server receives this data and adds "milk" to the shopping list.

[2117] Input: User speaks "Add milk to my shopping list"

[2118] Data processing: Converting voice data into text data

[2119] Output: "Milk" is added to the shopping list

[2120] Step 2:

[2121] The user records a past event. The user types, "Today I went for a walk with a friend" into the device. The device converts this voice data into text data and sends it to the server. The server identifies the received text data and adds its contents to a list of past events.

[2122] Input: User's voice input: "I went for a walk with a friend today"

[2123] Data processing: Converting voice data into text data

[2124] Output: "I went for a walk with a friend today" is added to the list of past events

[2125] Step 3:

[2126] The device detects the user's emotions. The user speaks, "I'm feeling happy today." The device converts the voice data into text data and analyzes it with the emotion engine. The emotion engine recognizes "happy" from the text and sends the information to the server.

[2127] Input: User's voice input: "I'm feeling happy today"

[2128] Data processing: Converts voice data into text data and analyzes emotions with an emotion engine

[2129] Output: Emotional state "happy" is sent to the server

[2130] Step 4:

[2131] The server generates appropriate reminders based on the user's emotional state. For example, if the server receives the emotion data "happy," it generates a reminder such as "Stay happy and enjoy shopping." Using this generative AI model, the server inputs prompts to create reminders.

[2132] Input: Emotional state "happy"

[2133] Data processing: Generative AI models are used to generate reminders

[2134] Output: Reminder: "Stay happy and enjoy your shopping"

[2135] Step 5:

[2136] The device notifies the user of the reminder. The device receives the reminder "Keep your shopping fun and enjoy yourself" sent from the server and displays it as a notification on the user's device.

[2137] Input: Reminder sent from the server

[2138] Data processing: Convert reminders into notifications

[2139] Output: The reminder appears on the user's device

[2140] Through these steps, the system supports the shopping experience of elderly people with dementia, providing users with appropriate feedback based on their emotional state, allowing them to carry out their daily shopping with greater peace of mind.

[2141] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2142] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2143] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2144] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2145] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2146] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2147] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2148] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2149] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2150] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2151] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2152] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2153] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2154] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2155] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2156] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2157] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2158] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2159] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2160] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2161] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2162] The following is further disclosed regarding the above embodiment.

[2163] (Claim 1)

[2164] means for receiving input data from a user;

[2165] means for analyzing the received data and identifying past memories;

[2166] means for generating content for stimulating memory based on the analysis results;

[2167] a means of providing reminders and notifications to the user;

[2168] A system including:

[2169] (Claim 2)

[2170] 10. The system of claim 1, further comprising means for a user to record daily occurrences.

[2171] (Claim 3)

[2172] 10. The system of claim 1, further comprising means for providing particular information if the user cannot remember the information during the conversation.

[2173] (Claim 4)

[2174] 10. The system of claim 1, further comprising means for stimulating memory using historical photographs and audio data.

[2175] (Claim 5)

[2176] 10. The system of claim 1, further comprising means for reminding the user of a routine or event at a specific time.

[2177] "Example 1"

[2178] (Claim 1)

[2179] means for receiving input data from a user;

[2180] means for analyzing the received data and identifying past memories;

[2181] means for generating content for stimulating memory based on the analysis results;

[2182] a means of providing reminders and notifications to the user;

[2183] means for transmitting user input data to a server via a network;

[2184] a means for analyzing the data received by the server using a plurality of analytical techniques;

[2185] A means of recording user behavior patterns and routines in real time based on the analysis results;

[2186] A system including:

[2187] (Claim 2)

[2188] 10. The system of claim 1, further comprising means for a user to record daily occurrences.

[2189] (Claim 3)

[2190] 10. The system of claim 1, further comprising means for providing particular information if the user cannot remember the information during the conversation.

[2191] "Application Example 1"

[2192] (Claim 1)

[2193] means for receiving input data from a user;

[2194] means for analyzing the received data and identifying past memories;

[2195] A means for displaying information about a specific person using a facial recognition function;

[2196] A means of recording daily events using voice recognition;

[2197] means for generating content for stimulating memory based on the analysis results;

[2198] a means for displaying past events based on the user's location and time;

[2199] a means of providing reminders and notifications to the user;

[2200] A system including:

[2201] (Claim 2)

[2202] 10. The system of claim 1, further comprising means for providing visual and audio reminders via the smart device.

[2203] (Claim 3)

[2204] 10. The system of claim 1, further comprising means for providing specific information during a conversation using a voice output function.

[2205] "Example 2: Combining Emotion Engines"

[2206] (Claim 1)

[2207] means for receiving and storing input data from a user;

[2208] a means for normalizing and analyzing the received data to identify past memories;

[2209] means for receiving and analyzing emotional data to recognize an emotional state of a user;

[2210] means for generating content for stimulating memory based on the analysis and sentiment analysis results;

[2211] A means of notifying users of generated content and reminders in a timely manner;

[2212] a means for providing specific information if the user cannot remember it during the conversation;

[2213] A system including:

[2214] (Claim 2)

[2215] 10. The system of claim 1, further comprising means for a user to record daily occurrences.

[2216] (Claim 3)

[2217] 10. The system of claim 1, further comprising means for adjusting reminders and notifications based on user emotions.

[2218] "Application example 2 when combining emotion engines"

[2219] (Claim 1)

[2220] means for receiving input data from a user;

[2221] means for analyzing the received data and identifying past memories;

[2222] means for generating content for stimulating memory based on the analysis results;

[2223] a means of providing reminders and notifications to the user;

[2224] A means of recognizing user emotions in real time using an emotion engine and incorporating that data into analysis;

[2225] A system including:

[2226] (Claim 2)

[2227] 10. The system of claim 1, further comprising means for a user to record daily occurrences.

[2228] (Claim 3)

[2229] 10. The system of claim 1, further comprising means for providing particular information if the user cannot remember the information during the conversation. [Explanation of symbols]

[2230] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving input data from a user; means for analyzing the received data and identifying past memories; means for generating content for stimulating memory based on the analysis results; a means of providing reminders and notifications to the user; A system including:

2. 10. The system of claim 1, further comprising means for a user to record daily occurrences.

3. 10. The system of claim 1, further comprising means for providing specific information if the user cannot remember that information during the conversation.

4. 10. The system of claim 1, further comprising means for stimulating memory using historical photographs and audio data.

5. 10. The system of claim 1, further comprising means for reminding the user of a routine or event at a specific time.

Images