System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting cognitive patient data; means for utilizing a generative AI model to analyze the data; means for generating memory replay content based on the analysis results; means for transmitting the memory replay content to a smartphone; means for displaying the memory replay content on the smartphone; means for recommending an optimal action; and means for notifying the smartphone of the action recommendation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Dementia patients often experience a significant burden in their daily lives due to memory loss and confusion. Family members and caregivers also face challenges in providing appropriate support, which can be stressful. In these circumstances, there is a need for methods to support patients' daily lives and reduce the burden on family members and caregivers. However, current systems lack customized support tailored to symptoms and technology to instantly recall memories, and do not adequately provide comprehensive support to enable patients to live with peace of mind. [Means for solving the problem]

[0005] The present invention is a system that generates personalized memory reproduction content by collecting data from dementia patients and analyzing it with a generative AI model. The system includes a means for transmitting analysis results to a smartphone and displaying the memory reproduction content. It also includes a means for recommending optimal actions based on the patient's condition and notifying the smartphone of the recommendations. Specifically, the system includes a means for collecting patient data, a means for analyzing the data using a generative AI model, a means for generating memory reproduction content based on the analysis results, a means for transmitting and displaying the memory reproduction content on the smartphone, and a means for recommending and notifying optimal actions. In this way, by supporting the memory of dementia patients and recommending appropriate actions, the system improves the patient's quality of life and reduces the burden on family and caregivers.

[0006] A "dementia patient" refers to a patient whose memory or cognitive function has declined, causing problems in daily life.

[0007] "Data" refers to information related to the dementia patient and their surroundings, including photos, videos, notes, conversation history, etc.

[0008] A "generative AI model" refers to an artificial intelligence model that analyzes collected data and generates output for a specific purpose.

[0009] "Memory reproduction content" refers to content generated to reproduce a patient's forgotten memories based on data analyzed by a generative AI model.

[0010] A "smartphone" refers to a portable communication device that can connect to the Internet and run applications.

[0011] "Optimal action" refers to the specific action that the generative AI model recommends based on the patient's data and current situation.

[0012] "Recommendation" means making suggestions or recommendations regarding actions or options.

[0013] "Notification" refers to the operation of displaying a message on a device such as a smartphone to appeal to the user's awareness. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention is a system for supporting the memory of dementia patients and recommending optimal actions. Specifically, the system collects data on dementia patients, analyzes it using a generative AI model, and generates memory-reconstruction content based on the results, which is then displayed on a smartphone. Furthermore, the system recommends optimal actions based on the patient's current condition and notifies the smartphone.

[0036] System Configuration

[0037] The system consists of the following main components:

[0038] 1. Data Collection Methods

[0039] 2. Generative AI Models

[0040] 3. Memory Recreation Content Generation Method

[0041] 4. Smartphone display and notification methods

[0042] Data collection methods

[0043] Users (patients or their family members) use a smartphone app to upload data such as past photos, videos, notes, and conversation history.

[0044] Generative AI Models

[0045] The server receives the data sent by the user and analyzes it using a generative AI model, which automatically generates photo captions and related stories based on the data.

[0046] Memory reproduction content generation means

[0047] The server creates memory-recreating content based on the analysis results, including adding captions related to photos and generating explanatory text about past episodes.

[0048] Smartphone display and notification methods

[0049] The device (smartphone) displays the memory-recreating content sent from the server to the user. The server then recommends optimal actions and sends a notification to the device. The device then prompts the user to take action via push notifications or alert messages.

[0050] Specific examples

[0051] A user opens a smartphone app and uploads photos from a previous trip, which the device then sends to the server.

[0052] The server analyzes the image file and its metadata and generates a caption, such as "This photo was taken in front of the Eiffel Tower during a trip to France in the summer of 2019." The server then sends the memory-recreating content, including the caption, to the device, which displays it to the user, allowing them to relive their memories by looking at the photo.

[0053] Also, if the server generates a recommendation that "It's a good time to take a walk," the device will send a notification saying, "It's time for a walk now. Let's go to the nearest park." By carrying out the suggested action, the user who receives this recommendation can make their daily life more fulfilling.

[0054] The system will help dementia patients relive their past memories and provide a sense of security, while also making it easier for family and caregivers to provide optimal support, improving overall quality of life.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] Users can launch a smartphone application and upload data such as past photos, videos, notes, conversation history, etc. Users can also select photos and enter captions and tags.

[0058] Step 2:

[0059] The device processes the uploaded data and prepares it for transmission to the server. Specifically, the device compresses the image files and adds appropriate metadata (e.g., date and time of the photo, location).

[0060] Step 3:

[0061] The device sends the prepared data to the server, which then sends a POST request to the server with the image and its metadata to the specified API endpoint.

[0062] Step 4:

[0063] The server stores the received data in storage. Specifically, the server stores the image file path and metadata in a database, and stores the image itself in cloud storage.

[0064] Step 5:

[0065] The server analyzes the stored data using a generative AI model. Specifically, the server inputs an image into the AI ​​model and generates an episode related to the image (e.g., the background of the photo, information about the people in the photo).

[0066] Step 6:

[0067] The server stores the analysis results in a database. Specifically, the server stores the analysis results (e.g., "This is a photo I took with my family at the beach last summer") in the database.

[0068] Step 7:

[0069] The server generates memory-recreating content based on the analysis results. Specifically, the server adds captions to images and generates related episodes as text.

[0070] Step 8:

[0071] The server sends the generated memory reproduction content to the device. Specifically, the server sends the generated captions along with the images in JSON format to the device.

[0072] Step 9:

[0073] The device displays the memory reproduction content received from the server to the user. Specifically, the device opens an image viewer and displays the image and its caption on the screen.

[0074] Step 10:

[0075] The server uses a generative AI model to recommend optimal actions based on the patient's condition and past data analysis results. Specifically, the server generates recommendations such as "Taking a walk at 3 p.m."

[0076] Step 11:

[0077] The server sends the generated action recommendation to the terminal. Specifically, the server sends the action recommendation to the terminal as a notification message.

[0078] Step 12:

[0079] The device notifies the user of recommended activities. Specifically, the device displays a push notification with the message, "It's time for a walk. Let's go to your favorite park."

[0080] Example 1

[0081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0082] Dementia patients have difficulty retaining their memories and are prone to confusion in their daily lives. Caregivers and families also have limited information to provide the necessary support. Furthermore, a lack of appropriate behavioral instructions based on the patient's current condition can lead to a decline in quality of life. There is a need to address these issues.

[0083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0084] In this invention, the server includes: means for a user to upload information such as past photos, videos, notes, and conversation history using a smartphone app; means for receiving and saving the information sent from the smartphone app; means for analyzing the saved information and generating photo captions and related episodes using a generative AI model; means for generating memory reproduction content based on the analysis results; means for sending the memory reproduction content to the smartphone; means for displaying the memory reproduction content on the smartphone; means for recommending optimal actions based on the patient's current condition; and means for notifying the smartphone of the recommended actions. This makes it easier for dementia patients to recall their memories, making it easier for caregivers and family members to provide the necessary support and providing appropriate instructions for actions based on the patient's current condition.

[0085] "User" refers to a person with dementia or their family member who uses the system.

[0086] "Smartphone app" refers to application software that runs on a smartphone and allows users to upload past information and receive memory recall content and behavioral recommendations.

[0087] "Information" refers to data uploaded by users, such as past photos, videos, notes, and conversation history.

[0088] "Server" refers to a computer system that receives, stores, and analyzes information sent by users, generates captions and episodes using generative AI models, and transmits memory reproduction content and behavioral recommendations to smartphones.

[0089] A "generative AI model" refers to an artificial intelligence model that analyzes information sent by users and automatically generates photo captions and related stories.

[0090] "Caption" refers to explanatory text associated with a photograph.

[0091] An "episode" is a descriptive text about a past event related to a photo or video.

[0092] "Memory recall content" refers to content that includes the analyzed photo captions and related episode text.

[0093] "Action recommendation" refers to a recommendation of the optimal action that is generated based on the patient's current condition.

[0094] "Notifications" refers to alerts sent to your smartphone with recommended actions or other important information.

[0095] This invention is a system for supporting memory and recommending optimal actions for dementia patients. The system consists of the following main components:

[0096] 1. Data Collection Methods

[0097] 2. Generative AI Models

[0098] 3. Memory Recreation Content Generation Method

[0099] 4. Smartphone display and notification methods

[0100] Data collection methods

[0101] Users upload information such as past photos, videos, memos, and conversation histories using a smartphone app. The smartphone app then sends the files selected by the user to the server via an HTTP request.

[0102] Generative AI Models

[0103] The server receives and stores information sent from the smartphone app. The stored information is then analyzed using a generative AI model, such as the latest natural language processing AI, such as GPT-4.

[0104] The following prompts are used during parsing:

[0105] text

[0106] This is a user-uploaded photo. Please provide information about the photo:

[0107] [Photo metadata (date, time, location, etc.)]

[0108] Based on this prompt, the generative AI model automatically generates captions for the photos and related stories.

[0109] Memory reproduction content generation means

[0110] The server generates memory-recreating content based on the results of analysis by the generative AI model. This content includes adding captions related to photos and generating explanatory text about past episodes. The generated content is combined in HTML or JSON format and organized in a format that is easy for users to access.

[0111] Smartphone display and notification methods

[0112] The device (smartphone) displays the memory replay content sent from the server to the user. By viewing this content, the user can relive past memories. The server also recommends optimal actions based on the patient's current condition and sends a notification of this recommendation to the device. The device then communicates this recommendation to the user in the form of a push notification or an alert message.

[0113] Specific examples

[0114] Consider a case where a user wants to upload photos from a previous trip. The user opens a smartphone app, selects a travel photo, and taps the upload button. This causes the device to send data including the photo to the server.

[0115] The server analyzes the image file and its metadata and generates a caption, such as "This photo was taken in front of the Eiffel Tower during a trip to France in the summer of 2019." Based on this, memory-recreating content is created and sent to the device. The device then displays this content to the user, allowing them to relive their past memories.

[0116] Furthermore, if the server generates a recommendation that "It's a good time to take a walk," the device will notify the user, "It's time for a walk now. Let's go to the nearest park." By carrying out the suggested action, the user who receives this recommendation can make their daily life more fulfilling.

[0117] This system will help dementia patients relive their past memories and provide a sense of security, while also making it easier for family and caregivers to provide optimal support, improving their overall quality of life.

[0118] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0119] Step 1:

[0120] A user uses a smartphone app to upload information such as past photos, videos, notes, and conversation history. Specifically, the user opens the smartphone app and taps the upload button to launch a file browser. When the user selects the files they want to upload, the device sends the selected files to the server via an HTTP request.

[0121] Input: Files such as past photos, videos, notes, and conversation history

[0122] Output: HTTP request from the terminal to the server

[0123] Step 2:

[0124] The server receives and stores data sent from the device. The server processes the received HTTP request and reads the sent file. The read data is stored in cloud storage or a database.

[0125] Input: File data sent from the terminal

[0126] Output: Saved file data

[0127] Step 3:

[0128] The server analyzes the stored data and generates photo captions and related episodes using a generative AI model. The server checks the file type and metadata and generates prompts for analysis. When the generated prompts are input into the generative AI model, the AI ​​model generates photo captions and episodes.

[0129] Input: Saved file data, prompt text

[0130] Output: Generated photo captions and related episodes

[0131] Step 4:

[0132] The server generates memory reproduction content based on the analysis results. The server integrates the analyzed captions and episodes into content in HTML or JSON format. This content is organized in a format that is easy for users to view.

[0133] Input: Generated photo captions, related episodes

[0134] Output: Memory-reproduced content (HTML or JSON format)

[0135] Step 5:

[0136] The device receives the memory reproduction content sent from the server and displays it to the user. The device receives the HTTP response and displays the content in the response within the app. By viewing this, the user can relive past memories.

[0137] Input: Memory-reproduced content sent from the server

[0138] Output: Memory-reproduced content displayed within the app

[0139] Step 6:

[0140] The server recommends the optimal action based on the patient's current condition and sends a notification to the device. The server analyzes past data and the patient's current condition to generate a recommendation such as "It's a good idea to take a walk at this time of day." The generated recommendation is sent to the device via an HTTP request.

[0141] Input: Current patient status, historical data

[0142] Output: Generated action recommendations

[0143] Step 7:

[0144] The device receives the recommended actions sent from the server and notifies the user. The device receives the HTTP response and displays the content to the user as a push notification or alert message. The user can check the notification and carry out the suggested actions to enrich their daily life.

[0145] Input: Action recommendations sent from the server

[0146] Output: Push notification or alert message to the user

[0147] (Application example 1)

[0148] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0149] Due to memory decline, dementia patients often forget past events and the locations of their daily activities. They may also get lost or find themselves in dangerous situations when out and about, which reduces the patient's quality of life and increases the anxiety of their families and caregivers. Conventional systems are unable to adequately address these issues, making emergency response and ensuring patient safety particularly challenging.

[0150] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0151] In this invention, the server includes means for collecting data on dementia patients, means for using a generative AI model to analyze the data, means for generating memory reproduction content based on the analysis results, means for transmitting the memory reproduction content to a smartphone, means for displaying the memory reproduction content on the smartphone, means for recommending optimal actions, means for notifying the smartphone of the recommended actions, means for collecting location information to track the patient's location, means for monitoring the surrounding environment using a camera and a microphone, means for issuing an emergency alert when a dangerous situation is detected, and means for sending notifications to caregivers in real time. This makes it possible to support the memory of dementia patients, recommend optimal actions, and ensure the safety of the patients.

[0152] A "dementia patient" is someone whose cognitive function has declined, resulting in memory loss, impaired judgment, and difficulty in daily life.

[0153] "Data" refers to information such as photos, videos, notes, and conversation histories related to dementia patients.

[0154] A "generative AI model" is an artificial intelligence model that analyzes collected data and generates the necessary information.

[0155] "Memory reproduction content" refers to content such as captions and episode texts that are created based on data analyzed by a generative AI model to reproduce a patient's memories.

[0156] A "smartphone" is a mobile device that can use a wide variety of applications in addition to the functions of a mobile phone.

[0157] "Action recommendations" are optimal actions suggested based on the patient's current situation and past data.

[0158] "Location information collection means" refers to a means for tracking the patient's current location using GPS or other means.

[0159] An "environmental monitoring means" is a means for monitoring the environment surrounding the patient using a camera or microphone.

[0160] An "emergency alert means" is a means for issuing a warning when a dangerous situation is detected.

[0161] "Caregivers" are family members or professionals who play a role in helping people with dementia on a day-to-day basis.

[0162] "Real-time" refers to processing occurring almost at the same time as the data is generated.

[0163] This invention is a system for supporting the memory of dementia patients and recommending optimal actions. Specifically, the system collects the patient's past data and analyzes it using a generative AI model to generate memory reproduction content and display it on a smartphone. It also has the function of recommending optimal actions based on the patient's current condition and notifying the smartphone.

[0164] The server first provides a means to collect data on dementia patients. This data, such as past photos, videos, notes, and conversation histories, is uploaded by patients and their families using a smartphone app. The server receives this data and analyzes it using a generative AI model. The generative AI model automatically generates photo captions and text for related episodes based on the collected data.

[0165] The server then generates memory-recreating content based on the analysis results. The generated memory-recreating content is sent to a smartphone, where it is displayed on the device. For example, if a past travel photo is uploaded, the generative AI model analyzes the photo and generates a caption such as, "This photo was taken at a tourist spot during a trip in the summer of 2019." The device then displays the memory-recreating content, including the caption, to the user, allowing them to view the photo and relive their memory.

[0166] Furthermore, the server has the function of recommending optimal actions. Based on past data and the patient's current condition, the generative AI model considers the current time of day and the patient's activity patterns to suggest appropriate actions. For example, a recommendation such as "It's a good idea to take a walk at this time of day" may be generated. This recommendation is sent as a push notification to the smartphone, displaying a message saying, "It's time for a walk now. Let's go to the nearest park."

[0167] The system also includes a location information collection means for tracking the patient's location, an environmental monitoring means for monitoring the patient's surroundings using a camera and microphone, an emergency alert means for issuing a warning when a dangerous situation is detected, and a means for sending a notification to a caregiver in real time. For example, if a patient goes out alone to a park, the generative AI model will track the patient's location using GPS and monitor the surrounding environment using a camera and microphone. Based on this information, the generative AI model will detect a dangerous situation and send a real-time notification to the caregiver saying, "Emergency: Patient may be in danger."

[0168] Examples of prompt sentences include the following:

[0169] "Analyze the photos and audio data sent by the user and generate a description of the environment and memory-recreating content. For example, if the sent photo was taken in a park, generate a description such as, 'This is a park near my house. I've been there before with my family.'"

[0170] This makes it easier for people with dementia to relive their past memories and feel more secure, and it also makes it easier for family and caregivers to ensure the patient's safety, improving their overall quality of life.

[0171] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0172] Step 1:

[0173] A user uses a smartphone app to upload data such as past photos, videos, notes, and conversation history. The input is the data uploaded by the user, and the output is the data sent to the server.

[0174] Step 2:

[0175] The server receives the data submitted by the user. The input is the data submitted by the user, and the output is the data to be fed into the generative AI model. The server converts this into a format suitable for analysis.

[0176] Step 3:

[0177] The server analyzes the received data using a generative AI model. The input is the received data, and the output is the analysis result. The generative AI model analyzes the content and metadata of the photo and generates captions and episode text.

[0178] Step 4:

[0179] The server generates memory-recreation content based on the analysis results. The input is the analysis results, and the output is the memory-recreation content. This content includes the generated captions and episode text.

[0180] Step 5:

[0181] The server transmits the generated memory reproduction content to the smartphone. The input is the memory reproduction content, and the output is the data transmitted to the smartphone.

[0182] Step 6:

[0183] The terminal displays the memory reproduction content sent from the server to the user. The input is the memory reproduction content, and the output is the screen displayed to the user. The terminal visually displays captions and episode text.

[0184] Step 7:

[0185] The server recommends the optimal action. The input is historical data and the patient's current condition, and the output is a recommended action. The generative AI model takes into account the current time of day and the patient's activity patterns to suggest appropriate actions.

[0186] Step 8:

[0187] The server notifies the smartphone of the recommended actions. The input is the recommended actions, and the output is a notification sent to the smartphone. The smartphone then suggests appropriate actions to the user through a push notification.

[0188] Step 9:

[0189] The terminal obtains real-time location information using a location information collection means that tracks the patient's location information. The input is GPS data, and the output is the patient's current location.

[0190] Step 10:

[0191] The device monitors the surrounding environment using a camera and microphone. The input is camera video and audio data, and the output is environmental information. The camera and microphone periodically collect data and provide it for analysis.

[0192] Step 11:

[0193] The server uses an emergency alert mechanism to issue a warning when a dangerous situation is detected. The input is location information and environmental information, and the output is a warning message. The generative AI model analyzes this data and generates a warning if a dangerous situation is detected.

[0194] Step 12:

[0195] The server sends real-time notifications to the caregiver: the input is the warning message and the output is the notification sent to the caregiver, allowing the caregiver to respond immediately.

[0196] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0197] This invention is a system for supporting the memory of dementia patients and recommending optimal actions. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide more personalized support. Specifically, this system has the function of analyzing the user's emotions and adjusting memory recall content and optimal action recommendations based on that.

[0198] System Configuration

[0199] The system consists of the following main components:

[0200] 1. Data Collection Methods

[0201] 2. Generative AI Models

[0202] 3. Memory Recreation Content Generation Method

[0203] 4. Smartphone display and notification methods

[0204] 5. Emotion Engine

[0205] Data collection methods

[0206] Users (patients or their family members) use a smartphone app to upload data such as past photos, videos, notes, and conversation history.

[0207] Generative AI Models

[0208] The server receives the data sent by the user and analyzes it using a generative AI model, which automatically generates photo captions and related stories based on the data.

[0209] Memory reproduction content generation means

[0210] The server creates memory-recreating content based on the analysis results, including adding captions related to photos and generating explanatory text about past episodes.

[0211] Smartphone display and notification methods

[0212] The device (smartphone) displays the memory-recreating content sent from the server to the user. The server then recommends optimal actions and sends a notification to the device. The device then prompts the user to take action via push notifications or alert messages.

[0213] Emotion Engine

[0214] The device has an emotion engine that recognizes the user's emotions. This emotion engine analyzes emotions based on the user's facial expressions, voice, and text input. The analyzed emotion information is sent to the server and reflected in memory reproduction content and behavioral recommendations.

[0215] Specific examples

[0216] A user opens a smartphone app and uploads photos from a previous trip, which the device then sends to the server.

[0217] The server analyzes the image file and its metadata and generates a caption, such as "This photo was taken in front of the Eiffel Tower during a trip to France in the summer of 2019." The server then sends the memory-recreating content, including the caption, to the device, which displays it to the user, allowing them to relive their memories by looking at the photo.

[0218] The device also recognizes emotions such as "happiness" and "surprise" through facial and voice analysis of the user. Based on this, the server generates a recommendation such as "It's time for a walk. Let's go to the nearest park" and sends it to the device. This recommendation is displayed at an appropriate time depending on the emotion.

[0219] For example, if the system recognizes that the user is feeling stressed, it will give a recommendation such as, "Why not take a break and read your favorite book?"

[0220] This system makes it easier for dementia patients to relive their past memories, providing a sense of security. It also recommends appropriate actions based on the user's emotions, further improving their quality of life. It also makes it easier for family members and caregivers to provide emotional support, enhancing overall care.

[0221] The processing flow will be explained below.

[0222] Step 1:

[0223] Users can launch a smartphone application and upload data such as past photos, videos, notes, conversation history, etc. Users can also select photos and enter captions and tags.

[0224] Step 2:

[0225] The device processes the uploaded data and prepares it for transmission to the server. Specifically, the device compresses the image files and adds appropriate metadata (e.g., date and time of the photo, location).

[0226] Step 3:

[0227] The device sends the prepared data to the server, which then sends a POST request to the server with the image and its metadata to the specified API endpoint.

[0228] Step 4:

[0229] The server stores the received data in storage. Specifically, the server stores the image file path and metadata in a database, and stores the image itself in cloud storage.

[0230] Step 5:

[0231] The server analyzes the stored data using a generative AI model. Specifically, the server inputs an image into the AI ​​model and generates an episode related to the image (e.g., the background of the photo, information about the people in the photo).

[0232] Step 6:

[0233] The server stores the analysis results in a database. Specifically, the server stores the analysis results (e.g., "This is a photo I took with my family at the beach last summer") in the database.

[0234] Step 7:

[0235] The server generates memory-recreating content based on the analysis results. Specifically, the server adds captions to images and generates related episodes as text.

[0236] Step 8:

[0237] The server sends the generated memory reproduction content to the device. Specifically, the server sends the generated captions along with the images in JSON format to the device.

[0238] Step 9:

[0239] The device displays the memory reproduction content received from the server to the user. Specifically, the device opens an image viewer and displays the image and its caption on the screen.

[0240] Step 10:

[0241] The device collects facial and voice data through a camera and microphone to recognize the user's emotions. Specifically, the device collects emotional data in real time while the user is browsing photos and stories.

[0242] Step 11:

[0243] The device analyzes the collected emotional data and identifies the user's current emotional state. Specifically, the emotion engine determines emotions such as "happiness," "surprise," and "sadness" from the user's facial expressions and voice.

[0244] Step 12:

[0245] The device sends the analyzed emotion data to the server by sending a POST request to the server's API endpoint in JSON format.

[0246] Step 13:

[0247] The server receives the emotion data and reflects it in memory-recreating content and optimal action recommendations. Specifically, it adjusts the content and recommendations to be displayed based on the emotion data.

[0248] Step 14:

[0249] The server uses a generative AI model to recommend optimal actions based on the patient's condition, past data analysis results, and emotional data. Specifically, it generates a recommendation such as "A walk is effective at the current time," but if the user is feeling stressed, it generates a different recommendation such as "Why don't you take a break and read your favorite book?"

[0250] Step 15:

[0251] The server sends the generated action recommendation to the terminal. Specifically, the server sends the action recommendation to the terminal as a notification message.

[0252] Step 16:

[0253] The device notifies the user of recommended actions. Specifically, it encourages the user to take action by means of push notifications or alert messages. For example, the device may notify the user, "It's time for a walk. Let's go to the nearest park."

[0254] Through these processing steps, this system supports the memory of dementia patients and improves their quality of life by recommending optimal actions based on their emotions. It also makes it easier for family members and caregivers to understand the patient's emotional state and provide more appropriate support.

[0255] Example 2

[0256] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0257] Dementia patients have difficulty retaining past memories and often experience stress in their daily lives. Furthermore, there are limited ways to encourage dementia patients to behave appropriately, placing a heavy burden on their families and caregivers. The problem this invention aims to solve is to improve the quality of life of dementia patients by supporting their memory recall and recommending optimal actions based on their emotions.

[0258] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user data, means for using a generative AI model to analyze the user data, means for generating memory reproduction content based on the analysis results, means for transmitting the memory reproduction content to a terminal, means for displaying the memory reproduction content on the terminal, means for analyzing the user's emotions, means for recommending optimal actions based on the user's emotional information, and means for notifying the terminal of the recommended actions. This allows dementia patients to easily reproduce past memories and receive appropriate recommended actions based on their emotions.

[0259] "User" refers to anyone who uses the system, including a dementia patient or their family or caregiver.

[0260] "Data collection means" refers to the ability of users to upload information such as photos, videos, notes, and conversation history using smartphone apps.

[0261] A "generative AI model" is an artificial intelligence technology that analyzes data collected from users and automatically generates photo captions and related stories.

[0262] "Memory reproduction content" is content that includes photo captions and text descriptions created based on the results of analysis by a generative AI model.

[0263] "Terminal" refers to a display device such as a smartphone or tablet used by a user.

[0264] "Emotion analysis means" is a technology that has the function of analyzing a user's facial expressions, voice, character input, etc., and recognizing the user's emotions.

[0265] "Behavioral recommendations" refer to specific actions suggested to users based on analyzed emotional information and past data.

[0266] The "notification means" is a function that sends the generated memory reproduction content and behavioral recommendations to the terminal as a push notification or an alert message and displays them to the user.

[0267] This invention is a system that supports memory for dementia patients and recommends optimal actions. By combining this system with an emotion engine that recognizes the user's emotions, it is possible to provide more personalized support.

[0268] System Configuration

[0269] The system consists of the following main components:

[0270] 1. Data Collection Methods

[0271] 2. Generative AI Models

[0272] 3. Memory Recreation Content Generation Method

[0273] 4. Terminal display and notification methods

[0274] 5. Emotion Engine

[0275] Data collection methods

[0276] Users use a smartphone app to upload data such as past photos, videos, memos, and conversation histories. For example, they can upload photos of a family trip or diary notes to the app. The device then sends this data to a server.

[0277] Generative AI Models

[0278] The server uses a generative AI model to analyze the data received from the device. This generative AI model is an advanced natural language processing model such as GPT-4. The server analyzes the metadata of the photo or video (e.g., date and time of the photo, location) and generates a caption based on that. For example, it generates a caption such as, "This photo was taken in front of the Eiffel Tower in Paris, France, in the summer of 2019."

[0279] Memory reproduction content generation means

[0280] The server generates memory-recreating content based on the analysis results. This content includes captions related to the photos and explanatory text about past episodes. For example, a photo of a trip to France might include a description such as, "On a beautiful summer day, my family and I visited the Eiffel Tower and then took a walk along the Seine."

[0281] Terminal display and notification means

[0282] The device displays the memory reproduction content sent from the server to the user. For example, the app's notification function can be used to notify the user, "A new memory has been created!" When the user clicks on the notification, the generated memory reproduction content is displayed within the app.

[0283] Emotion Engine

[0284] The device has an emotion engine that analyzes the user's facial expressions, voice, and text input to recognize emotions. For example, if the user smiles at a displayed photo, the device recognizes "joy." If the user writes "I had a lot of fun" when entering a comment, the device analyzes it as a "positive emotion."

[0285] Action recommendation generation and notification

[0286] The server generates optimal action recommendations based on the emotional information sent from the emotion engine. For example, if the user expresses the emotion "joy," the server generates a recommendation such as, "You seem to be in a good mood right now, so why not enjoy some tea at a nearby cafe?" The server sends this recommendation to the device. The device notifies the user of this recommendation, displaying, for example, "It's time for a walk now. Let's go to a nearby park."

[0287] Specific examples

[0288] Prompt Sentence Examples

[0289] Upload past travel photos and the system will analyze your emotions and recommend the best course of action.

[0290] Example

[0291] A user opens the smartphone app, selects a photo from their trip to France in 2019, and presses the upload button. The device sends this photo to the server. The server analyzes the received image and generates a caption such as, "This photo was taken in front of the Eiffel Tower during my trip to France in the summer of 2019." The server creates memory-recreating content including this caption and sends it to the device. The device notifies the user via push notification, "A new memory has been created!" When the user clicks the notification, the generated content is displayed within the app. The device analyzes the user's facial expression with an emotion engine and recognizes "joy." Based on this emotional information, the server generates an action recommendation such as, "It's time for a walk now. Let's go to the nearest park." The device notifies the user of this recommendation.

[0292] This system allows dementia patients to easily recall past memories and receive appropriate behavioral recommendations based on their emotions, enabling them to enrich their daily lives.

[0293] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0294] Step 1:

[0295] The user opens the smartphone app, selects past photos, videos, notes, and conversation history (input data), and presses the upload button. The device then sends this data to the server (specific operation of data transmission).

[0296] Step 2:

[0297] The server uses a generative AI model to analyze the data received from the device (input data). It uses a generative AI model (e.g., GPT-4) to check the metadata of the photo or video (e.g., date and time of the photo, location) and generates a caption (specific operation of data analysis and caption generation). For example, it generates a caption (output data) such as, "This photo was taken in front of the Eiffel Tower in Paris, France, in the summer of 2019."

[0298] Step 3:

[0299] The server creates memory-recreating content based on the generated captions (input data). This content includes captions related to the photos and explanatory text about past episodes. For example, to a photo of a trip to France, it adds the description (output data) "On a beautiful summer day, after visiting the Eiffel Tower with my family, we took a walk along the Seine River" (specific operation of generating memory-recreating content).

[0300] Step 4:

[0301] The server sends the generated memory reproduction content (input data) to the device. The device receives it and displays it to the user (specific actions for sending and displaying content). For example, the app's notification function can be used to notify the user, "A new memory has been created!" When the user clicks on the notification, the generated memory reproduction content (output data) is displayed within the app.

[0302] Step 5:

[0303] The device uses an emotion engine that analyzes the user's facial expressions, voice, and text input to recognize the user's emotions (input data). For example, if the user smiles at a displayed photo, the device recognizes "joy." If the user writes "I had a lot of fun" when entering a comment, the device analyzes it as a "positive emotion" (specific operation of emotion analysis).

[0304] Step 6:

[0305] The server generates optimal behavioral recommendations based on the emotional information (input data) sent from the emotion engine. For example, if the user expresses the emotion "joy," the server generates a behavioral recommendation (output data) such as "You seem to be in a good mood right now, so why don't you enjoy tea at a nearby cafe?" (Specific operation of behavioral recommendation generation).

[0306] Step 7:

[0307] The server sends the generated action recommendation (input data) to the device. The device displays this recommendation to the user as a push notification (specific operation of sending a recommendation and notification). For example, a message (output data) saying "It's time for a walk now. Let's go to a nearby park" is displayed.

[0308] In this way, each step of the system sequentially processes and analyzes data, and by providing the results to the user, it helps dementia patients recover their memories and improve their lives.

[0309] (Application example 2)

[0310] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0311] Current dementia patient support systems are inadequate in terms of memory recall and emotional recognition. Furthermore, in support of factory workers, they do not provide efficient work procedures or recommend breaks based on the worker's emotions, which prevents improvements in work efficiency and reduces the burden on workers. A comprehensive solution to these issues is needed.

[0312] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data on dementia patients, means for using a generative AI model to analyze the data, means for generating memory reproduction content based on the analysis results, means for collecting task data, means for using a work process analysis AI model to analyze the task data, and means for using an emotion recognition engine to recognize the user's emotions. This makes it possible to provide memory reproduction support to dementia patients and recommend efficient work procedures and emotion-based breaks to factory workers.

[0313] "Dementia" is a disease characterized by a decline in cognitive function that affects memory, judgment, communication, and daily activities.

[0314] "Data collection means" is a general term for any device or method for acquiring information from a user or object.

[0315] A "generative AI model" is an artificial intelligence algorithm that predicts, analyzes, and generates based on collected data.

[0316] "Memory reproduction content" refers to information and media materials that are used to reproduce a user's past memories based on analyzed data.

[0317] A "display terminal" is an electronic device such as a smartphone, tablet, or smart glasses that allows users to view and operate information.

[0318] "Behavioral recommendations" refer to notifications and advice that suggest optimal actions and options to users.

[0319] "Work data" is a set of information related to the progress, method, results, etc. of a work.

[0320] The "work process analysis AI model" is an artificial intelligence algorithm that analyzes work data and derives work efficiency and areas for improvement.

[0321] An "emotion recognition engine" is a technology for analyzing emotions based on a user's facial expressions, voice, text input, etc.

[0322] "Break recommendation" refers to notifications or advice that suggest taking a break at the optimal time based on the user's emotional state.

[0323] The system for implementing this invention provides integrated support for dementia patients and factory workers.

[0324] System Overview

[0325] This system includes a server, display terminals (smartphones, smart glasses, etc.), and a mobile robot within the factory. The server plays a central role in performing key processes such as data collection, analysis, generation, and display.

[0326] Key Components

[0327] 1. Data Collection Methods

[0328] 2. Generative AI Models

[0329] 3. Memory Recreation Content Generation Method

[0330] 4. Display and notification methods on display devices (smartphones, smart glasses, etc.)

[0331] 5. Emotion Recognition Engine

[0332] 6. Work Process Analysis AI Model

[0333] 1. Data Collection Methods

[0334] The dementia patient or worker uploads past photos, videos, notes, conversation history, and work data to a server through a data collection method, such as using a camera or microphone on a smartphone or smart glasses.

[0335] 2. Generative AI Models

[0336] The server receives the collected data and analyzes it using a generative AI model (e.g., TensorFlow, PyTorch), which automatically generates captions, anecdotes, and efficient work procedures from the user's past photos and work data.

[0337] 3. Memory Recreation Content Generation Method

[0338] Based on the analysis results, memory recall content is generated. For example, this includes adding captions to past photos and generating text about related episodes. In the case of work data, efficient work procedures are generated.

[0339] 4. Display terminal display and notification means

[0340] The memory reproduction content and work procedures sent from the server are displayed on a display device, and notifications are sent to the user via a smartphone or smart glasses, for example.

[0341] 5. Emotion Recognition Engine

[0342] The system analyzes the user's emotions in real time using a camera and microphone installed on the display device (smartphone, smart glasses). The analyzed emotional information is sent to a server and reflected in memory reproduction content and work procedure recommendations.

[0343] 6. Work Process Analysis AI Model

[0344] The server analyzes the work data using a work process analysis AI model (e.g., Keras, Scikit-learn) to extract efficient work procedures and areas for improvement. This information is then appropriately notified to the workers.

[0345] Specific examples

[0346] Let's say a user wears smart glasses and performs work in a factory. The work progress and sensor data are collected and sent to a server. The server analyzes this data using an AI model for work process analysis and generates work instructions such as, "This work usually takes 15 minutes to complete. Please try doing it this way."

[0347] The smart glasses also use a camera and microphone to analyze the worker's facial expressions and voice, which are then analyzed by an emotion recognition engine. If stress is detected, the system will recommend a break, such as "Take a 5-minute break now."

[0348] Prompt Sentence Examples

[0349] Build a system that recognizes workers' emotions in real time and sends them notifications to encourage them to take a break if they feel stressed or impatient, by using smart glasses on a parts assembly line and analyzing past work data to generate specific recommendations for optimal procedures.

[0350] As described above, this invention realizes a system that can restore memories, provide efficient work procedures, and recommend optimal actions based on emotions for both dementia patients and factory workers.

[0351] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0352] Step 1: Data collection

[0353] A user uses smart glasses or a smartphone to collect past data (e.g., photos, videos, notes, conversation history, and work data). This data is sent to a server by a data collection means. The input data is photos and work progress, and the output data is information stored in a database on the server.

[0354] Step 2: Data analysis

[0355] The server analyzes the received data using a generative AI model (e.g., TensorFlow, PyTorch). The input data is the user's data stored on the server, and the output data is the analysis results: captions, episodes, and work procedures. Specifically, it generates captions for photos and extracts efficient procedures from work data.

[0356] Step 3: Generate memory recall content

[0357] The server generates memory-recreating content based on the analysis results. The input data is the analysis results from the generative AI model, and the output data is the memory-recreating content (for example, photo captions or episode text). Specifically, it adds captions to photos and generates work procedures as text.

[0358] Step 4: Send to display device

[0359] The server sends the generated memory reproduction content and work procedures to a display device (smartphone or smart glasses). The input data is the memory reproduction content and work procedures, and the output data is the content displayed on the device.

[0360] Step 5: Emotion Recognition

[0361] To recognize the user's emotions, the device (smart glasses or smartphone) uses a camera and microphone to collect the user's facial expression and voice data. This data is analyzed by an emotion recognition engine. The input data is the user's facial expression and voice, and the output data is the emotion analysis results. Specifically, the device analyzes the user's emotional state, such as stress or joy.

[0362] Step 6: Generate action recommendations

[0363] The server recommends optimal actions to the user based on the results of emotion analysis and past data. The input data are the results of emotion analysis and past data, and the output data is the recommended action. Specifically, notifications such as "Take a 5-minute break now" or "Try this work procedure" are generated.

[0364] Step 7: Notify the display device

[0365] The server notifies the generated action recommendation to the display device. The input data is the action recommendation, and the output data is the notification displayed on the device. Specifically, it is displayed as a push notification on smart glasses or a smartphone.

[0366] In this way, it is possible to support memory recall for dementia patients, provide efficient work procedures for factory workers, and even recommend optimal actions based on emotions.

[0367] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0368] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0369] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0370] [Second embodiment]

[0371] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0372] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0373] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0374] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0375] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0376] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0377] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0378] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0379] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0380] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0381] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0382] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0383] This invention is a system for supporting the memory of dementia patients and recommending optimal actions. Specifically, the system collects data on dementia patients, analyzes it using a generative AI model, and generates memory-reconstruction content based on the results, which is then displayed on a smartphone. Furthermore, the system recommends optimal actions based on the patient's current condition and notifies the smartphone.

[0384] System Configuration

[0385] The system consists of the following main components:

[0386] 1. Data Collection Methods

[0387] 2. Generative AI Models

[0388] 3. Memory Recreation Content Generation Method

[0389] 4. Smartphone display and notification methods

[0390] Data collection methods

[0391] Users (patients or their family members) use a smartphone app to upload data such as past photos, videos, notes, and conversation history.

[0392] Generative AI Models

[0393] The server receives the data sent by the user and analyzes it using a generative AI model, which automatically generates photo captions and related stories based on the data.

[0394] Memory reproduction content generation means

[0395] The server creates memory-recreating content based on the analysis results, including adding captions related to photos and generating explanatory text about past episodes.

[0396] Smartphone display and notification methods

[0397] The device (smartphone) displays the memory-recreating content sent from the server to the user. The server then recommends optimal actions and sends a notification to the device. The device then prompts the user to take action via push notifications or alert messages.

[0398] Specific examples

[0399] A user opens a smartphone app and uploads photos from a previous trip, which the device then sends to the server.

[0400] The server analyzes the image file and its metadata and generates a caption, such as "This photo was taken in front of the Eiffel Tower during a trip to France in the summer of 2019." The server then sends the memory-recreating content, including the caption, to the device, which displays it to the user, allowing them to relive their memories by looking at the photo.

[0401] Also, if the server generates a recommendation that "It's a good time to take a walk," the device will send a notification saying, "It's time for a walk now. Let's go to the nearest park." By carrying out the suggested action, the user who receives this recommendation can make their daily life more fulfilling.

[0402] The system will help dementia patients relive their past memories and provide a sense of security, while also making it easier for family and caregivers to provide optimal support, improving overall quality of life.

[0403] The processing flow will be explained below.

[0404] Step 1:

[0405] Users can launch a smartphone application and upload data such as past photos, videos, notes, conversation history, etc. Users can also select photos and enter captions and tags.

[0406] Step 2:

[0407] The device processes the uploaded data and prepares it for transmission to the server. Specifically, the device compresses the image files and adds appropriate metadata (e.g., date and time of the photo, location).

[0408] Step 3:

[0409] The device sends the prepared data to the server, which then sends a POST request to the server with the image and its metadata to the specified API endpoint.

[0410] Step 4:

[0411] The server stores the received data in storage. Specifically, the server stores the image file path and metadata in a database, and stores the image itself in cloud storage.

[0412] Step 5:

[0413] The server analyzes the stored data using a generative AI model. Specifically, the server inputs an image into the AI ​​model and generates an episode related to the image (e.g., the background of the photo, information about the people in the photo).

[0414] Step 6:

[0415] The server stores the analysis results in a database. Specifically, the server stores the analysis results (e.g., "This is a photo I took with my family at the beach last summer") in the database.

[0416] Step 7:

[0417] The server generates memory-recreating content based on the analysis results. Specifically, the server adds captions to images and generates related episodes as text.

[0418] Step 8:

[0419] The server sends the generated memory reproduction content to the device. Specifically, the server sends the generated captions along with the images in JSON format to the device.

[0420] Step 9:

[0421] The device displays the memory reproduction content received from the server to the user. Specifically, the device opens an image viewer and displays the image and its caption on the screen.

[0422] Step 10:

[0423] The server uses a generative AI model to recommend optimal actions based on the patient's condition and past data analysis results. Specifically, the server generates recommendations such as "Taking a walk at 3 p.m."

[0424] Step 11:

[0425] The server sends the generated action recommendation to the terminal. Specifically, the server sends the action recommendation to the terminal as a notification message.

[0426] Step 12:

[0427] The device notifies the user of recommended activities. Specifically, the device displays a push notification with the message, "It's time for a walk. Let's go to your favorite park."

[0428] Example 1

[0429] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0430] Dementia patients have difficulty retaining their memories and are prone to confusion in their daily lives. Caregivers and families also have limited information to provide the necessary support. Furthermore, a lack of appropriate behavioral instructions based on the patient's current condition can lead to a decline in quality of life. There is a need to address these issues.

[0431] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0432] In this invention, the server includes: means for a user to upload information such as past photos, videos, notes, and conversation history using a smartphone app; means for receiving and saving the information sent from the smartphone app; means for analyzing the saved information and generating photo captions and related episodes using a generative AI model; means for generating memory reproduction content based on the analysis results; means for sending the memory reproduction content to the smartphone; means for displaying the memory reproduction content on the smartphone; means for recommending optimal actions based on the patient's current condition; and means for notifying the smartphone of the recommended actions. This makes it easier for dementia patients to recall their memories, making it easier for caregivers and family members to provide the necessary support and providing appropriate instructions for actions based on the patient's current condition.

[0433] "User" refers to a person with dementia or their family member who uses the system.

[0434] "Smartphone app" refers to application software that runs on a smartphone and allows users to upload past information and receive memory recall content and behavioral recommendations.

[0435] "Information" refers to data uploaded by users, such as past photos, videos, notes, and conversation history.

[0436] "Server" refers to a computer system that receives, stores, and analyzes information sent by users, generates captions and episodes using generative AI models, and transmits memory reproduction content and behavioral recommendations to smartphones.

[0437] A "generative AI model" refers to an artificial intelligence model that analyzes information sent by users and automatically generates photo captions and related stories.

[0438] "Caption" refers to explanatory text associated with a photograph.

[0439] An "episode" is a descriptive text about a past event related to a photo or video.

[0440] "Memory recall content" refers to content that includes the analyzed photo captions and related episode text.

[0441] "Action recommendation" refers to a recommendation of the optimal action that is generated based on the patient's current condition.

[0442] "Notifications" refers to alerts sent to your smartphone with recommended actions or other important information.

[0443] This invention is a system for supporting memory and recommending optimal actions for dementia patients. The system consists of the following main components:

[0444] 1. Data Collection Methods

[0445] 2. Generative AI Models

[0446] 3. Memory Recreation Content Generation Method

[0447] 4. Smartphone display and notification methods

[0448] Data collection methods

[0449] Users upload information such as past photos, videos, memos, and conversation histories using a smartphone app. The smartphone app then sends the files selected by the user to the server via an HTTP request.

[0450] Generative AI Models

[0451] The server receives and stores information sent from the smartphone app. The stored information is then analyzed using a generative AI model, such as the latest natural language processing AI, such as GPT-4.

[0452] The following prompts are used during parsing:

[0453] text

[0454] This is a user-uploaded photo. Please provide information about the photo:

[0455] [Photo metadata (date, time, location, etc.)]

[0456] Based on this prompt, the generative AI model automatically generates captions for the photos and related stories.

[0457] Memory reproduction content generation means

[0458] The server generates memory-recreating content based on the results of analysis by the generative AI model. This content includes adding captions related to photos and generating explanatory text about past episodes. The generated content is combined in HTML or JSON format and organized in a format that is easy for users to access.

[0459] Smartphone display and notification methods

[0460] The device (smartphone) displays the memory replay content sent from the server to the user. By viewing this content, the user can relive past memories. The server also recommends optimal actions based on the patient's current condition and sends a notification of this recommendation to the device. The device then communicates this recommendation to the user in the form of a push notification or an alert message.

[0461] Specific examples

[0462] Consider a case where a user wants to upload photos from a previous trip. The user opens a smartphone app, selects a travel photo, and taps the upload button. This causes the device to send data including the photo to the server.

[0463] The server analyzes the image file and its metadata and generates a caption, such as "This photo was taken in front of the Eiffel Tower during a trip to France in the summer of 2019." Based on this, memory-recreating content is created and sent to the device. The device then displays this content to the user, allowing them to relive their past memories.

[0464] Furthermore, if the server generates a recommendation that "It's a good time to take a walk," the device will notify the user, "It's time for a walk now. Let's go to the nearest park." By carrying out the suggested action, the user who receives this recommendation can make their daily life more fulfilling.

[0465] This system will help dementia patients relive their past memories and provide a sense of security, while also making it easier for family and caregivers to provide optimal support, improving their overall quality of life.

[0466] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0467] Step 1:

[0468] A user uses a smartphone app to upload information such as past photos, videos, notes, and conversation history. Specifically, the user opens the smartphone app and taps the upload button to launch a file browser. When the user selects the files they want to upload, the device sends the selected files to the server via an HTTP request.

[0469] Input: Files such as past photos, videos, notes, and conversation history

[0470] Output: HTTP request from the terminal to the server

[0471] Step 2:

[0472] The server receives and stores data sent from the device. The server processes the received HTTP request and reads the sent file. The read data is stored in cloud storage or a database.

[0473] Input: File data sent from the terminal

[0474] Output: Saved file data

[0475] Step 3:

[0476] The server analyzes the stored data and generates photo captions and related episodes using a generative AI model. The server checks the file type and metadata and generates prompts for analysis. When the generated prompts are input into the generative AI model, the AI ​​model generates photo captions and episodes.

[0477] Input: Saved file data, prompt text

[0478] Output: Generated photo captions and related episodes

[0479] Step 4:

[0480] The server generates memory reproduction content based on the analysis results. The server integrates the analyzed captions and episodes into content in HTML or JSON format. This content is organized in a format that is easy for users to view.

[0481] Input: Generated photo captions, related episodes

[0482] Output: Memory-reproduced content (HTML or JSON format)

[0483] Step 5:

[0484] The device receives the memory reproduction content sent from the server and displays it to the user. The device receives the HTTP response and displays the content in the response within the app. By viewing this, the user can relive past memories.

[0485] Input: Memory-reproduced content sent from the server

[0486] Output: Memory-reproduced content displayed within the app

[0487] Step 6:

[0488] The server recommends the optimal action based on the patient's current condition and sends a notification to the device. The server analyzes past data and the patient's current condition to generate a recommendation such as "It's a good idea to take a walk at this time of day." The generated recommendation is sent to the device via an HTTP request.

[0489] Input: Current patient status, historical data

[0490] Output: Generated action recommendations

[0491] Step 7:

[0492] The device receives the recommended actions sent from the server and notifies the user. The device receives the HTTP response and displays the content to the user as a push notification or alert message. The user can check the notification and carry out the suggested actions to enrich their daily life.

[0493] Input: Action recommendations sent from the server

[0494] Output: Push notification or alert message to the user

[0495] (Application example 1)

[0496] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0497] Due to memory decline, dementia patients often forget past events and the locations of their daily activities. They may also get lost or find themselves in dangerous situations when out and about, which reduces the patient's quality of life and increases the anxiety of their families and caregivers. Conventional systems are unable to adequately address these issues, making emergency response and ensuring patient safety particularly challenging.

[0498] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0499] In this invention, the server includes means for collecting data on dementia patients, means for using a generative AI model to analyze the data, means for generating memory reproduction content based on the analysis results, means for transmitting the memory reproduction content to a smartphone, means for displaying the memory reproduction content on the smartphone, means for recommending optimal actions, means for notifying the smartphone of the recommended actions, means for collecting location information to track the patient's location, means for monitoring the surrounding environment using a camera and a microphone, means for issuing an emergency alert when a dangerous situation is detected, and means for sending notifications to caregivers in real time. This makes it possible to support the memory of dementia patients, recommend optimal actions, and ensure the safety of the patients.

[0500] A "dementia patient" is someone whose cognitive function has declined, resulting in memory loss, impaired judgment, and difficulty in daily life.

[0501] "Data" refers to information such as photos, videos, notes, and conversation histories related to dementia patients.

[0502] A "generative AI model" is an artificial intelligence model that analyzes collected data and generates the necessary information.

[0503] "Memory reproduction content" refers to content such as captions and episode texts that are created based on data analyzed by a generative AI model to reproduce a patient's memories.

[0504] A "smartphone" is a mobile device that can use a wide variety of applications in addition to the functions of a mobile phone.

[0505] "Action recommendations" are optimal actions suggested based on the patient's current situation and past data.

[0506] "Location information collection means" refers to a means for tracking the patient's current location using GPS or other means.

[0507] An "environmental monitoring means" is a means for monitoring the environment surrounding the patient using a camera or microphone.

[0508] An "emergency alert means" is a means for issuing a warning when a dangerous situation is detected.

[0509] "Caregivers" are family members or professionals who play a role in helping people with dementia on a day-to-day basis.

[0510] "Real-time" refers to processing occurring almost at the same time as the data is generated.

[0511] This invention is a system for supporting the memory of dementia patients and recommending optimal actions. Specifically, the system collects the patient's past data and analyzes it using a generative AI model to generate memory reproduction content and display it on a smartphone. It also has the function of recommending optimal actions based on the patient's current condition and notifying the smartphone.

[0512] The server first provides a means to collect data on dementia patients. This data, such as past photos, videos, notes, and conversation histories, is uploaded by patients and their families using a smartphone app. The server receives this data and analyzes it using a generative AI model. The generative AI model automatically generates photo captions and text for related episodes based on the collected data.

[0513] The server then generates memory-recreating content based on the analysis results. The generated memory-recreating content is sent to a smartphone, where it is displayed on the device. For example, if a past travel photo is uploaded, the generative AI model analyzes the photo and generates a caption such as, "This photo was taken at a tourist spot during a trip in the summer of 2019." The device then displays the memory-recreating content, including the caption, to the user, allowing them to view the photo and relive their memory.

[0514] Furthermore, the server has the function of recommending optimal actions. Based on past data and the patient's current condition, the generative AI model considers the current time of day and the patient's activity patterns to suggest appropriate actions. For example, a recommendation such as "It's a good idea to take a walk at this time of day" may be generated. This recommendation is sent as a push notification to the smartphone, displaying a message saying, "It's time for a walk now. Let's go to the nearest park."

[0515] The system also includes a location information collection means for tracking the patient's location, an environmental monitoring means for monitoring the patient's surroundings using a camera and microphone, an emergency alert means for issuing a warning when a dangerous situation is detected, and a means for sending a notification to a caregiver in real time. For example, if a patient goes out alone to a park, the generative AI model will track the patient's location using GPS and monitor the surrounding environment using a camera and microphone. Based on this information, the generative AI model will detect a dangerous situation and send a real-time notification to the caregiver saying, "Emergency: Patient may be in danger."

[0516] Examples of prompt sentences include the following:

[0517] "Analyze the photos and audio data sent by the user and generate a description of the environment and memory-recreating content. For example, if the sent photo was taken in a park, generate a description such as, 'This is a park near my house. I've been there before with my family.'"

[0518] This makes it easier for people with dementia to relive their past memories and feel more secure, and it also makes it easier for family and caregivers to ensure the patient's safety, improving their overall quality of life.

[0519] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0520] Step 1:

[0521] A user uses a smartphone app to upload data such as past photos, videos, notes, and conversation history. The input is the data uploaded by the user, and the output is the data sent to the server.

[0522] Step 2:

[0523] The server receives the data submitted by the user. The input is the data submitted by the user, and the output is the data to be fed into the generative AI model. The server converts this into a format suitable for analysis.

[0524] Step 3:

[0525] The server analyzes the received data using a generative AI model. The input is the received data, and the output is the analysis result. The generative AI model analyzes the content and metadata of the photo and generates captions and episode text.

[0526] Step 4:

[0527] The server generates memory-recreation content based on the analysis results. The input is the analysis results, and the output is the memory-recreation content. This content includes the generated captions and episode text.

[0528] Step 5:

[0529] The server transmits the generated memory reproduction content to the smartphone. The input is the memory reproduction content, and the output is the data transmitted to the smartphone.

[0530] Step 6:

[0531] The terminal displays the memory reproduction content sent from the server to the user. The input is the memory reproduction content, and the output is the screen displayed to the user. The terminal visually displays captions and episode text.

[0532] Step 7:

[0533] The server recommends the optimal action. The input is historical data and the patient's current condition, and the output is a recommended action. The generative AI model takes into account the current time of day and the patient's activity patterns to suggest appropriate actions.

[0534] Step 8:

[0535] The server notifies the smartphone of the recommended actions. The input is the recommended actions, and the output is a notification sent to the smartphone. The smartphone then suggests appropriate actions to the user through a push notification.

[0536] Step 9:

[0537] The terminal obtains real-time location information using a location information collection means that tracks the patient's location information. The input is GPS data, and the output is the patient's current location.

[0538] Step 10:

[0539] The device monitors the surrounding environment using a camera and microphone. The input is camera video and audio data, and the output is environmental information. The camera and microphone periodically collect data and provide it for analysis.

[0540] Step 11:

[0541] The server uses an emergency alert mechanism to issue a warning when a dangerous situation is detected. The input is location information and environmental information, and the output is a warning message. The generative AI model analyzes this data and generates a warning if a dangerous situation is detected.

[0542] Step 12:

[0543] The server sends real-time notifications to the caregiver: the input is the warning message and the output is the notification sent to the caregiver, allowing the caregiver to respond immediately.

[0544] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0545] This invention is a system for supporting the memory of dementia patients and recommending optimal actions. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide more personalized support. Specifically, this system has the function of analyzing the user's emotions and adjusting memory recall content and optimal action recommendations based on that.

[0546] System Configuration

[0547] The system consists of the following main components:

[0548] 1. Data Collection Methods

[0549] 2. Generative AI Models

[0550] 3. Memory Recreation Content Generation Method

[0551] 4. Smartphone display and notification methods

[0552] 5. Emotion Engine

[0553] Data collection methods

[0554] Users (patients or their family members) use a smartphone app to upload data such as past photos, videos, notes, and conversation history.

[0555] Generative AI Models

[0556] The server receives the data sent by the user and analyzes it using a generative AI model, which automatically generates photo captions and related stories based on the data.

[0557] Memory reproduction content generation means

[0558] The server creates memory-recreating content based on the analysis results, including adding captions related to photos and generating explanatory text about past episodes.

[0559] Smartphone display and notification methods

[0560] The device (smartphone) displays the memory-recreating content sent from the server to the user. The server then recommends optimal actions and sends a notification to the device. The device then prompts the user to take action via push notifications or alert messages.

[0561] Emotion Engine

[0562] The device has an emotion engine that recognizes the user's emotions. This emotion engine analyzes emotions based on the user's facial expressions, voice, and text input. The analyzed emotion information is sent to the server and reflected in memory reproduction content and behavioral recommendations.

[0563] Specific examples

[0564] A user opens a smartphone app and uploads photos from a previous trip, which the device then sends to the server.

[0565] The server analyzes the image file and its metadata and generates a caption, such as "This photo was taken in front of the Eiffel Tower during a trip to France in the summer of 2019." The server then sends the memory-recreating content, including the caption, to the device, which displays it to the user, allowing them to relive their memories by looking at the photo.

[0566] The device also recognizes emotions such as "happiness" and "surprise" through facial and voice analysis of the user. Based on this, the server generates a recommendation such as "It's time for a walk. Let's go to the nearest park" and sends it to the device. This recommendation is displayed at an appropriate time depending on the emotion.

[0567] For example, if the system recognizes that the user is feeling stressed, it will give a recommendation such as, "Why not take a break and read your favorite book?"

[0568] This system makes it easier for dementia patients to relive their past memories, providing a sense of security. It also recommends appropriate actions based on the user's emotions, further improving their quality of life. It also makes it easier for family members and caregivers to provide emotional support, enhancing overall care.

[0569] The processing flow will be explained below.

[0570] Step 1:

[0571] Users can launch a smartphone application and upload data such as past photos, videos, notes, conversation history, etc. Users can also select photos and enter captions and tags.

[0572] Step 2:

[0573] The device processes the uploaded data and prepares it for transmission to the server. Specifically, the device compresses the image files and adds appropriate metadata (e.g., date and time of the photo, location).

[0574] Step 3:

[0575] The device sends the prepared data to the server, which then sends a POST request to the server with the image and its metadata to the specified API endpoint.

[0576] Step 4:

[0577] The server stores the received data in storage. Specifically, the server stores the image file path and metadata in a database, and stores the image itself in cloud storage.

[0578] Step 5:

[0579] The server analyzes the stored data using a generative AI model. Specifically, the server inputs an image into the AI ​​model and generates an episode related to the image (e.g., the background of the photo, information about the people in the photo).

[0580] Step 6:

[0581] The server stores the analysis results in a database. Specifically, the server stores the analysis results (e.g., "This is a photo I took with my family at the beach last summer") in the database.

[0582] Step 7:

[0583] The server generates memory-recreating content based on the analysis results. Specifically, the server adds captions to images and generates related episodes as text.

[0584] Step 8:

[0585] The server sends the generated memory reproduction content to the device. Specifically, the server sends the generated captions along with the images in JSON format to the device.

[0586] Step 9:

[0587] The device displays the memory reproduction content received from the server to the user. Specifically, the device opens an image viewer and displays the image and its caption on the screen.

[0588] Step 10:

[0589] The device collects facial and voice data through a camera and microphone to recognize the user's emotions. Specifically, the device collects emotional data in real time while the user is browsing photos and stories.

[0590] Step 11:

[0591] The device analyzes the collected emotional data and identifies the user's current emotional state. Specifically, the emotion engine determines emotions such as "happiness," "surprise," and "sadness" from the user's facial expressions and voice.

[0592] Step 12:

[0593] The device sends the analyzed emotion data to the server by sending a POST request to the server's API endpoint in JSON format.

[0594] Step 13:

[0595] The server receives the emotion data and reflects it in memory-recreating content and optimal action recommendations. Specifically, it adjusts the content and recommendations to be displayed based on the emotion data.

[0596] Step 14:

[0597] The server uses a generative AI model to recommend optimal actions based on the patient's condition, past data analysis results, and emotional data. Specifically, it generates a recommendation such as "A walk is effective at the current time," but if the user is feeling stressed, it generates a different recommendation such as "Why don't you take a break and read your favorite book?"

[0598] Step 15:

[0599] The server sends the generated action recommendation to the terminal. Specifically, the server sends the action recommendation to the terminal as a notification message.

[0600] Step 16:

[0601] The device notifies the user of recommended actions. Specifically, it encourages the user to take action by means of push notifications or alert messages. For example, the device may notify the user, "It's time for a walk. Let's go to the nearest park."

[0602] Through these processing steps, this system supports the memory of dementia patients and improves their quality of life by recommending optimal actions based on their emotions. It also makes it easier for family members and caregivers to understand the patient's emotional state and provide more appropriate support.

[0603] Example 2

[0604] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0605] Dementia patients have difficulty retaining past memories and often experience stress in their daily lives. Furthermore, there are limited ways to encourage dementia patients to behave appropriately, placing a heavy burden on their families and caregivers. The problem this invention aims to solve is to improve the quality of life of dementia patients by supporting their memory recall and recommending optimal actions based on their emotions.

[0606] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user data, means for using a generative AI model to analyze the user data, means for generating memory reproduction content based on the analysis results, means for transmitting the memory reproduction content to a terminal, means for displaying the memory reproduction content on the terminal, means for analyzing the user's emotions, means for recommending optimal actions based on the user's emotional information, and means for notifying the terminal of the recommended actions. This allows dementia patients to easily reproduce past memories and receive appropriate recommended actions based on their emotions.

[0607] "User" refers to anyone who uses the system, including a dementia patient or their family or caregiver.

[0608] "Data collection means" refers to the ability of users to upload information such as photos, videos, notes, and conversation history using smartphone apps.

[0609] A "generative AI model" is an artificial intelligence technology that analyzes data collected from users and automatically generates photo captions and related stories.

[0610] "Memory reproduction content" is content that includes photo captions and text descriptions created based on the results of analysis by a generative AI model.

[0611] "Terminal" refers to a display device such as a smartphone or tablet used by a user.

[0612] "Emotion analysis means" is a technology that has the function of analyzing a user's facial expressions, voice, character input, etc., and recognizing the user's emotions.

[0613] "Behavioral recommendations" refer to specific actions suggested to users based on analyzed emotional information and past data.

[0614] The "notification means" is a function that sends the generated memory reproduction content and behavioral recommendations to the terminal as a push notification or an alert message and displays them to the user.

[0615] This invention is a system that supports memory for dementia patients and recommends optimal actions. By combining this system with an emotion engine that recognizes the user's emotions, it is possible to provide more personalized support.

[0616] System Configuration

[0617] The system consists of the following main components:

[0618] 1. Data Collection Methods

[0619] 2. Generative AI Models

[0620] 3. Memory Recreation Content Generation Method

[0621] 4. Terminal display and notification methods

[0622] 5. Emotion Engine

[0623] Data collection methods

[0624] Users use a smartphone app to upload data such as past photos, videos, memos, and conversation histories. For example, they can upload photos of a family trip or diary notes to the app. The device then sends this data to a server.

[0625] Generative AI Models

[0626] The server uses a generative AI model to analyze the data received from the device. This generative AI model is an advanced natural language processing model such as GPT-4. The server analyzes the metadata of the photo or video (e.g., date and time of the photo, location) and generates a caption based on that. For example, it generates a caption such as, "This photo was taken in front of the Eiffel Tower in Paris, France, in the summer of 2019."

[0627] Memory reproduction content generation means

[0628] The server generates memory-recreating content based on the analysis results. This content includes captions related to the photos and explanatory text about past episodes. For example, a photo of a trip to France might include a description such as, "On a beautiful summer day, my family and I visited the Eiffel Tower and then took a walk along the Seine."

[0629] Terminal display and notification means

[0630] The device displays the memory reproduction content sent from the server to the user. For example, the app's notification function can be used to notify the user, "A new memory has been created!" When the user clicks on the notification, the generated memory reproduction content is displayed within the app.

[0631] Emotion Engine

[0632] The device has an emotion engine that analyzes the user's facial expressions, voice, and text input to recognize emotions. For example, if the user smiles at a displayed photo, the device recognizes "joy." If the user writes "I had a lot of fun" when entering a comment, the device analyzes it as a "positive emotion."

[0633] Action recommendation generation and notification

[0634] The server generates optimal action recommendations based on the emotional information sent from the emotion engine. For example, if the user expresses the emotion "joy," the server generates a recommendation such as, "You seem to be in a good mood right now, so why not enjoy some tea at a nearby cafe?" The server sends this recommendation to the device. The device notifies the user of this recommendation, displaying, for example, "It's time for a walk now. Let's go to a nearby park."

[0635] Specific examples

[0636] Prompt Sentence Examples

[0637] Upload past travel photos and the system will analyze your emotions and recommend the best course of action.

[0638] Example

[0639] A user opens the smartphone app, selects a photo from their trip to France in 2019, and presses the upload button. The device sends this photo to the server. The server analyzes the received image and generates a caption such as, "This photo was taken in front of the Eiffel Tower during my trip to France in the summer of 2019." The server creates memory-recreating content including this caption and sends it to the device. The device notifies the user via push notification, "A new memory has been created!" When the user clicks the notification, the generated content is displayed within the app. The device analyzes the user's facial expression with an emotion engine and recognizes "joy." Based on this emotional information, the server generates an action recommendation such as, "It's time for a walk now. Let's go to the nearest park." The device notifies the user of this recommendation.

[0640] This system allows dementia patients to easily recall past memories and receive appropriate behavioral recommendations based on their emotions, enabling them to enrich their daily lives.

[0641] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0642] Step 1:

[0643] The user opens the smartphone app, selects past photos, videos, notes, and conversation history (input data), and presses the upload button. The device then sends this data to the server (specific operation of data transmission).

[0644] Step 2:

[0645] The server uses a generative AI model to analyze the data received from the device (input data). It uses a generative AI model (e.g., GPT-4) to check the metadata of the photo or video (e.g., date and time of the photo, location) and generates a caption (specific operation of data analysis and caption generation). For example, it generates a caption (output data) such as, "This photo was taken in front of the Eiffel Tower in Paris, France, in the summer of 2019."

[0646] Step 3:

[0647] The server creates memory-recreating content based on the generated captions (input data). This content includes captions related to the photos and explanatory text about past episodes. For example, to a photo of a trip to France, it adds the description (output data) "On a beautiful summer day, after visiting the Eiffel Tower with my family, we took a walk along the Seine River" (specific operation of generating memory-recreating content).

[0648] Step 4:

[0649] The server sends the generated memory reproduction content (input data) to the device. The device receives it and displays it to the user (specific actions for sending and displaying content). For example, the app's notification function can be used to notify the user, "A new memory has been created!" When the user clicks on the notification, the generated memory reproduction content (output data) is displayed within the app.

[0650] Step 5:

[0651] The device uses an emotion engine that analyzes the user's facial expressions, voice, and text input to recognize the user's emotions (input data). For example, if the user smiles at a displayed photo, the device recognizes "joy." If the user writes "I had a lot of fun" when entering a comment, the device analyzes it as a "positive emotion" (specific operation of emotion analysis).

[0652] Step 6:

[0653] The server generates optimal behavioral recommendations based on the emotional information (input data) sent from the emotion engine. For example, if the user expresses the emotion "joy," the server generates a behavioral recommendation (output data) such as "You seem to be in a good mood right now, so why don't you enjoy tea at a nearby cafe?" (Specific operation of behavioral recommendation generation).

[0654] Step 7:

[0655] The server sends the generated action recommendation (input data) to the device. The device displays this recommendation to the user as a push notification (specific operation of sending a recommendation and notification). For example, a message (output data) saying "It's time for a walk now. Let's go to a nearby park" is displayed.

[0656] In this way, each step of the system sequentially processes and analyzes data, and by providing the results to the user, it helps dementia patients recover their memories and improve their lives.

[0657] (Application example 2)

[0658] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0659] Current dementia patient support systems are inadequate in terms of memory recall and emotional recognition. Furthermore, in support of factory workers, they do not provide efficient work procedures or recommend breaks based on the worker's emotions, which prevents improvements in work efficiency and reduces the burden on workers. A comprehensive solution to these issues is needed.

[0660] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data on dementia patients, means for using a generative AI model to analyze the data, means for generating memory reproduction content based on the analysis results, means for collecting task data, means for using a work process analysis AI model to analyze the task data, and means for using an emotion recognition engine to recognize the user's emotions. This makes it possible to provide memory reproduction support to dementia patients and recommend efficient work procedures and emotion-based breaks to factory workers.

[0661] "Dementia" is a disease characterized by a decline in cognitive function that affects memory, judgment, communication, and daily activities.

[0662] "Data collection means" is a general term for any device or method for acquiring information from a user or object.

[0663] A "generative AI model" is an artificial intelligence algorithm that predicts, analyzes, and generates based on collected data.

[0664] "Memory reproduction content" refers to information and media materials that are used to reproduce a user's past memories based on analyzed data.

[0665] A "display terminal" is an electronic device such as a smartphone, tablet, or smart glasses that allows users to view and operate information.

[0666] "Behavioral recommendations" refer to notifications and advice that suggest optimal actions and options to users.

[0667] "Work data" is a set of information related to the progress, method, results, etc. of a work.

[0668] The "work process analysis AI model" is an artificial intelligence algorithm that analyzes work data and derives work efficiency and areas for improvement.

[0669] An "emotion recognition engine" is a technology for analyzing emotions based on a user's facial expressions, voice, text input, etc.

[0670] "Break recommendation" refers to notifications or advice that suggest taking a break at the optimal time based on the user's emotional state.

[0671] The system for implementing this invention provides integrated support for dementia patients and factory workers.

[0672] System Overview

[0673] This system includes a server, display terminals (smartphones, smart glasses, etc.), and a mobile robot within the factory. The server plays a central role in performing key processes such as data collection, analysis, generation, and display.

[0674] Key Components

[0675] 1. Data Collection Methods

[0676] 2. Generative AI Models

[0677] 3. Memory Recreation Content Generation Method

[0678] 4. Display and notification methods on display devices (smartphones, smart glasses, etc.)

[0679] 5. Emotion Recognition Engine

[0680] 6. Work Process Analysis AI Model

[0681] 1. Data Collection Methods

[0682] The dementia patient or worker uploads past photos, videos, notes, conversation history, and work data to a server through a data collection method, such as using a camera or microphone on a smartphone or smart glasses.

[0683] 2. Generative AI Models

[0684] The server receives the collected data and analyzes it using a generative AI model (e.g., TensorFlow, PyTorch), which automatically generates captions, anecdotes, and efficient work procedures from the user's past photos and work data.

[0685] 3. Memory Recreation Content Generation Method

[0686] Based on the analysis results, memory recall content is generated. For example, this includes adding captions to past photos and generating text about related episodes. In the case of work data, efficient work procedures are generated.

[0687] 4. Display terminal display and notification means

[0688] The memory reproduction content and work procedures sent from the server are displayed on a display device, and notifications are sent to the user via a smartphone or smart glasses, for example.

[0689] 5. Emotion Recognition Engine

[0690] The system analyzes the user's emotions in real time using a camera and microphone installed on the display device (smartphone, smart glasses). The analyzed emotional information is sent to a server and reflected in memory reproduction content and work procedure recommendations.

[0691] 6. Work Process Analysis AI Model

[0692] The server analyzes the work data using a work process analysis AI model (e.g., Keras, Scikit-learn) to extract efficient work procedures and areas for improvement. This information is then appropriately notified to the workers.

[0693] Specific examples

[0694] Let's say a user wears smart glasses and performs work in a factory. The work progress and sensor data are collected and sent to a server. The server analyzes this data using an AI model for work process analysis and generates work instructions such as, "This work usually takes 15 minutes to complete. Please try doing it this way."

[0695] The smart glasses also use a camera and microphone to analyze the worker's facial expressions and voice, which are then analyzed by an emotion recognition engine. If stress is detected, the system will recommend a break, such as "Take a 5-minute break now."

[0696] Prompt Sentence Examples

[0697] Build a system that recognizes workers' emotions in real time and sends them notifications to encourage them to take a break if they feel stressed or impatient, by using smart glasses on a parts assembly line and analyzing past work data to generate specific recommendations for optimal procedures.

[0698] As described above, this invention realizes a system that can restore memories, provide efficient work procedures, and recommend optimal actions based on emotions for both dementia patients and factory workers.

[0699] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0700] Step 1: Data collection

[0701] A user uses smart glasses or a smartphone to collect past data (e.g., photos, videos, notes, conversation history, and work data). This data is sent to a server by a data collection means. The input data is photos and work progress, and the output data is information stored in a database on the server.

[0702] Step 2: Data analysis

[0703] The server analyzes the received data using a generative AI model (e.g., TensorFlow, PyTorch). The input data is the user's data stored on the server, and the output data is the analysis results: captions, episodes, and work procedures. Specifically, it generates captions for photos and extracts efficient procedures from work data.

[0704] Step 3: Generate memory recall content

[0705] The server generates memory-recreating content based on the analysis results. The input data is the analysis results from the generative AI model, and the output data is the memory-recreating content (for example, photo captions or episode text). Specifically, it adds captions to photos and generates work procedures as text.

[0706] Step 4: Send to display device

[0707] The server sends the generated memory reproduction content and work procedures to a display device (smartphone or smart glasses). The input data is the memory reproduction content and work procedures, and the output data is the content displayed on the device.

[0708] Step 5: Emotion Recognition

[0709] To recognize the user's emotions, the device (smart glasses or smartphone) uses a camera and microphone to collect the user's facial expression and voice data. This data is analyzed by an emotion recognition engine. The input data is the user's facial expression and voice, and the output data is the emotion analysis results. Specifically, the device analyzes the user's emotional state, such as stress or joy.

[0710] Step 6: Generate action recommendations

[0711] The server recommends optimal actions to the user based on the results of emotion analysis and past data. The input data are the results of emotion analysis and past data, and the output data is the recommended action. Specifically, notifications such as "Take a 5-minute break now" or "Try this work procedure" are generated.

[0712] Step 7: Notify the display device

[0713] The server notifies the generated action recommendation to the display device. The input data is the action recommendation, and the output data is the notification displayed on the device. Specifically, it is displayed as a push notification on smart glasses or a smartphone.

[0714] In this way, it is possible to support memory recall for dementia patients, provide efficient work procedures for factory workers, and even recommend optimal actions based on emotions.

[0715] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0716] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0717] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0718] [Third embodiment]

[0719] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0720] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0721] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0722] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0723] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0724] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0725] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0726] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0727] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0728] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0729] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0730] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0731] This invention is a system for supporting the memory of dementia patients and recommending optimal actions. Specifically, the system collects data on dementia patients, analyzes it using a generative AI model, and generates memory-reconstruction content based on the results, which is then displayed on a smartphone. Furthermore, the system recommends optimal actions based on the patient's current condition and notifies the smartphone.

[0732] System Configuration

[0733] The system consists of the following main components:

[0734] 1. Data Collection Methods

[0735] 2. Generative AI Models

[0736] 3. Memory Recreation Content Generation Method

[0737] 4. Smartphone display and notification methods

[0738] Data collection methods

[0739] Users (patients or their family members) use a smartphone app to upload data such as past photos, videos, notes, and conversation history.

[0740] Generative AI Models

[0741] The server receives the data sent by the user and analyzes it using a generative AI model, which automatically generates photo captions and related stories based on the data.

[0742] Memory reproduction content generation means

[0743] The server creates memory-recreating content based on the analysis results, including adding captions related to photos and generating explanatory text about past episodes.

[0744] Smartphone display and notification methods

[0745] The device (smartphone) displays the memory-recreating content sent from the server to the user. The server then recommends optimal actions and sends a notification to the device. The device then prompts the user to take action via push notifications or alert messages.

[0746] Specific examples

[0747] A user opens a smartphone app and uploads photos from a previous trip, which the device then sends to the server.

[0748] The server analyzes the image file and its metadata and generates a caption, such as "This photo was taken in front of the Eiffel Tower during a trip to France in the summer of 2019." The server then sends the memory-recreating content, including the caption, to the device, which displays it to the user, allowing them to relive their memories by looking at the photo.

[0749] Also, if the server generates a recommendation that "It's a good time to take a walk," the device will send a notification saying, "It's time for a walk now. Let's go to the nearest park." By carrying out the suggested action, the user who receives this recommendation can make their daily life more fulfilling.

[0750] The system will help dementia patients relive their past memories and provide a sense of security, while also making it easier for family and caregivers to provide optimal support, improving overall quality of life.

[0751] The processing flow will be explained below.

[0752] Step 1:

[0753] Users can launch a smartphone application and upload data such as past photos, videos, notes, conversation history, etc. Users can also select photos and enter captions and tags.

[0754] Step 2:

[0755] The device processes the uploaded data and prepares it for transmission to the server. Specifically, the device compresses the image files and adds appropriate metadata (e.g., date and time of the photo, location).

[0756] Step 3:

[0757] The device sends the prepared data to the server, which then sends a POST request to the server with the image and its metadata to the specified API endpoint.

[0758] Step 4:

[0759] The server stores the received data in storage. Specifically, the server stores the image file path and metadata in a database, and stores the image itself in cloud storage.

[0760] Step 5:

[0761] The server analyzes the stored data using a generative AI model. Specifically, the server inputs an image into the AI ​​model and generates an episode related to the image (e.g., the background of the photo, information about the people in the photo).

[0762] Step 6:

[0763] The server stores the analysis results in a database. Specifically, the server stores the analysis results (e.g., "This is a photo I took with my family at the beach last summer") in the database.

[0764] Step 7:

[0765] The server generates memory-recreating content based on the analysis results. Specifically, the server adds captions to images and generates related episodes as text.

[0766] Step 8:

[0767] The server sends the generated memory reproduction content to the device. Specifically, the server sends the generated captions along with the images in JSON format to the device.

[0768] Step 9:

[0769] The device displays the memory reproduction content received from the server to the user. Specifically, the device opens an image viewer and displays the image and its caption on the screen.

[0770] Step 10:

[0771] The server uses a generative AI model to recommend optimal actions based on the patient's condition and past data analysis results. Specifically, the server generates recommendations such as "Taking a walk at 3 p.m."

[0772] Step 11:

[0773] The server sends the generated action recommendation to the terminal. Specifically, the server sends the action recommendation to the terminal as a notification message.

[0774] Step 12:

[0775] The device notifies the user of recommended activities. Specifically, the device displays a push notification with the message, "It's time for a walk. Let's go to your favorite park."

[0776] Example 1

[0777] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0778] Dementia patients have difficulty retaining their memories and are prone to confusion in their daily lives. Caregivers and families also have limited information to provide the necessary support. Furthermore, a lack of appropriate behavioral instructions based on the patient's current condition can lead to a decline in quality of life. There is a need to address these issues.

[0779] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0780] In this invention, the server includes: means for a user to upload information such as past photos, videos, notes, and conversation history using a smartphone app; means for receiving and saving the information sent from the smartphone app; means for analyzing the saved information and generating photo captions and related episodes using a generative AI model; means for generating memory reproduction content based on the analysis results; means for sending the memory reproduction content to the smartphone; means for displaying the memory reproduction content on the smartphone; means for recommending optimal actions based on the patient's current condition; and means for notifying the smartphone of the recommended actions. This makes it easier for dementia patients to recall their memories, making it easier for caregivers and family members to provide the necessary support and providing appropriate instructions for actions based on the patient's current condition.

[0781] "User" refers to a person with dementia or their family member who uses the system.

[0782] "Smartphone app" refers to application software that runs on a smartphone and allows users to upload past information and receive memory recall content and behavioral recommendations.

[0783] "Information" refers to data uploaded by users, such as past photos, videos, notes, and conversation history.

[0784] "Server" refers to a computer system that receives, stores, and analyzes information sent by users, generates captions and episodes using generative AI models, and transmits memory reproduction content and behavioral recommendations to smartphones.

[0785] A "generative AI model" refers to an artificial intelligence model that analyzes information sent by users and automatically generates photo captions and related stories.

[0786] "Caption" refers to explanatory text associated with a photograph.

[0787] An "episode" is a descriptive text about a past event related to a photo or video.

[0788] "Memory recall content" refers to content that includes the analyzed photo captions and related episode text.

[0789] "Action recommendation" refers to a recommendation of the optimal action that is generated based on the patient's current condition.

[0790] "Notifications" refers to alerts sent to your smartphone with recommended actions or other important information.

[0791] This invention is a system for supporting memory and recommending optimal actions for dementia patients. The system consists of the following main components:

[0792] 1. Data Collection Methods

[0793] 2. Generative AI Models

[0794] 3. Memory Recreation Content Generation Method

[0795] 4. Smartphone display and notification methods

[0796] Data collection methods

[0797] Users upload information such as past photos, videos, memos, and conversation histories using a smartphone app. The smartphone app then sends the files selected by the user to the server via an HTTP request.

[0798] Generative AI Models

[0799] The server receives and stores information sent from the smartphone app. The stored information is then analyzed using a generative AI model, such as the latest natural language processing AI, such as GPT-4.

[0800] The following prompts are used during parsing:

[0801] text

[0802] This is a user-uploaded photo. Please provide information about the photo:

[0803] [Photo metadata (date, time, location, etc.)]

[0804] Based on this prompt, the generative AI model automatically generates captions for the photos and related stories.

[0805] Memory reproduction content generation means

[0806] The server generates memory-recreating content based on the results of analysis by the generative AI model. This content includes adding captions related to photos and generating explanatory text about past episodes. The generated content is combined in HTML or JSON format and organized in a format that is easy for users to access.

[0807] Smartphone display and notification methods

[0808] The device (smartphone) displays the memory replay content sent from the server to the user. By viewing this content, the user can relive past memories. The server also recommends optimal actions based on the patient's current condition and sends a notification of this recommendation to the device. The device then communicates this recommendation to the user in the form of a push notification or an alert message.

[0809] Specific examples

[0810] Consider a case where a user wants to upload photos from a previous trip. The user opens a smartphone app, selects a travel photo, and taps the upload button. This causes the device to send data including the photo to the server.

[0811] The server analyzes the image file and its metadata and generates a caption, such as "This photo was taken in front of the Eiffel Tower during a trip to France in the summer of 2019." Based on this, memory-recreating content is created and sent to the device. The device then displays this content to the user, allowing them to relive their past memories.

[0812] Furthermore, if the server generates a recommendation that "It's a good time to take a walk," the device will notify the user, "It's time for a walk now. Let's go to the nearest park." By carrying out the suggested action, the user who receives this recommendation can make their daily life more fulfilling.

[0813] This system will help dementia patients relive their past memories and provide a sense of security, while also making it easier for family and caregivers to provide optimal support, improving their overall quality of life.

[0814] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0815] Step 1:

[0816] A user uses a smartphone app to upload information such as past photos, videos, notes, and conversation history. Specifically, the user opens the smartphone app and taps the upload button to launch a file browser. When the user selects the files they want to upload, the device sends the selected files to the server via an HTTP request.

[0817] Input: Files such as past photos, videos, notes, and conversation history

[0818] Output: HTTP request from the terminal to the server

[0819] Step 2:

[0820] The server receives and stores data sent from the device. The server processes the received HTTP request and reads the sent file. The read data is stored in cloud storage or a database.

[0821] Input: File data sent from the terminal

[0822] Output: Saved file data

[0823] Step 3:

[0824] The server analyzes the stored data and generates photo captions and related episodes using a generative AI model. The server checks the file type and metadata and generates prompts for analysis. When the generated prompts are input into the generative AI model, the AI ​​model generates photo captions and episodes.

[0825] Input: Saved file data, prompt text

[0826] Output: Generated photo captions and related episodes

[0827] Step 4:

[0828] The server generates memory reproduction content based on the analysis results. The server integrates the analyzed captions and episodes into content in HTML or JSON format. This content is organized in a format that is easy for users to view.

[0829] Input: Generated photo captions, related episodes

[0830] Output: Memory-reproduced content (HTML or JSON format)

[0831] Step 5:

[0832] The device receives the memory reproduction content sent from the server and displays it to the user. The device receives the HTTP response and displays the content in the response within the app. By viewing this, the user can relive past memories.

[0833] Input: Memory-reproduced content sent from the server

[0834] Output: Memory-reproduced content displayed within the app

[0835] Step 6:

[0836] The server recommends the optimal action based on the patient's current condition and sends a notification to the device. The server analyzes past data and the patient's current condition to generate a recommendation such as "It's a good idea to take a walk at this time of day." The generated recommendation is sent to the device via an HTTP request.

[0837] Input: Current patient status, historical data

[0838] Output: Generated action recommendations

[0839] Step 7:

[0840] The device receives the recommended actions sent from the server and notifies the user. The device receives the HTTP response and displays the content to the user as a push notification or alert message. The user can check the notification and carry out the suggested actions to enrich their daily life.

[0841] Input: Action recommendations sent from the server

[0842] Output: Push notification or alert message to the user

[0843] (Application example 1)

[0844] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0845] Due to memory decline, dementia patients often forget past events and the locations of their daily activities. They may also get lost or find themselves in dangerous situations when out and about, which reduces the patient's quality of life and increases the anxiety of their families and caregivers. Conventional systems are unable to adequately address these issues, making emergency response and ensuring patient safety particularly challenging.

[0846] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0847] In this invention, the server includes means for collecting data on dementia patients, means for using a generative AI model to analyze the data, means for generating memory reproduction content based on the analysis results, means for transmitting the memory reproduction content to a smartphone, means for displaying the memory reproduction content on the smartphone, means for recommending optimal actions, means for notifying the smartphone of the recommended actions, means for collecting location information to track the patient's location, means for monitoring the surrounding environment using a camera and a microphone, means for issuing an emergency alert when a dangerous situation is detected, and means for sending notifications to caregivers in real time. This makes it possible to support the memory of dementia patients, recommend optimal actions, and ensure the safety of the patients.

[0848] A "dementia patient" is someone whose cognitive function has declined, resulting in memory loss, impaired judgment, and difficulty in daily life.

[0849] "Data" refers to information such as photos, videos, notes, and conversation histories related to dementia patients.

[0850] A "generative AI model" is an artificial intelligence model that analyzes collected data and generates the necessary information.

[0851] "Memory reproduction content" refers to content such as captions and episode texts that are created based on data analyzed by a generative AI model to reproduce a patient's memories.

[0852] A "smartphone" is a mobile device that can use a wide variety of applications in addition to the functions of a mobile phone.

[0853] "Action recommendations" are optimal actions suggested based on the patient's current situation and past data.

[0854] "Location information collection means" refers to a means for tracking the patient's current location using GPS or other means.

[0855] An "environmental monitoring means" is a means for monitoring the environment surrounding the patient using a camera or microphone.

[0856] An "emergency alert means" is a means for issuing a warning when a dangerous situation is detected.

[0857] "Caregivers" are family members or professionals who play a role in helping people with dementia on a day-to-day basis.

[0858] "Real-time" refers to processing occurring almost at the same time as the data is generated.

[0859] This invention is a system for supporting the memory of dementia patients and recommending optimal actions. Specifically, the system collects the patient's past data and analyzes it using a generative AI model to generate memory reproduction content and display it on a smartphone. It also has the function of recommending optimal actions based on the patient's current condition and notifying the smartphone.

[0860] The server first provides a means to collect data on dementia patients. This data, such as past photos, videos, notes, and conversation histories, is uploaded by patients and their families using a smartphone app. The server receives this data and analyzes it using a generative AI model. The generative AI model automatically generates photo captions and text for related episodes based on the collected data.

[0861] The server then generates memory-recreating content based on the analysis results. The generated memory-recreating content is sent to a smartphone, where it is displayed on the device. For example, if a past travel photo is uploaded, the generative AI model analyzes the photo and generates a caption such as, "This photo was taken at a tourist spot during a trip in the summer of 2019." The device then displays the memory-recreating content, including the caption, to the user, allowing them to view the photo and relive their memory.

[0862] Furthermore, the server has the function of recommending optimal actions. Based on past data and the patient's current condition, the generative AI model considers the current time of day and the patient's activity patterns to suggest appropriate actions. For example, a recommendation such as "It's a good idea to take a walk at this time of day" may be generated. This recommendation is sent as a push notification to the smartphone, displaying a message saying, "It's time for a walk now. Let's go to the nearest park."

[0863] The system also includes a location information collection means for tracking the patient's location, an environmental monitoring means for monitoring the patient's surroundings using a camera and microphone, an emergency alert means for issuing a warning when a dangerous situation is detected, and a means for sending a notification to a caregiver in real time. For example, if a patient goes out alone to a park, the generative AI model will track the patient's location using GPS and monitor the surrounding environment using a camera and microphone. Based on this information, the generative AI model will detect a dangerous situation and send a real-time notification to the caregiver saying, "Emergency: Patient may be in danger."

[0864] Examples of prompt sentences include the following:

[0865] "Analyze the photos and audio data sent by the user and generate a description of the environment and memory-recreating content. For example, if the sent photo was taken in a park, generate a description such as, 'This is a park near my house. I've been there before with my family.'"

[0866] This makes it easier for people with dementia to relive their past memories and feel more secure, and it also makes it easier for family and caregivers to ensure the patient's safety, improving their overall quality of life.

[0867] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0868] Step 1:

[0869] A user uses a smartphone app to upload data such as past photos, videos, notes, and conversation history. The input is the data uploaded by the user, and the output is the data sent to the server.

[0870] Step 2:

[0871] The server receives the data submitted by the user. The input is the data submitted by the user, and the output is the data to be fed into the generative AI model. The server converts this into a format suitable for analysis.

[0872] Step 3:

[0873] The server analyzes the received data using a generative AI model. The input is the received data, and the output is the analysis result. The generative AI model analyzes the content and metadata of the photo and generates captions and episode text.

[0874] Step 4:

[0875] The server generates memory-recreation content based on the analysis results. The input is the analysis results, and the output is the memory-recreation content. This content includes the generated captions and episode text.

[0876] Step 5:

[0877] The server transmits the generated memory reproduction content to the smartphone. The input is the memory reproduction content, and the output is the data transmitted to the smartphone.

[0878] Step 6:

[0879] The terminal displays the memory reproduction content sent from the server to the user. The input is the memory reproduction content, and the output is the screen displayed to the user. The terminal visually displays captions and episode text.

[0880] Step 7:

[0881] The server recommends the optimal action. The input is historical data and the patient's current condition, and the output is a recommended action. The generative AI model takes into account the current time of day and the patient's activity patterns to suggest appropriate actions.

[0882] Step 8:

[0883] The server notifies the smartphone of the recommended actions. The input is the recommended actions, and the output is a notification sent to the smartphone. The smartphone then suggests appropriate actions to the user through a push notification.

[0884] Step 9:

[0885] The terminal obtains real-time location information using a location information collection means that tracks the patient's location information. The input is GPS data, and the output is the patient's current location.

[0886] Step 10:

[0887] The device monitors the surrounding environment using a camera and microphone. The input is camera video and audio data, and the output is environmental information. The camera and microphone periodically collect data and provide it for analysis.

[0888] Step 11:

[0889] The server uses an emergency alert mechanism to issue a warning when a dangerous situation is detected. The input is location information and environmental information, and the output is a warning message. The generative AI model analyzes this data and generates a warning if a dangerous situation is detected.

[0890] Step 12:

[0891] The server sends real-time notifications to the caregiver: the input is the warning message and the output is the notification sent to the caregiver, allowing the caregiver to respond immediately.

[0892] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0893] This invention is a system for supporting the memory of dementia patients and recommending optimal actions. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide more personalized support. Specifically, this system has the function of analyzing the user's emotions and adjusting memory recall content and optimal action recommendations based on that.

[0894] System Configuration

[0895] The system consists of the following main components:

[0896] 1. Data Collection Methods

[0897] 2. Generative AI Models

[0898] 3. Memory Recreation Content Generation Method

[0899] 4. Smartphone display and notification methods

[0900] 5. Emotion Engine

[0901] Data collection methods

[0902] Users (patients or their family members) use a smartphone app to upload data such as past photos, videos, notes, and conversation history.

[0903] Generative AI Models

[0904] The server receives the data sent by the user and analyzes it using a generative AI model, which automatically generates photo captions and related stories based on the data.

[0905] Memory reproduction content generation means

[0906] The server creates memory-recreating content based on the analysis results, including adding captions related to photos and generating explanatory text about past episodes.

[0907] Smartphone display and notification methods

[0908] The device (smartphone) displays the memory-recreating content sent from the server to the user. The server then recommends optimal actions and sends a notification to the device. The device then prompts the user to take action via push notifications or alert messages.

[0909] Emotion Engine

[0910] The device has an emotion engine that recognizes the user's emotions. This emotion engine analyzes emotions based on the user's facial expressions, voice, and text input. The analyzed emotion information is sent to the server and reflected in memory reproduction content and behavioral recommendations.

[0911] Specific examples

[0912] A user opens a smartphone app and uploads photos from a previous trip, which the device then sends to the server.

[0913] The server analyzes the image file and its metadata and generates a caption, such as "This photo was taken in front of the Eiffel Tower during a trip to France in the summer of 2019." The server then sends the memory-recreating content, including the caption, to the device, which displays it to the user, allowing them to relive their memories by looking at the photo.

[0914] The device also recognizes emotions such as "happiness" and "surprise" through facial and voice analysis of the user. Based on this, the server generates a recommendation such as "It's time for a walk. Let's go to the nearest park" and sends it to the device. This recommendation is displayed at an appropriate time depending on the emotion.

[0915] For example, if the system recognizes that the user is feeling stressed, it will give a recommendation such as, "Why not take a break and read your favorite book?"

[0916] This system makes it easier for dementia patients to relive their past memories, providing a sense of security. It also recommends appropriate actions based on the user's emotions, further improving their quality of life. It also makes it easier for family members and caregivers to provide emotional support, enhancing overall care.

[0917] The processing flow will be explained below.

[0918] Step 1:

[0919] Users can launch a smartphone application and upload data such as past photos, videos, notes, conversation history, etc. Users can also select photos and enter captions and tags.

[0920] Step 2:

[0921] The device processes the uploaded data and prepares it for transmission to the server. Specifically, the device compresses the image files and adds appropriate metadata (e.g., date and time of the photo, location).

[0922] Step 3:

[0923] The device sends the prepared data to the server, which then sends a POST request to the server with the image and its metadata to the specified API endpoint.

[0924] Step 4:

[0925] The server stores the received data in storage. Specifically, the server stores the image file path and metadata in a database, and stores the image itself in cloud storage.

[0926] Step 5:

[0927] The server analyzes the stored data using a generative AI model. Specifically, the server inputs an image into the AI ​​model and generates an episode related to the image (e.g., the background of the photo, information about the people in the photo).

[0928] Step 6:

[0929] The server stores the analysis results in a database. Specifically, the server stores the analysis results (e.g., "This is a photo I took with my family at the beach last summer") in the database.

[0930] Step 7:

[0931] The server generates memory-recreating content based on the analysis results. Specifically, the server adds captions to images and generates related episodes as text.

[0932] Step 8:

[0933] The server sends the generated memory reproduction content to the device. Specifically, the server sends the generated captions along with the images in JSON format to the device.

[0934] Step 9:

[0935] The device displays the memory reproduction content received from the server to the user. Specifically, the device opens an image viewer and displays the image and its caption on the screen.

[0936] Step 10:

[0937] The device collects facial and voice data through a camera and microphone to recognize the user's emotions. Specifically, the device collects emotional data in real time while the user is browsing photos and stories.

[0938] Step 11:

[0939] The device analyzes the collected emotional data and identifies the user's current emotional state. Specifically, the emotion engine determines emotions such as "happiness," "surprise," and "sadness" from the user's facial expressions and voice.

[0940] Step 12:

[0941] The device sends the analyzed emotion data to the server by sending a POST request to the server's API endpoint in JSON format.

[0942] Step 13:

[0943] The server receives the emotion data and reflects it in memory-recreating content and optimal action recommendations. Specifically, it adjusts the content and recommendations to be displayed based on the emotion data.

[0944] Step 14:

[0945] The server uses a generative AI model to recommend optimal actions based on the patient's condition, past data analysis results, and emotional data. Specifically, it generates a recommendation such as "A walk is effective at the current time," but if the user is feeling stressed, it generates a different recommendation such as "Why don't you take a break and read your favorite book?"

[0946] Step 15:

[0947] The server sends the generated action recommendation to the terminal. Specifically, the server sends the action recommendation to the terminal as a notification message.

[0948] Step 16:

[0949] The device notifies the user of recommended actions. Specifically, it encourages the user to take action by means of push notifications or alert messages. For example, the device may notify the user, "It's time for a walk. Let's go to the nearest park."

[0950] Through these processing steps, this system supports the memory of dementia patients and improves their quality of life by recommending optimal actions based on their emotions. It also makes it easier for family members and caregivers to understand the patient's emotional state and provide more appropriate support.

[0951] Example 2

[0952] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0953] Dementia patients have difficulty retaining past memories and often experience stress in their daily lives. Furthermore, there are limited ways to encourage dementia patients to behave appropriately, placing a heavy burden on their families and caregivers. The problem this invention aims to solve is to improve the quality of life of dementia patients by supporting their memory recall and recommending optimal actions based on their emotions.

[0954] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user data, means for using a generative AI model to analyze the user data, means for generating memory reproduction content based on the analysis results, means for transmitting the memory reproduction content to a terminal, means for displaying the memory reproduction content on the terminal, means for analyzing the user's emotions, means for recommending optimal actions based on the user's emotional information, and means for notifying the terminal of the recommended actions. This allows dementia patients to easily reproduce past memories and receive appropriate recommended actions based on their emotions.

[0955] "User" refers to anyone who uses the system, including a dementia patient or their family or caregiver.

[0956] "Data collection means" refers to the ability of users to upload information such as photos, videos, notes, and conversation history using smartphone apps.

[0957] A "generative AI model" is an artificial intelligence technology that analyzes data collected from users and automatically generates photo captions and related stories.

[0958] "Memory reproduction content" is content that includes photo captions and text descriptions created based on the results of analysis by a generative AI model.

[0959] "Terminal" refers to a display device such as a smartphone or tablet used by a user.

[0960] "Emotion analysis means" is a technology that has the function of analyzing a user's facial expressions, voice, character input, etc., and recognizing the user's emotions.

[0961] "Behavioral recommendations" refer to specific actions suggested to users based on analyzed emotional information and past data.

[0962] The "notification means" is a function that sends the generated memory reproduction content and behavioral recommendations to the terminal as a push notification or an alert message and displays them to the user.

[0963] This invention is a system that supports memory for dementia patients and recommends optimal actions. By combining this system with an emotion engine that recognizes the user's emotions, it is possible to provide more personalized support.

[0964] System Configuration

[0965] The system consists of the following main components:

[0966] 1. Data Collection Methods

[0967] 2. Generative AI Models

[0968] 3. Memory Recreation Content Generation Method

[0969] 4. Terminal display and notification methods

[0970] 5. Emotion Engine

[0971] Data collection methods

[0972] Users use a smartphone app to upload data such as past photos, videos, memos, and conversation histories. For example, they can upload photos of a family trip or diary notes to the app. The device then sends this data to a server.

[0973] Generative AI Models

[0974] The server uses a generative AI model to analyze the data received from the device. This generative AI model is an advanced natural language processing model such as GPT-4. The server analyzes the metadata of the photo or video (e.g., date and time of the photo, location) and generates a caption based on that. For example, it generates a caption such as, "This photo was taken in front of the Eiffel Tower in Paris, France, in the summer of 2019."

[0975] Memory reproduction content generation means

[0976] The server generates memory-recreating content based on the analysis results. This content includes captions related to the photos and explanatory text about past episodes. For example, a photo of a trip to France might include a description such as, "On a beautiful summer day, my family and I visited the Eiffel Tower and then took a walk along the Seine."

[0977] Terminal display and notification means

[0978] The device displays the memory reproduction content sent from the server to the user. For example, the app's notification function can be used to notify the user, "A new memory has been created!" When the user clicks on the notification, the generated memory reproduction content is displayed within the app.

[0979] Emotion Engine

[0980] The device has an emotion engine that analyzes the user's facial expressions, voice, and text input to recognize emotions. For example, if the user smiles at a displayed photo, the device recognizes "joy." If the user writes "I had a lot of fun" when entering a comment, the device analyzes it as a "positive emotion."

[0981] Action recommendation generation and notification

[0982] The server generates optimal action recommendations based on the emotional information sent from the emotion engine. For example, if the user expresses the emotion "joy," the server generates a recommendation such as, "You seem to be in a good mood right now, so why not enjoy some tea at a nearby cafe?" The server sends this recommendation to the device. The device notifies the user of this recommendation, displaying, for example, "It's time for a walk now. Let's go to a nearby park."

[0983] Specific examples

[0984] Prompt Sentence Examples

[0985] Upload past travel photos and the system will analyze your emotions and recommend the best course of action.

[0986] Example

[0987] A user opens the smartphone app, selects a photo from their trip to France in 2019, and presses the upload button. The device sends this photo to the server. The server analyzes the received image and generates a caption such as, "This photo was taken in front of the Eiffel Tower during my trip to France in the summer of 2019." The server creates memory-recreating content including this caption and sends it to the device. The device notifies the user via push notification, "A new memory has been created!" When the user clicks the notification, the generated content is displayed within the app. The device analyzes the user's facial expression with an emotion engine and recognizes "joy." Based on this emotional information, the server generates an action recommendation such as, "It's time for a walk now. Let's go to the nearest park." The device notifies the user of this recommendation.

[0988] This system allows dementia patients to easily recall past memories and receive appropriate behavioral recommendations based on their emotions, enabling them to enrich their daily lives.

[0989] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0990] Step 1:

[0991] The user opens the smartphone app, selects past photos, videos, notes, and conversation history (input data), and presses the upload button. The device then sends this data to the server (specific operation of data transmission).

[0992] Step 2:

[0993] The server uses a generative AI model to analyze the data received from the device (input data). It uses a generative AI model (e.g., GPT-4) to check the metadata of the photo or video (e.g., date and time of the photo, location) and generates a caption (specific operation of data analysis and caption generation). For example, it generates a caption (output data) such as, "This photo was taken in front of the Eiffel Tower in Paris, France, in the summer of 2019."

[0994] Step 3:

[0995] The server creates memory-recreating content based on the generated captions (input data). This content includes captions related to the photos and explanatory text about past episodes. For example, to a photo of a trip to France, it adds the description (output data) "On a beautiful summer day, after visiting the Eiffel Tower with my family, we took a walk along the Seine River" (specific operation of generating memory-recreating content).

[0996] Step 4:

[0997] The server sends the generated memory reproduction content (input data) to the device. The device receives it and displays it to the user (specific actions for sending and displaying content). For example, the app's notification function can be used to notify the user, "A new memory has been created!" When the user clicks on the notification, the generated memory reproduction content (output data) is displayed within the app.

[0998] Step 5:

[0999] The device uses an emotion engine that analyzes the user's facial expressions, voice, and text input to recognize the user's emotions (input data). For example, if the user smiles at a displayed photo, the device recognizes "joy." If the user writes "I had a lot of fun" when entering a comment, the device analyzes it as a "positive emotion" (specific operation of emotion analysis).

[1000] Step 6:

[1001] The server generates optimal behavioral recommendations based on the emotional information (input data) sent from the emotion engine. For example, if the user expresses the emotion "joy," the server generates a behavioral recommendation (output data) such as "You seem to be in a good mood right now, so why don't you enjoy tea at a nearby cafe?" (Specific operation of behavioral recommendation generation).

[1002] Step 7:

[1003] The server sends the generated action recommendation (input data) to the device. The device displays this recommendation to the user as a push notification (specific operation of sending a recommendation and notification). For example, a message (output data) saying "It's time for a walk now. Let's go to a nearby park" is displayed.

[1004] In this way, each step of the system sequentially processes and analyzes data, and by providing the results to the user, it helps dementia patients recover their memories and improve their lives.

[1005] (Application example 2)

[1006] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1007] Current dementia patient support systems are inadequate in terms of memory recall and emotional recognition. Furthermore, in support of factory workers, they do not provide efficient work procedures or recommend breaks based on the worker's emotions, which prevents improvements in work efficiency and reduces the burden on workers. A comprehensive solution to these issues is needed.

[1008] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data on dementia patients, means for using a generative AI model to analyze the data, means for generating memory reproduction content based on the analysis results, means for collecting task data, means for using a work process analysis AI model to analyze the task data, and means for using an emotion recognition engine to recognize the user's emotions. This makes it possible to provide memory reproduction support to dementia patients and recommend efficient work procedures and emotion-based breaks to factory workers.

[1009] "Dementia" is a disease characterized by a decline in cognitive function that affects memory, judgment, communication, and daily activities.

[1010] "Data collection means" is a general term for any device or method for acquiring information from a user or object.

[1011] A "generative AI model" is an artificial intelligence algorithm that predicts, analyzes, and generates based on collected data.

[1012] "Memory reproduction content" refers to information and media materials that are used to reproduce a user's past memories based on analyzed data.

[1013] A "display terminal" is an electronic device such as a smartphone, tablet, or smart glasses that allows users to view and operate information.

[1014] "Behavioral recommendations" refer to notifications and advice that suggest optimal actions and options to users.

[1015] "Work data" is a set of information related to the progress, method, results, etc. of a work.

[1016] The "work process analysis AI model" is an artificial intelligence algorithm that analyzes work data and derives work efficiency and areas for improvement.

[1017] An "emotion recognition engine" is a technology for analyzing emotions based on a user's facial expressions, voice, text input, etc.

[1018] "Break recommendation" refers to notifications or advice that suggest taking a break at the optimal time based on the user's emotional state.

[1019] The system for implementing this invention provides integrated support for dementia patients and factory workers.

[1020] System Overview

[1021] This system includes a server, display terminals (smartphones, smart glasses, etc.), and a mobile robot within the factory. The server plays a central role in performing key processes such as data collection, analysis, generation, and display.

[1022] Key Components

[1023] 1. Data Collection Methods

[1024] 2. Generative AI Models

[1025] 3. Memory Recreation Content Generation Method

[1026] 4. Display and notification methods on display devices (smartphones, smart glasses, etc.)

[1027] 5. Emotion Recognition Engine

[1028] 6. Work Process Analysis AI Model

[1029] 1. Data Collection Methods

[1030] The dementia patient or worker uploads past photos, videos, notes, conversation history, and work data to a server through a data collection method, such as using a camera or microphone on a smartphone or smart glasses.

[1031] 2. Generative AI Models

[1032] The server receives the collected data and analyzes it using a generative AI model (e.g., TensorFlow, PyTorch), which automatically generates captions, anecdotes, and efficient work procedures from the user's past photos and work data.

[1033] 3. Memory Recreation Content Generation Method

[1034] Based on the analysis results, memory recall content is generated. For example, this includes adding captions to past photos and generating text about related episodes. In the case of work data, efficient work procedures are generated.

[1035] 4. Display terminal display and notification means

[1036] The memory reproduction content and work procedures sent from the server are displayed on a display device, and notifications are sent to the user via a smartphone or smart glasses, for example.

[1037] 5. Emotion Recognition Engine

[1038] The system analyzes the user's emotions in real time using a camera and microphone installed on the display device (smartphone, smart glasses). The analyzed emotional information is sent to a server and reflected in memory reproduction content and work procedure recommendations.

[1039] 6. Work Process Analysis AI Model

[1040] The server analyzes the work data using a work process analysis AI model (e.g., Keras, Scikit-learn) to extract efficient work procedures and areas for improvement. This information is then appropriately notified to the workers.

[1041] Specific examples

[1042] Let's say a user wears smart glasses and performs work in a factory. The work progress and sensor data are collected and sent to a server. The server analyzes this data using an AI model for work process analysis and generates work instructions such as, "This work usually takes 15 minutes to complete. Please try doing it this way."

[1043] The smart glasses also use a camera and microphone to analyze the worker's facial expressions and voice, which are then analyzed by an emotion recognition engine. If stress is detected, the system will recommend a break, such as "Take a 5-minute break now."

[1044] Prompt Sentence Examples

[1045] Build a system that recognizes workers' emotions in real time and sends them notifications to encourage them to take a break if they feel stressed or impatient, by using smart glasses on a parts assembly line and analyzing past work data to generate specific recommendations for optimal procedures.

[1046] As described above, this invention realizes a system that can restore memories, provide efficient work procedures, and recommend optimal actions based on emotions for both dementia patients and factory workers.

[1047] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1048] Step 1: Data collection

[1049] A user uses smart glasses or a smartphone to collect past data (e.g., photos, videos, notes, conversation history, and work data). This data is sent to a server by a data collection means. The input data is photos and work progress, and the output data is information stored in a database on the server.

[1050] Step 2: Data analysis

[1051] The server analyzes the received data using a generative AI model (e.g., TensorFlow, PyTorch). The input data is the user's data stored on the server, and the output data is the analysis results: captions, episodes, and work procedures. Specifically, it generates captions for photos and extracts efficient procedures from work data.

[1052] Step 3: Generate memory recall content

[1053] The server generates memory-recreating content based on the analysis results. The input data is the analysis results from the generative AI model, and the output data is the memory-recreating content (for example, photo captions or episode text). Specifically, it adds captions to photos and generates work procedures as text.

[1054] Step 4: Send to display device

[1055] The server sends the generated memory reproduction content and work procedures to a display device (smartphone or smart glasses). The input data is the memory reproduction content and work procedures, and the output data is the content displayed on the device.

[1056] Step 5: Emotion Recognition

[1057] To recognize the user's emotions, the device (smart glasses or smartphone) uses a camera and microphone to collect the user's facial expression and voice data. This data is analyzed by an emotion recognition engine. The input data is the user's facial expression and voice, and the output data is the emotion analysis results. Specifically, the device analyzes the user's emotional state, such as stress or joy.

[1058] Step 6: Generate action recommendations

[1059] The server recommends optimal actions to the user based on the results of emotion analysis and past data. The input data are the results of emotion analysis and past data, and the output data is the recommended action. Specifically, notifications such as "Take a 5-minute break now" or "Try this work procedure" are generated.

[1060] Step 7: Notify the display device

[1061] The server notifies the generated action recommendation to the display device. The input data is the action recommendation, and the output data is the notification displayed on the device. Specifically, it is displayed as a push notification on smart glasses or a smartphone.

[1062] In this way, it is possible to support memory recall for dementia patients, provide efficient work procedures for factory workers, and even recommend optimal actions based on emotions.

[1063] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1064] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1065] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1066] [Fourth embodiment]

[1067] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1068] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1069] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1070] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1071] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1072] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1073] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1074] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1075] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1076] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1077] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1078] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1079] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1080] This invention is a system for supporting the memory of dementia patients and recommending optimal actions. Specifically, the system collects data on dementia patients, analyzes it using a generative AI model, and generates memory-reconstruction content based on the results, which is then displayed on a smartphone. Furthermore, the system recommends optimal actions based on the patient's current condition and notifies the smartphone.

[1081] System Configuration

[1082] The system consists of the following main components:

[1083] 1. Data Collection Methods

[1084] 2. Generative AI Models

[1085] 3. Memory Recreation Content Generation Method

[1086] 4. Smartphone display and notification methods

[1087] Data collection methods

[1088] Users (patients or their family members) use a smartphone app to upload data such as past photos, videos, notes, and conversation history.

[1089] Generative AI Models

[1090] The server receives the data sent by the user and analyzes it using a generative AI model, which automatically generates photo captions and related stories based on the data.

[1091] Memory reproduction content generation means

[1092] The server creates memory-recreating content based on the analysis results, including adding captions related to photos and generating explanatory text about past episodes.

[1093] Smartphone display and notification methods

[1094] The device (smartphone) displays the memory-recreating content sent from the server to the user. The server then recommends optimal actions and sends a notification to the device. The device then prompts the user to take action via push notifications or alert messages.

[1095] Specific examples

[1096] A user opens a smartphone app and uploads photos from a previous trip, which the device then sends to the server.

[1097] The server analyzes the image file and its metadata and generates a caption, such as "This photo was taken in front of the Eiffel Tower during a trip to France in the summer of 2019." The server then sends the memory-recreating content, including the caption, to the device, which displays it to the user, allowing them to relive their memories by looking at the photo.

[1098] Also, if the server generates a recommendation that "It's a good time to take a walk," the device will send a notification saying, "It's time for a walk now. Let's go to the nearest park." By carrying out the suggested action, the user who receives this recommendation can make their daily life more fulfilling.

[1099] The system will help dementia patients relive their past memories and provide a sense of security, while also making it easier for family and caregivers to provide optimal support, improving overall quality of life.

[1100] The processing flow will be explained below.

[1101] Step 1:

[1102] Users can launch a smartphone application and upload data such as past photos, videos, notes, conversation history, etc. Users can also select photos and enter captions and tags.

[1103] Step 2:

[1104] The device processes the uploaded data and prepares it for transmission to the server. Specifically, the device compresses the image files and adds appropriate metadata (e.g., date and time of the photo, location).

[1105] Step 3:

[1106] The device sends the prepared data to the server, which then sends a POST request to the server with the image and its metadata to the specified API endpoint.

[1107] Step 4:

[1108] The server stores the received data in storage. Specifically, the server stores the image file path and metadata in a database, and stores the image itself in cloud storage.

[1109] Step 5:

[1110] The server analyzes the stored data using a generative AI model. Specifically, the server inputs an image into the AI ​​model and generates an episode related to the image (e.g., the background of the photo, information about the people in the photo).

[1111] Step 6:

[1112] The server stores the analysis results in a database. Specifically, the server stores the analysis results (e.g., "This is a photo I took with my family at the beach last summer") in the database.

[1113] Step 7:

[1114] The server generates memory-recreating content based on the analysis results. Specifically, the server adds captions to images and generates related episodes as text.

[1115] Step 8:

[1116] The server sends the generated memory reproduction content to the device. Specifically, the server sends the generated captions along with the images in JSON format to the device.

[1117] Step 9:

[1118] The device displays the memory reproduction content received from the server to the user. Specifically, the device opens an image viewer and displays the image and its caption on the screen.

[1119] Step 10:

[1120] The server uses a generative AI model to recommend optimal actions based on the patient's condition and past data analysis results. Specifically, the server generates recommendations such as "Taking a walk at 3 p.m."

[1121] Step 11:

[1122] The server sends the generated action recommendation to the terminal. Specifically, the server sends the action recommendation to the terminal as a notification message.

[1123] Step 12:

[1124] The device notifies the user of recommended activities. Specifically, the device displays a push notification with the message, "It's time for a walk. Let's go to your favorite park."

[1125] Example 1

[1126] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1127] Dementia patients have difficulty retaining their memories and are prone to confusion in their daily lives. Caregivers and families also have limited information to provide the necessary support. Furthermore, a lack of appropriate behavioral instructions based on the patient's current condition can lead to a decline in quality of life. There is a need to address these issues.

[1128] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1129] In this invention, the server includes: means for a user to upload information such as past photos, videos, notes, and conversation history using a smartphone app; means for receiving and saving the information sent from the smartphone app; means for analyzing the saved information and generating photo captions and related episodes using a generative AI model; means for generating memory reproduction content based on the analysis results; means for sending the memory reproduction content to the smartphone; means for displaying the memory reproduction content on the smartphone; means for recommending optimal actions based on the patient's current condition; and means for notifying the smartphone of the recommended actions. This makes it easier for dementia patients to recall their memories, making it easier for caregivers and family members to provide the necessary support and providing appropriate instructions for actions based on the patient's current condition.

[1130] "User" refers to a person with dementia or their family member who uses the system.

[1131] "Smartphone app" refers to application software that runs on a smartphone and allows users to upload past information and receive memory recall content and behavioral recommendations.

[1132] "Information" refers to data uploaded by users, such as past photos, videos, notes, and conversation history.

[1133] "Server" refers to a computer system that receives, stores, and analyzes information sent by users, generates captions and episodes using generative AI models, and transmits memory reproduction content and behavioral recommendations to smartphones.

[1134] A "generative AI model" refers to an artificial intelligence model that analyzes information sent by users and automatically generates photo captions and related stories.

[1135] "Caption" refers to explanatory text associated with a photograph.

[1136] An "episode" is a descriptive text about a past event related to a photo or video.

[1137] "Memory recall content" refers to content that includes the analyzed photo captions and related episode text.

[1138] "Action recommendation" refers to a recommendation of the optimal action that is generated based on the patient's current condition.

[1139] "Notifications" refers to alerts sent to your smartphone with recommended actions or other important information.

[1140] This invention is a system for supporting memory and recommending optimal actions for dementia patients. The system consists of the following main components:

[1141] 1. Data Collection Methods

[1142] 2. Generative AI Models

[1143] 3. Memory Recreation Content Generation Method

[1144] 4. Smartphone display and notification methods

[1145] Data collection methods

[1146] Users upload information such as past photos, videos, memos, and conversation histories using a smartphone app. The smartphone app then sends the files selected by the user to the server via an HTTP request.

[1147] Generative AI Models

[1148] The server receives and stores information sent from the smartphone app. The stored information is then analyzed using a generative AI model, such as the latest natural language processing AI, such as GPT-4.

[1149] The following prompts are used during parsing:

[1150] text

[1151] This is a user-uploaded photo. Please provide information about the photo:

[1152] [Photo metadata (date, time, location, etc.)]

[1153] Based on this prompt, the generative AI model automatically generates captions for the photos and related stories.

[1154] Memory reproduction content generation means

[1155] The server generates memory-recreating content based on the results of analysis by the generative AI model. This content includes adding captions related to photos and generating explanatory text about past episodes. The generated content is combined in HTML or JSON format and organized in a format that is easy for users to access.

[1156] Smartphone display and notification methods

[1157] The device (smartphone) displays the memory replay content sent from the server to the user. By viewing this content, the user can relive past memories. The server also recommends optimal actions based on the patient's current condition and sends a notification of this recommendation to the device. The device then communicates this recommendation to the user in the form of a push notification or an alert message.

[1158] Specific examples

[1159] Consider a case where a user wants to upload photos from a previous trip. The user opens a smartphone app, selects a travel photo, and taps the upload button. This causes the device to send data including the photo to the server.

[1160] The server analyzes the image file and its metadata and generates a caption, such as "This photo was taken in front of the Eiffel Tower during a trip to France in the summer of 2019." Based on this, memory-recreating content is created and sent to the device. The device then displays this content to the user, allowing them to relive their past memories.

[1161] Furthermore, if the server generates a recommendation that "It's a good time to take a walk," the device will notify the user, "It's time for a walk now. Let's go to the nearest park." By carrying out the suggested action, the user who receives this recommendation can make their daily life more fulfilling.

[1162] This system will help dementia patients relive their past memories and provide a sense of security, while also making it easier for family and caregivers to provide optimal support, improving their overall quality of life.

[1163] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1164] Step 1:

[1165] A user uses a smartphone app to upload information such as past photos, videos, notes, and conversation history. Specifically, the user opens the smartphone app and taps the upload button to launch a file browser. When the user selects the files they want to upload, the device sends the selected files to the server via an HTTP request.

[1166] Input: Files such as past photos, videos, notes, and conversation history

[1167] Output: HTTP request from the terminal to the server

[1168] Step 2:

[1169] The server receives and stores data sent from the device. The server processes the received HTTP request and reads the sent file. The read data is stored in cloud storage or a database.

[1170] Input: File data sent from the terminal

[1171] Output: Saved file data

[1172] Step 3:

[1173] The server analyzes the stored data and generates photo captions and related episodes using a generative AI model. The server checks the file type and metadata and generates prompts for analysis. When the generated prompts are input into the generative AI model, the AI ​​model generates photo captions and episodes.

[1174] Input: Saved file data, prompt text

[1175] Output: Generated photo captions and related episodes

[1176] Step 4:

[1177] The server generates memory reproduction content based on the analysis results. The server integrates the analyzed captions and episodes into content in HTML or JSON format. This content is organized in a format that is easy for users to view.

[1178] Input: Generated photo captions, related episodes

[1179] Output: Memory-reproduced content (HTML or JSON format)

[1180] Step 5:

[1181] The device receives the memory reproduction content sent from the server and displays it to the user. The device receives the HTTP response and displays the content in the response within the app. By viewing this, the user can relive past memories.

[1182] Input: Memory-reproduced content sent from the server

[1183] Output: Memory-reproduced content displayed within the app

[1184] Step 6:

[1185] The server recommends the optimal action based on the patient's current condition and sends a notification to the device. The server analyzes past data and the patient's current condition to generate a recommendation such as "It's a good idea to take a walk at this time of day." The generated recommendation is sent to the device via an HTTP request.

[1186] Input: Current patient status, historical data

[1187] Output: Generated action recommendations

[1188] Step 7:

[1189] The device receives the recommended actions sent from the server and notifies the user. The device receives the HTTP response and displays the content to the user as a push notification or alert message. The user can check the notification and carry out the suggested actions to enrich their daily life.

[1190] Input: Action recommendations sent from the server

[1191] Output: Push notification or alert message to the user

[1192] (Application example 1)

[1193] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1194] Due to memory decline, dementia patients often forget past events and the locations of their daily activities. They may also get lost or find themselves in dangerous situations when out and about, which reduces the patient's quality of life and increases the anxiety of their families and caregivers. Conventional systems are unable to adequately address these issues, making emergency response and ensuring patient safety particularly challenging.

[1195] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1196] In this invention, the server includes means for collecting data on dementia patients, means for using a generative AI model to analyze the data, means for generating memory reproduction content based on the analysis results, means for transmitting the memory reproduction content to a smartphone, means for displaying the memory reproduction content on the smartphone, means for recommending optimal actions, means for notifying the smartphone of the recommended actions, means for collecting location information to track the patient's location, means for monitoring the surrounding environment using a camera and a microphone, means for issuing an emergency alert when a dangerous situation is detected, and means for sending notifications to caregivers in real time. This makes it possible to support the memory of dementia patients, recommend optimal actions, and ensure the safety of the patients.

[1197] A "dementia patient" is someone whose cognitive function has declined, resulting in memory loss, impaired judgment, and difficulty in daily life.

[1198] "Data" refers to information such as photos, videos, notes, and conversation histories related to dementia patients.

[1199] A "generative AI model" is an artificial intelligence model that analyzes collected data and generates the necessary information.

[1200] "Memory reproduction content" refers to content such as captions and episode texts that are created based on data analyzed by a generative AI model to reproduce a patient's memories.

[1201] A "smartphone" is a mobile device that can use a wide variety of applications in addition to the functions of a mobile phone.

[1202] "Action recommendations" are optimal actions suggested based on the patient's current situation and past data.

[1203] "Location information collection means" refers to a means for tracking the patient's current location using GPS or other means.

[1204] An "environmental monitoring means" is a means for monitoring the environment surrounding the patient using a camera or microphone.

[1205] An "emergency alert means" is a means for issuing a warning when a dangerous situation is detected.

[1206] "Caregivers" are family members or professionals who play a role in helping people with dementia on a day-to-day basis.

[1207] "Real-time" refers to processing occurring almost at the same time as the data is generated.

[1208] This invention is a system for supporting the memory of dementia patients and recommending optimal actions. Specifically, the system collects the patient's past data and analyzes it using a generative AI model to generate memory reproduction content and display it on a smartphone. It also has the function of recommending optimal actions based on the patient's current condition and notifying the smartphone.

[1209] The server first provides a means to collect data on dementia patients. This data, such as past photos, videos, notes, and conversation histories, is uploaded by patients and their families using a smartphone app. The server receives this data and analyzes it using a generative AI model. The generative AI model automatically generates photo captions and text for related episodes based on the collected data.

[1210] The server then generates memory-recreating content based on the analysis results. The generated memory-recreating content is sent to a smartphone, where it is displayed on the device. For example, if a past travel photo is uploaded, the generative AI model analyzes the photo and generates a caption such as, "This photo was taken at a tourist spot during a trip in the summer of 2019." The device then displays the memory-recreating content, including the caption, to the user, allowing them to view the photo and relive their memory.

[1211] Furthermore, the server has the function of recommending optimal actions. Based on past data and the patient's current condition, the generative AI model considers the current time of day and the patient's activity patterns to suggest appropriate actions. For example, a recommendation such as "It's a good idea to take a walk at this time of day" may be generated. This recommendation is sent as a push notification to the smartphone, displaying a message saying, "It's time for a walk now. Let's go to the nearest park."

[1212] The system also includes a location information collection means for tracking the patient's location, an environmental monitoring means for monitoring the patient's surroundings using a camera and microphone, an emergency alert means for issuing a warning when a dangerous situation is detected, and a means for sending a notification to a caregiver in real time. For example, if a patient goes out alone to a park, the generative AI model will track the patient's location using GPS and monitor the surrounding environment using a camera and microphone. Based on this information, the generative AI model will detect a dangerous situation and send a real-time notification to the caregiver saying, "Emergency: Patient may be in danger."

[1213] Examples of prompt sentences include the following:

[1214] "Analyze the photos and audio data sent by the user and generate a description of the environment and memory-recreating content. For example, if the sent photo was taken in a park, generate a description such as, 'This is a park near my house. I've been there before with my family.'"

[1215] This makes it easier for people with dementia to relive their past memories and feel more secure, and it also makes it easier for family and caregivers to ensure the patient's safety, improving their overall quality of life.

[1216] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1217] Step 1:

[1218] A user uses a smartphone app to upload data such as past photos, videos, notes, and conversation history. The input is the data uploaded by the user, and the output is the data sent to the server.

[1219] Step 2:

[1220] The server receives the data submitted by the user. The input is the data submitted by the user, and the output is the data to be fed into the generative AI model. The server converts this into a format suitable for analysis.

[1221] Step 3:

[1222] The server analyzes the received data using a generative AI model. The input is the received data, and the output is the analysis result. The generative AI model analyzes the content and metadata of the photo and generates captions and episode text.

[1223] Step 4:

[1224] The server generates memory-recreation content based on the analysis results. The input is the analysis results, and the output is the memory-recreation content. This content includes the generated captions and episode text.

[1225] Step 5:

[1226] The server transmits the generated memory reproduction content to the smartphone. The input is the memory reproduction content, and the output is the data transmitted to the smartphone.

[1227] Step 6:

[1228] The terminal displays the memory reproduction content sent from the server to the user. The input is the memory reproduction content, and the output is the screen displayed to the user. The terminal visually displays captions and episode text.

[1229] Step 7:

[1230] The server recommends the optimal action. The input is historical data and the patient's current condition, and the output is a recommended action. The generative AI model takes into account the current time of day and the patient's activity patterns to suggest appropriate actions.

[1231] Step 8:

[1232] The server notifies the smartphone of the recommended actions. The input is the recommended actions, and the output is a notification sent to the smartphone. The smartphone then suggests appropriate actions to the user through a push notification.

[1233] Step 9:

[1234] The terminal obtains real-time location information using a location information collection means that tracks the patient's location information. The input is GPS data, and the output is the patient's current location.

[1235] Step 10:

[1236] The device monitors the surrounding environment using a camera and microphone. The input is camera video and audio data, and the output is environmental information. The camera and microphone periodically collect data and provide it for analysis.

[1237] Step 11:

[1238] The server uses an emergency alert mechanism to issue a warning when a dangerous situation is detected. The input is location information and environmental information, and the output is a warning message. The generative AI model analyzes this data and generates a warning if a dangerous situation is detected.

[1239] Step 12:

[1240] The server sends real-time notifications to the caregiver: the input is the warning message and the output is the notification sent to the caregiver, allowing the caregiver to respond immediately.

[1241] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1242] This invention is a system for supporting the memory of dementia patients and recommending optimal actions. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide more personalized support. Specifically, this system has the function of analyzing the user's emotions and adjusting memory recall content and optimal action recommendations based on that.

[1243] System Configuration

[1244] The system consists of the following main components:

[1245] 1. Data Collection Methods

[1246] 2. Generative AI Models

[1247] 3. Memory Recreation Content Generation Method

[1248] 4. Smartphone display and notification methods

[1249] 5. Emotion Engine

[1250] Data collection methods

[1251] Users (patients or their family members) use a smartphone app to upload data such as past photos, videos, notes, and conversation history.

[1252] Generative AI Models

[1253] The server receives the data sent by the user and analyzes it using a generative AI model, which automatically generates photo captions and related stories based on the data.

[1254] Memory reproduction content generation means

[1255] The server creates memory-recreating content based on the analysis results, including adding captions related to photos and generating explanatory text about past episodes.

[1256] Smartphone display and notification methods

[1257] The device (smartphone) displays the memory-recreating content sent from the server to the user. The server then recommends optimal actions and sends a notification to the device. The device then prompts the user to take action via push notifications or alert messages.

[1258] Emotion Engine

[1259] The device has an emotion engine that recognizes the user's emotions. This emotion engine analyzes emotions based on the user's facial expressions, voice, and text input. The analyzed emotion information is sent to the server and reflected in memory reproduction content and behavioral recommendations.

[1260] Specific examples

[1261] A user opens a smartphone app and uploads photos from a previous trip, which the device then sends to the server.

[1262] The server analyzes the image file and its metadata and generates a caption, such as "This photo was taken in front of the Eiffel Tower during a trip to France in the summer of 2019." The server then sends the memory-recreating content, including the caption, to the device, which displays it to the user, allowing them to relive their memories by looking at the photo.

[1263] The device also recognizes emotions such as "happiness" and "surprise" through facial and voice analysis of the user. Based on this, the server generates a recommendation such as "It's time for a walk. Let's go to the nearest park" and sends it to the device. This recommendation is displayed at an appropriate time depending on the emotion.

[1264] For example, if the system recognizes that the user is feeling stressed, it will give a recommendation such as, "Why not take a break and read your favorite book?"

[1265] This system makes it easier for dementia patients to relive their past memories, providing a sense of security. It also recommends appropriate actions based on the user's emotions, further improving their quality of life. It also makes it easier for family members and caregivers to provide emotional support, enhancing overall care.

[1266] The processing flow will be explained below.

[1267] Step 1:

[1268] Users can launch a smartphone application and upload data such as past photos, videos, notes, conversation history, etc. Users can also select photos and enter captions and tags.

[1269] Step 2:

[1270] The device processes the uploaded data and prepares it for transmission to the server. Specifically, the device compresses the image files and adds appropriate metadata (e.g., date and time of the photo, location).

[1271] Step 3:

[1272] The device sends the prepared data to the server, which then sends a POST request to the server with the image and its metadata to the specified API endpoint.

[1273] Step 4:

[1274] The server stores the received data in storage. Specifically, the server stores the image file path and metadata in a database, and stores the image itself in cloud storage.

[1275] Step 5:

[1276] The server analyzes the stored data using a generative AI model. Specifically, the server inputs an image into the AI ​​model and generates an episode related to the image (e.g., the background of the photo, information about the people in the photo).

[1277] Step 6:

[1278] The server stores the analysis results in a database. Specifically, the server stores the analysis results (e.g., "This is a photo I took with my family at the beach last summer") in the database.

[1279] Step 7:

[1280] The server generates memory-recreating content based on the analysis results. Specifically, the server adds captions to images and generates related episodes as text.

[1281] Step 8:

[1282] The server sends the generated memory reproduction content to the device. Specifically, the server sends the generated captions along with the images in JSON format to the device.

[1283] Step 9:

[1284] The device displays the memory reproduction content received from the server to the user. Specifically, the device opens an image viewer and displays the image and its caption on the screen.

[1285] Step 10:

[1286] The device collects facial and voice data through a camera and microphone to recognize the user's emotions. Specifically, the device collects emotional data in real time while the user is browsing photos and stories.

[1287] Step 11:

[1288] The device analyzes the collected emotional data and identifies the user's current emotional state. Specifically, the emotion engine determines emotions such as "happiness," "surprise," and "sadness" from the user's facial expressions and voice.

[1289] Step 12:

[1290] The device sends the analyzed emotion data to the server by sending a POST request to the server's API endpoint in JSON format.

[1291] Step 13:

[1292] The server receives the emotion data and reflects it in memory-recreating content and optimal action recommendations. Specifically, it adjusts the content and recommendations to be displayed based on the emotion data.

[1293] Step 14:

[1294] The server uses a generative AI model to recommend optimal actions based on the patient's condition, past data analysis results, and emotional data. Specifically, it generates a recommendation such as "A walk is effective at the current time," but if the user is feeling stressed, it generates a different recommendation such as "Why don't you take a break and read your favorite book?"

[1295] Step 15:

[1296] The server sends the generated action recommendation to the terminal. Specifically, the server sends the action recommendation to the terminal as a notification message.

[1297] Step 16:

[1298] The device notifies the user of recommended actions. Specifically, it encourages the user to take action by means of push notifications or alert messages. For example, the device may notify the user, "It's time for a walk. Let's go to the nearest park."

[1299] Through these processing steps, this system supports the memory of dementia patients and improves their quality of life by recommending optimal actions based on their emotions. It also makes it easier for family members and caregivers to understand the patient's emotional state and provide more appropriate support.

[1300] Example 2

[1301] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1302] Dementia patients have difficulty retaining past memories and often experience stress in their daily lives. Furthermore, there are limited ways to encourage dementia patients to behave appropriately, placing a heavy burden on their families and caregivers. The problem this invention aims to solve is to improve the quality of life of dementia patients by supporting their memory recall and recommending optimal actions based on their emotions.

[1303] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user data, means for using a generative AI model to analyze the user data, means for generating memory reproduction content based on the analysis results, means for transmitting the memory reproduction content to a terminal, means for displaying the memory reproduction content on the terminal, means for analyzing the user's emotions, means for recommending optimal actions based on the user's emotional information, and means for notifying the terminal of the recommended actions. This allows dementia patients to easily reproduce past memories and receive appropriate recommended actions based on their emotions.

[1304] "User" refers to anyone who uses the system, including a dementia patient or their family or caregiver.

[1305] "Data collection means" refers to the ability of users to upload information such as photos, videos, notes, and conversation history using smartphone apps.

[1306] A "generative AI model" is an artificial intelligence technology that analyzes data collected from users and automatically generates photo captions and related stories.

[1307] "Memory reproduction content" is content that includes photo captions and text descriptions created based on the results of analysis by a generative AI model.

[1308] "Terminal" refers to a display device such as a smartphone or tablet used by a user.

[1309] "Emotion analysis means" is a technology that has the function of analyzing a user's facial expressions, voice, character input, etc., and recognizing the user's emotions.

[1310] "Behavioral recommendations" refer to specific actions suggested to users based on analyzed emotional information and past data.

[1311] The "notification means" is a function that sends the generated memory reproduction content and behavioral recommendations to the terminal as a push notification or an alert message and displays them to the user.

[1312] This invention is a system that supports memory for dementia patients and recommends optimal actions. By combining this system with an emotion engine that recognizes the user's emotions, it is possible to provide more personalized support.

[1313] System Configuration

[1314] The system consists of the following main components:

[1315] 1. Data Collection Methods

[1316] 2. Generative AI Models

[1317] 3. Memory Recreation Content Generation Method

[1318] 4. Terminal display and notification methods

[1319] 5. Emotion Engine

[1320] Data collection methods

[1321] Users use a smartphone app to upload data such as past photos, videos, memos, and conversation histories. For example, they can upload photos of a family trip or diary notes to the app. The device then sends this data to a server.

[1322] Generative AI Models

[1323] The server uses a generative AI model to analyze the data received from the device. This generative AI model is an advanced natural language processing model such as GPT-4. The server analyzes the metadata of the photo or video (e.g., date and time of the photo, location) and generates a caption based on that. For example, it generates a caption such as, "This photo was taken in front of the Eiffel Tower in Paris, France, in the summer of 2019."

[1324] Memory reproduction content generation means

[1325] The server generates memory-recreating content based on the analysis results. This content includes captions related to the photos and explanatory text about past episodes. For example, a photo of a trip to France might include a description such as, "On a beautiful summer day, my family and I visited the Eiffel Tower and then took a walk along the Seine."

[1326] Terminal display and notification means

[1327] The device displays the memory reproduction content sent from the server to the user. For example, the app's notification function can be used to notify the user, "A new memory has been created!" When the user clicks on the notification, the generated memory reproduction content is displayed within the app.

[1328] Emotion Engine

[1329] The device has an emotion engine that analyzes the user's facial expressions, voice, and text input to recognize emotions. For example, if the user smiles at a displayed photo, the device recognizes "joy." If the user writes "I had a lot of fun" when entering a comment, the device analyzes it as a "positive emotion."

[1330] Action recommendation generation and notification

[1331] The server generates optimal action recommendations based on the emotional information sent from the emotion engine. For example, if the user expresses the emotion "joy," the server generates a recommendation such as, "You seem to be in a good mood right now, so why not enjoy some tea at a nearby cafe?" The server sends this recommendation to the device. The device notifies the user of this recommendation, displaying, for example, "It's time for a walk now. Let's go to a nearby park."

[1332] Specific examples

[1333] Prompt Sentence Examples

[1334] Upload past travel photos and the system will analyze your emotions and recommend the best course of action.

[1335] Example

[1336] A user opens the smartphone app, selects a photo from their trip to France in 2019, and presses the upload button. The device sends this photo to the server. The server analyzes the received image and generates a caption such as, "This photo was taken in front of the Eiffel Tower during my trip to France in the summer of 2019." The server creates memory-recreating content including this caption and sends it to the device. The device notifies the user via push notification, "A new memory has been created!" When the user clicks the notification, the generated content is displayed within the app. The device analyzes the user's facial expression with an emotion engine and recognizes "joy." Based on this emotional information, the server generates an action recommendation such as, "It's time for a walk now. Let's go to the nearest park." The device notifies the user of this recommendation.

[1337] This system allows dementia patients to easily recall past memories and receive appropriate behavioral recommendations based on their emotions, enabling them to enrich their daily lives.

[1338] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1339] Step 1:

[1340] The user opens the smartphone app, selects past photos, videos, notes, and conversation history (input data), and presses the upload button. The device then sends this data to the server (specific operation of data transmission).

[1341] Step 2:

[1342] The server uses a generative AI model to analyze the data received from the device (input data). It uses a generative AI model (e.g., GPT-4) to check the metadata of the photo or video (e.g., date and time of the photo, location) and generates a caption (specific operation of data analysis and caption generation). For example, it generates a caption (output data) such as, "This photo was taken in front of the Eiffel Tower in Paris, France, in the summer of 2019."

[1343] Step 3:

[1344] The server creates memory-recreating content based on the generated captions (input data). This content includes captions related to the photos and explanatory text about past episodes. For example, to a photo of a trip to France, it adds the description (output data) "On a beautiful summer day, after visiting the Eiffel Tower with my family, we took a walk along the Seine River" (specific operation of generating memory-recreating content).

[1345] Step 4:

[1346] The server sends the generated memory reproduction content (input data) to the device. The device receives it and displays it to the user (specific actions for sending and displaying content). For example, the app's notification function can be used to notify the user, "A new memory has been created!" When the user clicks on the notification, the generated memory reproduction content (output data) is displayed within the app.

[1347] Step 5:

[1348] The device uses an emotion engine that analyzes the user's facial expressions, voice, and text input to recognize the user's emotions (input data). For example, if the user smiles at a displayed photo, the device recognizes "joy." If the user writes "I had a lot of fun" when entering a comment, the device analyzes it as a "positive emotion" (specific operation of emotion analysis).

[1349] Step 6:

[1350] The server generates optimal behavioral recommendations based on the emotional information (input data) sent from the emotion engine. For example, if the user expresses the emotion "joy," the server generates a behavioral recommendation (output data) such as "You seem to be in a good mood right now, so why don't you enjoy tea at a nearby cafe?" (Specific operation of behavioral recommendation generation).

[1351] Step 7:

[1352] The server sends the generated action recommendation (input data) to the device. The device displays this recommendation to the user as a push notification (specific operation of sending a recommendation and notification). For example, a message (output data) saying "It's time for a walk now. Let's go to a nearby park" is displayed.

[1353] In this way, each step of the system sequentially processes and analyzes data, and by providing the results to the user, it helps dementia patients recover their memories and improve their lives.

[1354] (Application example 2)

[1355] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1356] Current dementia patient support systems are inadequate in terms of memory recall and emotional recognition. Furthermore, in support of factory workers, they do not provide efficient work procedures or recommend breaks based on the worker's emotions, which prevents improvements in work efficiency and reduces the burden on workers. A comprehensive solution to these issues is needed.

[1357] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data on dementia patients, means for using a generative AI model to analyze the data, means for generating memory reproduction content based on the analysis results, means for collecting task data, means for using a work process analysis AI model to analyze the task data, and means for using an emotion recognition engine to recognize the user's emotions. This makes it possible to provide memory reproduction support to dementia patients and recommend efficient work procedures and emotion-based breaks to factory workers.

[1358] "Dementia" is a disease characterized by a decline in cognitive function that affects memory, judgment, communication, and daily activities.

[1359] "Data collection means" is a general term for any device or method for acquiring information from a user or object.

[1360] A "generative AI model" is an artificial intelligence algorithm that predicts, analyzes, and generates based on collected data.

[1361] "Memory reproduction content" refers to information and media materials that are used to reproduce a user's past memories based on analyzed data.

[1362] A "display terminal" is an electronic device such as a smartphone, tablet, or smart glasses that allows users to view and operate information.

[1363] "Behavioral recommendations" refer to notifications and advice that suggest optimal actions and options to users.

[1364] "Work data" is a set of information related to the progress, method, results, etc. of a work.

[1365] The "work process analysis AI model" is an artificial intelligence algorithm that analyzes work data and derives work efficiency and areas for improvement.

[1366] An "emotion recognition engine" is a technology for analyzing emotions based on a user's facial expressions, voice, text input, etc.

[1367] "Break recommendation" refers to notifications or advice that suggest taking a break at the optimal time based on the user's emotional state.

[1368] The system for implementing this invention provides integrated support for dementia patients and factory workers.

[1369] System Overview

[1370] This system includes a server, display terminals (smartphones, smart glasses, etc.), and a mobile robot within the factory. The server plays a central role in performing key processes such as data collection, analysis, generation, and display.

[1371] Key Components

[1372] 1. Data Collection Methods

[1373] 2. Generative AI Models

[1374] 3. Memory Recreation Content Generation Method

[1375] 4. Display and notification methods on display devices (smartphones, smart glasses, etc.)

[1376] 5. Emotion Recognition Engine

[1377] 6. Work Process Analysis AI Model

[1378] 1. Data Collection Methods

[1379] The dementia patient or worker uploads past photos, videos, notes, conversation history, and work data to a server through a data collection method, such as using a camera or microphone on a smartphone or smart glasses.

[1380] 2. Generative AI Models

[1381] The server receives the collected data and analyzes it using a generative AI model (e.g., TensorFlow, PyTorch), which automatically generates captions, anecdotes, and efficient work procedures from the user's past photos and work data.

[1382] 3. Memory Recreation Content Generation Method

[1383] Based on the analysis results, memory recall content is generated. For example, this includes adding captions to past photos and generating text about related episodes. In the case of work data, efficient work procedures are generated.

[1384] 4. Display terminal display and notification means

[1385] The memory reproduction content and work procedures sent from the server are displayed on a display device, and notifications are sent to the user via a smartphone or smart glasses, for example.

[1386] 5. Emotion Recognition Engine

[1387] The system analyzes the user's emotions in real time using a camera and microphone installed on the display device (smartphone, smart glasses). The analyzed emotional information is sent to a server and reflected in memory reproduction content and work procedure recommendations.

[1388] 6. Work Process Analysis AI Model

[1389] The server analyzes the work data using a work process analysis AI model (e.g., Keras, Scikit-learn) to extract efficient work procedures and areas for improvement. This information is then appropriately notified to the workers.

[1390] Specific examples

[1391] Let's say a user wears smart glasses and performs work in a factory. The work progress and sensor data are collected and sent to a server. The server analyzes this data using an AI model for work process analysis and generates work instructions such as, "This work usually takes 15 minutes to complete. Please try doing it this way."

[1392] The smart glasses also use a camera and microphone to analyze the worker's facial expressions and voice, which are then analyzed by an emotion recognition engine. If stress is detected, the system will recommend a break, such as "Take a 5-minute break now."

[1393] Prompt Sentence Examples

[1394] Build a system that recognizes workers' emotions in real time and sends them notifications to encourage them to take a break if they feel stressed or impatient, by using smart glasses on a parts assembly line and analyzing past work data to generate specific recommendations for optimal procedures.

[1395] As described above, this invention realizes a system that can restore memories, provide efficient work procedures, and recommend optimal actions based on emotions for both dementia patients and factory workers.

[1396] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1397] Step 1: Data collection

[1398] A user uses smart glasses or a smartphone to collect past data (e.g., photos, videos, notes, conversation history, and work data). This data is sent to a server by a data collection means. The input data is photos and work progress, and the output data is information stored in a database on the server.

[1399] Step 2: Data analysis

[1400] The server analyzes the received data using a generative AI model (e.g., TensorFlow, PyTorch). The input data is the user's data stored on the server, and the output data is the analysis results: captions, episodes, and work procedures. Specifically, it generates captions for photos and extracts efficient procedures from work data.

[1401] Step 3: Generate memory recall content

[1402] The server generates memory-recreating content based on the analysis results. The input data is the analysis results from the generative AI model, and the output data is the memory-recreating content (for example, photo captions or episode text). Specifically, it adds captions to photos and generates work procedures as text.

[1403] Step 4: Send to display device

[1404] The server sends the generated memory reproduction content and work procedures to a display device (smartphone or smart glasses). The input data is the memory reproduction content and work procedures, and the output data is the content displayed on the device.

[1405] Step 5: Emotion Recognition

[1406] To recognize the user's emotions, the device (smart glasses or smartphone) uses a camera and microphone to collect the user's facial expression and voice data. This data is analyzed by an emotion recognition engine. The input data is the user's facial expression and voice, and the output data is the emotion analysis results. Specifically, the device analyzes the user's emotional state, such as stress or joy.

[1407] Step 6: Generate action recommendations

[1408] The server recommends optimal actions to the user based on the results of emotion analysis and past data. The input data are the results of emotion analysis and past data, and the output data is the recommended action. Specifically, notifications such as "Take a 5-minute break now" or "Try this work procedure" are generated.

[1409] Step 7: Notify the display device

[1410] The server notifies the generated action recommendation to the display device. The input data is the action recommendation, and the output data is the notification displayed on the device. Specifically, it is displayed as a push notification on smart glasses or a smartphone.

[1411] In this way, it is possible to support memory recall for dementia patients, provide efficient work procedures for factory workers, and even recommend optimal actions based on emotions.

[1412] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1413] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1414] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1415] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1416] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1417] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1418] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1419] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1420] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1421] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1422] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1423] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1424] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1425] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1426] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1427] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1428] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1429] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1430] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1431] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1432] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1433] The following is further disclosed regarding the above embodiment.

[1434] (Claim 1)

[1435] a means of collecting data on dementia patients;

[1436] means for utilizing a generative AI model to analyze said data;

[1437] a means for generating memory reproduction content based on the analysis results;

[1438] means for transmitting the memory reproduction content to a smartphone;

[1439] a means for displaying the memory reproduction content on the smartphone;

[1440] A means of recommending optimal actions;

[1441] means for notifying a smartphone of the behavioral recommendation;

[1442] A system including:

[1443] (Claim 2)

[1444] The system of claim 1 , wherein the memory reconstruction content is analyzed photograph captions and associated episode text.

[1445] (Claim 3)

[1446] The system of claim 1 , wherein the optimal action recommendation is generated based on past data analysis results and a current patient condition.

[1447] "Example 1"

[1448] (Claim 1)

[1449] A means for users to upload information such as past photos, videos, notes, and conversation history using a smartphone app;

[1450] means for receiving and storing information transmitted from the smartphone app;

[1451] A means of analyzing the stored information and using generative AI models to generate photo captions and related anecdotes;

[1452] a means for generating memory reproduction content based on the analysis results;

[1453] means for transmitting the memory reproduction content to a smartphone;

[1454] a means for displaying the memory reproduction content on the smartphone;

[1455] a means for recommending optimal actions based on the patient's current condition;

[1456] means for notifying a smartphone of the behavioral recommendation;

[1457] A system including:

[1458] (Claim 2)

[1459] 10. The system of claim 1, wherein the memory reconstruction content is analyzed photograph captions and associated episode text.

[1460] (Claim 3)

[1461] 10. The system of claim 1, wherein the recommendation of the optimal action is generated based on past information analysis results and the patient's current condition.

[1462] "Application Example 1"

[1463] (Claim 1)

[1464] a means of collecting data on dementia patients;

[1465] means for utilizing a generative AI model to analyze said data;

[1466] a means for generating memory reproduction content based on the analysis results;

[1467] means for transmitting the memory reproduction content to a smartphone;

[1468] a means for displaying the memory reproduction content on the smartphone;

[1469] A means of recommending optimal actions;

[1470] means for notifying a smartphone of the behavioral recommendation;

[1471] a location information collecting means for tracking location information of a patient;

[1472] an environment monitoring means for monitoring the surrounding environment using a camera and a microphone;

[1473] an emergency alert means for issuing a warning when a dangerous situation is detected;

[1474] a means for sending real-time notifications to caregivers;

[1475] A system including:

[1476] (Claim 2)

[1477] The system of claim 1, wherein the memory reproduction content is a caption of an analyzed photograph and text of an associated episode, and further includes memory reproduction content based on a patient's location information and surrounding environmental information.

[1478] (Claim 3)

[1479] 10. The system of claim 1, wherein the optimal action recommendation is generated based on past data analysis results and the patient's current condition, and further based on the patient's current location information and environmental information.

[1480] "Example 2: Combining Emotion Engines"

[1481] (Claim 1)

[1482] a means for collecting user data;

[1483] a means for utilizing a generative AI model to analyze said user's data;

[1484] a means for generating memory reproduction content based on the analysis results;

[1485] means for transmitting the stored reproduction content to a terminal;

[1486] means for displaying the memory reproduction content on the terminal;

[1487] means for analyzing user emotions;

[1488] means for recommending an optimal action based on the emotional information of the user;

[1489] means for notifying a terminal of the behavioral recommendation;

[1490] A system including:

[1491] (Claim 2)

[1492] The system of claim 1 , wherein the memory reconstruction content is analyzed photograph captions and associated episode text.

[1493] (Claim 3)

[1494] The system of claim 1 , wherein the optimal behavior recommendation is generated based on past data analysis results and a current state of the user.

[1495] "Application example 2 when combining emotion engines"

[1496] (Claim 1)

[1497] a means of collecting data on dementia patients;

[1498] means for utilizing a generative AI model to analyze said data;

[1499] a means for generating memory reproduction content based on the analysis results;

[1500] means for transmitting the stored reproduction content to a display terminal;

[1501] means for displaying the stored reproduction content on the display terminal;

[1502] A means of recommending optimal actions;

[1503] means for notifying a display terminal of the behavioral recommendation;

[1504] data collection means for collecting work data;

[1505] A means for utilizing a work process analysis AI model for analyzing the work data;

[1506] means for generating an optimal work procedure based on the results of the work process analysis;

[1507] a means for utilizing an emotion recognition engine for recognizing an emotion of a user;

[1508] a means for recommending a break between tasks based on the emotion recognition result;

[1509] A system including:

[1510] (Claim 2)

[1511] The system of claim 1 , wherein the memory reconstruction content is analyzed photograph captions and associated episode text.

[1512] (Claim 3)

[1513] The system of claim 1 , wherein the optimal action recommendation is generated based on past data analysis results and a current patient condition. [Explanation of symbols]

[1514] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means of collecting data on dementia patients; means for utilizing a generative AI model to analyze said data; a means for generating memory reproduction content based on the analysis results; means for transmitting the memory reproduction content to a smartphone; a means for displaying the memory reproduction content on the smartphone; A means of recommending optimal actions; means for notifying a smartphone of the behavioral recommendation; A system including:

2. The system of claim 1 , wherein the memory reconstruction content is analyzed photograph captions and associated episode text.

3. The system of claim 1 , wherein the optimal action recommendation is generated based on past data analysis results and a current patient condition.

Images