System

Abstract

A system is provided.SOLUTION: The system includes a means for receiving photograph data from a terminal of a user, a means for analyzing the received photograph data by an image analysis engine to specify a person or an object in the photograph, a means for collecting metadata of the photograph on the basis of an analysis result to generate past reminiscence information, a means for creating an interaction scenario on the basis of the generated reminiscence information, and a means for performing interaction with the user on the basis of the interaction scenario.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the number of patients with early-stage dementia and elderly people suspected of having memory disorders has been increasing, creating a need for methods to stimulate and improve the memory of these individuals. Conventional treatments and rehabilitation methods require the intervention of medical institutions and specialists, and are problematic in that they are time-consuming and costly. Furthermore, there has been a lack of easy-to-use tools that allow patients to enjoyably reminisce about their past memories. Therefore, the present invention aims to provide a system that allows users to have fun conversations using past photographs, thereby providing a means to stimulate memory and alleviate the symptoms of early-stage dementia. [Means for solving the problem]

[0005] The present invention provides a system that solves the above problems by the following means. First, a means for receiving photo data from a user's terminal is provided. Next, a means for analyzing the received photo data using an image analysis engine and identifying people and objects in the photos is provided. Furthermore, a means for collecting photo metadata and generating past memory information based on the analysis results is provided. A means for creating a dialogue scenario based on the generated memory information is provided, and a means for engaging in a dialogue with the user based on the dialogue scenario is provided. Furthermore, by including a means for monitoring the state of the user's memory impairment and providing feedback on the dialogue, continuous memory improvement is supported. A means for identifying people in photos using facial recognition technology is also provided. This realizes a system that allows users to have fun dialogue through past photos and stimulate their memories.

[0006] "User Device" means the electronic device used by the User to manipulate and upload Photo Data.

[0007] "Photo data" refers to a digital file of a captured still image.

[0008] An "image analysis engine" is a collection of software and algorithms that analyzes incoming photo data and identifies people and objects within the photo.

[0009] "Metadata" is information related to a photograph, including, for example, the date and time the photograph was taken, location information, and information about identified people.

[0010] "Memories information" is information about past events and episodes that is generated based on photos and analyzed metadata.

[0011] A "dialogue scenario" is a set of questions and answers that are generated based on memory information and guide the dialogue between the user and the system.

[0012] "Monitoring" is the process of observing and evaluating a user's memory state and responses in interaction.

[0013] "Feedback" refers to information such as improvement suggestions and additional questions provided to users based on the monitoring results.

[0014] "Facial recognition technology" is a technology for identifying and identifying the faces of people in photographs. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention provides a system for stimulating a user's memory by using past photographs and alleviating early symptoms of dementia. Specific embodiments of the system will be described below.

[0037] System Overview

[0038] This system works by interacting with the server, the device, and the user. The user uploads past photos from their device to the server, and the server analyzes the photos to generate a dialogue scenario, which is then presented to the user via their device. The user can then have a conversation based on the scenario, allowing them to enjoyably reminisce about their memories.

[0039] System processing flow

[0040] 1. User selects and uploads a photo:

[0041] User: First, open the application on the device and select some old photos, such as photos from a family trip or a special event.

[0042] Terminal: It then displays a preview of the selected photo to the user and provides an "Upload" button.

[0043] User: Check the preview and click the "Upload" button.

[0044] 2. The device sends the photo data to the server:

[0045] Terminal: Uses a business API to send the selected photo data to the server. At this time, the network connection status and progress are displayed to the user.

[0046] 3. The server receives the photo and begins analyzing it:

[0047] Server: Temporarily stores the received photo data in storage and launches the image analysis engine, using technologies such as OpenCV and TENSORFLOW (registered trademark).

[0048] Server: Identifies people and objects in the photo and uses facial recognition technology to identify people in the shot, leveraging libraries like Dlib and FaceNet.

[0049] 4. The server collects the metadata:

[0050] Server: Extracts photo metadata (such as the date and time of the photo, location information, etc.) from the EXIF ​​data and obtains information about identified people from an internal database. For example, it may collect information such as that the photo was taken at Shonan Beach or that a specific person is a family member.

[0051] 5. The server generates the relevant memories:

[0052] Server: Based on the collected metadata, the server creates text about memorable episodes related to the photo. For example, it generates information such as "Memories of a family trip to Shonan Beach in the summer of 2005."

[0053] Server: Based on this generated information, a dialogue scenario is formulated and a list of questions is created to guide the conversation with the user.

[0054] 6. The server sends the dialogue scenario to the device:

[0055] Server: Sends the dialogue scenario and the generated question list to the terminal.

[0056] Terminal: Prepares to display the dialogue scenario on the user's screen based on the information received from the server.

[0057] 7. The device presents the dialogue scenario to the user:

[0058] Terminal: Displays the initial dialogue questions to the user and notifies them that they are ready to begin the dialogue.

[0059] User: Responds to questions displayed on the device by text input or voice input.

[0060] 8. User and system begin interaction:

[0061] Terminal: Records the user's responses and continuously sends them to the server.

[0062] Server: Analyzes the user's response, generates and selects the next question or feedback, and sends it to the device. Through this cycle, the user can continue the dialogue.

[0063] 9. The server analyzes the dialogue and provides feedback:

[0064] Server: Collects dialogue logs and analyzes user responses using natural language processing technology. Evaluates the user's memory status and emotional changes, and uses this information for the next dialogue.

[0065] Server: Creates individual feedback and follow-up questions based on the user's memory impairment status and provides them to the user via the device.

[0066] Specific examples

[0067] For example, consider a scenario in which a user selects a photo from a family trip from their childhood and starts a conversation based on the topic of "Memories of having fun with my family at Shonan Beach in the summer of 2005." The user uploads the photo, and the server analyzes it to generate anecdotes and questions like the following:

[0068] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[0069] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[0070] As the user answers these questions, the system elicits more detailed information and stimulates the user's memory.

[0071] This system allows users to enjoy memories of the past while stimulating and improving their memory.

[0072] The processing flow will be explained below.

[0073] Step 1:

[0074] The user launches the application on the device, opens an interface within the application for selecting past photos, and selects a specific photo from the saved photos.

[0075] Step 2:

[0076] The device displays a preview of the selected photo to the user and provides an "Upload" button to confirm that the photo is correct.

[0077] Step 3:

[0078] The user clicks the "Upload" button, which completes preparations for uploading the photo data.

[0079] Step 4:

[0080] The device sends the selected photo data to the server using a business API, and the progress of the network connection is displayed to the user.

[0081] Step 5:

[0082] The server stores the received photo data in temporary storage, then launches the image analysis engine and begins analyzing the photo.

[0083] Step 6:

[0084] The server uses an image analysis engine to identify people and objects in the photo, for example, by using facial recognition technology to identify people, using libraries such as Dlib and FaceNet.

[0085] Step 7:

[0086] The server extracts the photo's metadata (e.g., EXIF ​​data) and obtains the date and time the photo was taken and its location, which identifies when and where the photo was taken.

[0087] Step 8:

[0088] The server references an internal database and obtains detailed information about the people in the photo (e.g., names, relationships, etc.) from the facial recognition results.

[0089] Step 9:

[0090] The server generates information about past memories based on the analysis results and metadata. For example, it creates a story such as "I went on a family trip to Shonan Beach in the summer of 2005."

[0091] Step 10:

[0092] The server creates a dialogue scenario based on the generated memory information, which includes a list of questions to prompt dialogue with the user.

[0093] Step 11:

[0094] The server sends the created dialogue scenario to the terminal. The scenario includes episodes and related questions.

[0095] Step 12:

[0096] The terminal prepares to display the dialogue scenario received from the server on the user screen, presents the first question, and notifies the user that the dialogue is ready to begin.

[0097] Step 13:

[0098] The user responds to questions displayed on the screen by text input or voice input, and the device records the user's responses.

[0099] Step 14:

[0100] The device sends the user's responses in real time to the server, which then generates the next question or feedback based on the received data.

[0101] Step 15:

[0102] The server analyzes the user's responses and uses natural language processing techniques (e.g., sentiment analysis) to assess the user's memory state and emotional changes.

[0103] Step 16:

[0104] The server generates the next question and feedback and sends it to the device, which may include personalized questions and suggestions to help the user improve their memory.

[0105] Step 17:

[0106] The device receives new questions and feedback from the server and presents them to the user, allowing the user to continue interacting and stimulating their memory.

[0107] Step 18:

[0108] When the user indicates an intention to end the interactive session (for example, by clicking an "end" button), the terminal notifies the server of this information.

[0109] Step 19:

[0110] The server acknowledges the end of the interaction session and generates a summary and feedback of the interaction, including an assessment of the user's memory state and the progress of the interaction.

[0111] Step 20:

[0112] The device displays feedback from the server to the user, suggests next sessions, saves records, and allows the user to see the results of the interaction.

[0113] Example 1

[0114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] Users in the early stages of dementia have difficulty maintaining and recalling memories in their daily lives, which increases their mental burden. Current systems have limited methods for effectively utilizing users' past photos to stimulate memories, and the lack of an interactive approach reduces opportunities for users to actively reflect on the past. Furthermore, the system lacks mechanisms for monitoring the progress of memory impairment and providing appropriate feedback.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0117] In this invention, the server includes a means for receiving image data from the user's information processing device, a means for analyzing the received image data using an image analysis device and identifying objects in the image, a means for collecting additional information about the image based on the analysis results and generating past memory information, a means for creating a dialogue plan based on the generated memory information, a means for engaging in a dialogue with the user based on the dialogue plan, a means for recording the user's responses during the dialogue and transmitting the data, a means for generating dialogue questions and feedback using a generative AI model, and a means for collecting dialogue logs and analyzing them using natural language processing technology. This allows the user to use past photos to stimulate their memories and efficiently recall them. Furthermore, by monitoring the user's memory status through dialogue and providing appropriate feedback, it is possible to maintain and improve cognitive function.

[0118] "User" refers to the person who uses this system, and is particularly targeted at people in the early stages of dementia.

[0119] An "information processing device" is a terminal or device operated by a user, and allows operations such as selecting and uploading photos.

[0120] "Image data" refers to digital images such as photographs and drawings uploaded by users.

[0121] An "image analysis device" is a device that analyzes image data and identifies subjects (people or objects) within the image, and includes an image analysis engine and library.

[0122] "Additional information" refers to metadata such as the image's EXIF ​​data, as well as information related to the subject matter within the image.

[0123] "Memories" refers to past events and episodes that are generated based on the results of image analysis and additional information.

[0124] "Dialogue planning" refers to a scenario or question list that guides a dialogue with the user based on memory information.

[0125] "Generative AI model" refers to an artificial intelligence model used to automatically generate dialogue questions and feedback.

[0126] "Natural language processing technology" refers to technology for analyzing dialogue logs and understanding user responses, and includes technology used for sentence generation and sentiment analysis.

[0127] The present invention provides a system for stimulating past memories in a user and alleviating early symptoms of dementia. Specific processing and embodiments of the program of this system will be described in detail below.

[0128] This system functions through the interaction of three parties: a server, a terminal, and a user. The server is responsible for a wide range of processes, including receiving and analyzing image data, extracting metadata, and generating a dialogue plan. The terminal provides an interface for users to upload photos and initiate dialogue. Users use this interface to select and upload photos and engage in dialogue according to the dialogue scenario provided by the server.

[0129] Server processing

[0130] The server first receives image data from the user. To do this, the server receives image data sent from the device using a RESTful API. The received image data is then temporarily stored in cloud storage. The main technology used is likely to be a cloud service such as Amazon S3.

[0131] The stored image data is analyzed using an image analyzer (e.g., OpenCV or TensorFlow) to identify people and objects in the image and use facial recognition technology (e.g., Dlib or FaceNet). Based on the analysis results, the EXIF ​​data of the photo is extracted and metadata such as the date and time of the photo's capture and location are collected using a library such as ExifTool.

[0132] The collected metadata and analysis results are integrated to generate memory information. Natural language generation technology (generative AI model) is used to create sentences about specific episodes, such as "Memories of a family trip to Shonan Beach in the summer of 2005." Based on the generated memory information, a dialogue plan is also created, and a list of questions is created to guide the conversation with the user.

[0133] The generated dialogue plan and question list are sent to the terminal via encrypted communication.

[0134] About terminal processing

[0135] The device accepts operations from the user and sends them to the server. The user opens the dedicated application, selects and uploads photos. During uploading, a progress bar is displayed to notify the user of the progress. The device displays the dialogue scenario received from the server, allowing the user to start a dialogue. During the dialogue, the device records the user's responses and continuously sends them to the server.

[0136] About User Roles

[0137] The user selects and uploads past photos using the device. Based on the dialogue scenario provided by the server, the user responds by text input or voice input. Through this process, the user's past memories are stimulated, and it is expected that their memory will be improved.

[0138] Specific examples

[0139] For example, if a user uploads a photo of a family trip to Shonan Beach in the summer of 2005, the server processes it as follows: First, it extracts "Date and time of photo: Summer 2005, Location: Shonan Beach" from the photo's EXIF ​​data. Next, it uses facial recognition technology to identify family members and generates anecdotes and questions like the following:

[0140] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[0141] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[0142] As the user answers these questions, the system elicits more detailed information and stimulates the user's memory.

[0143] Prompt Sentence Examples

[0144] "Generate a conversation scenario based on memories from photos of a family trip to Shonan Beach."

[0145] "Create a story related to the person in this photo."

[0146] As described above, this system aims to effectively stimulate the user's memory by linking various hardware and software.

[0147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0148] Step 1: Select and upload a photo

[0149] Subject: User

[0150] Input: Photo data selected on your device

[0151] Output: Uploaded photo data

[0152] The user opens a dedicated application on their device and selects a past photo. The selected photo is previewed, and when the user clicks the "Upload" button, the photo data is sent to the server.

[0153] Step 2: Receiving and saving photo data

[0154] Subject: Server

[0155] Input: Photo data uploaded by the user

[0156] Output: Saved photo data (cloud storage)

[0157] The server receives the photo data sent from the device and temporarily stores it in cloud storage, using a cloud service such as Amazon S3.

[0158] Step 3: Analyze the photo data

[0159] Subject: Server

[0160] Input: Saved photo data

[0161] Output: Analysis results (specific information about people and objects)

[0162] The server analyzes the stored photo data using image analysis equipment (e.g., OpenCV or TensorFlow). The analysis identifies people and objects in the photos. The analysis results also include information on people identified using facial recognition technology (Dlib or FaceNet).

[0163] Step 4: Collecting Metadata

[0164] Subject: Server

[0165] Input: Analysis results and photo EXIF ​​data

[0166] Output: Collected metadata (date and time of photo, location, etc.)

[0167] The server extracts the EXIF ​​data from the photos using tools such as ExifTool, and collects metadata such as the date and time the photo was taken and the location information. This is then combined with the analysis results to collect information on identified individuals.

[0168] Step 5: Generate memories

[0169] Subject: Server

[0170] Input: Collected metadata and analysis results

[0171] Output: Generated memories

[0172] The server uses natural language generation technology (generative AI model) to generate memory information based on the collected metadata and analysis results. For example, a detailed episode such as "Memories of a family trip to Shonan Beach in the summer of 2005" can be written.

[0173] Step 6: Developing dialogue scenarios

[0174] Subject: Server

[0175] Input: Generated memory information

[0176] Output: Dialogue scenario and questionnaire

[0177] The server formulates a dialogue plan based on the generated memory information and creates a specific list of questions for the user, such as "What is your first memory of seeing this photo?"

[0178] Step 7: Sending the dialogue scenario to the device

[0179] Subject: Server

[0180] Input: Dialogue scenario and questionnaire

[0181] Output: Dialogue scenario and question list sent to the terminal

[0182] The server sends the dialogue scenario and question list to the terminal through encrypted communication.

[0183] Step 8: Present the dialogue scenario

[0184] Subject: Device

[0185] Input: Dialogue scenario and question list received from the server

[0186] Output: The dialogue scenario and question list presented to the user

[0187] The terminal presents the dialogue scenario and question list received from the server to the user, and displays the initial dialogue questions in a pop-up format on the screen to prompt the user to start the dialogue.

[0188] Step 9: User responses and recording

[0189] Subject: User and Device

[0190] Input: User input (text or voice)

[0191] Output: Recorded user responses

[0192] The user responds to questions displayed on the device by text input or voice input, and the device records the user's responses and continuously transmits them to the server.

[0193] Step 10: Continued dialogue and feedback

[0194] Subject: Server

[0195] Input: User response data

[0196] Output: Next dialogue question or feedback

[0197] The server analyzes the user's response data and uses a generative AI model to generate the next question or feedback, allowing the user to continue the dialogue.

[0198] These are the specific processing steps of the system, which aims to effectively stimulate the user's memory and alleviate the early symptoms of dementia.

[0199] (Application example 1)

[0200] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0201] Today, there is a need for effective methods to stimulate memories and alleviate the condition of people in the early stages of dementia. However, existing systems are limited to reviewing old photos and lack the ability to delve deeper into memories or re-experience memories in a more immersive way. Furthermore, they lack the precision to elicit memories through dialogue and the ability to personalize the user experience. Therefore, new technologies are needed to enable users to re-experience past memories in a more realistic way and stimulate and improve memory.

[0202] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0203] In this invention, the server includes means for receiving photo data from a user's device, means for analyzing the received photo data using an image analysis engine and identifying people and objects in the photo, means for collecting photo metadata and generating past memory information based on the analysis results, means for creating a dialogue scenario based on the generated memory information, means for presenting the dialogue scenario to the user and displaying it on a head-mounted display so that the user can re-experience the experience in a virtual reality space, and means for analyzing the user's response using a natural language processing engine and generating the next question and feedback. This allows the user to re-experience memories in a realistic virtual reality space, making it possible to effectively stimulate and improve memory in the early stages of dementia.

[0204] A "user" is an individual who uses the system to re-experience past memories and stimulate their memories.

[0205] "Device" means an electronic device used by a user to upload photo data and receive and display dialogue scenarios.

[0206] "Photo data" refers to image files that have been taken in the past and are uploaded by users.

[0207] The "image analysis engine" is a software component that analyzes received photo data and identifies people and objects in the photo.

[0208] "Metadata" refers to accompanying information included in photo data, such as the date and time of shooting, location information, and information about the shooting device.

[0209] "Memories information" is information about past events generated based on the results of image analysis and photo metadata.

[0210] A "dialogue scenario" is a scenario that includes questions and topics to promote dialogue with the user based on the generated memory information.

[0211] "Virtual reality" is a virtual space generated using computer graphics, in which users can experience a realistic virtual environment.

[0212] A "head-mounted display" is a display device worn by the user on the head to allow the user to visually experience a virtual reality space.

[0213] A "natural language processing engine" is a software component that analyzes a user's dialogue responses and generates the next question or feedback.

[0214] System Overview

[0215] This invention is a system that allows users to use past photographs to stimulate their memories and alleviate the early symptoms of dementia. This system functions through the interaction of a server, a terminal, and a user.

[0216] How to run the program

[0217] 1. User Conduct

[0218] Users use their devices to select past photos, such as photos from a family trip or special event, and upload them to the server.

[0219] 2. Device Operation

[0220] The terminal displays a preview of the selected photo to the user and provides an "Upload" button, which causes the terminal to send the photo data to the server.

[0221] 3. Server Processing

[0222] The server analyzes the received photo data using an image analysis engine (e.g., OpenCV or TensorFlow). It performs facial and object recognition to identify people and objects in the photo. It also recognizes identified people using libraries such as Dlib and FaceNet. It also extracts photo metadata (such as the date and time of the photo shoot and location information) from the EXIF ​​data and obtains information about the identified people from an internal database.

[0223] Next, the server generates a memorable episode related to the photo based on the collected metadata. For example, it writes down information such as "Memories of a family trip to Shonan Beach in the summer of 2005." It then formulates a dialogue scenario based on the generated information and creates a list of questions to encourage conversation with the user.

[0224] 4. Use of head-mounted displays

[0225] The dialogue scenario and question list are sent from the server to the device and head-mounted display, which generates a virtual reality space for the user to re-experience, and presents the dialogue scenario within it.

[0226] 5. Dialogue with users

[0227] The user wears a head-mounted display and responds according to the displayed dialogue scenario. The user's responses are analyzed by a natural language processing engine (e.g., spaCy), which generates the next question or feedback. This allows the dialogue with the user to continue and elicit more memories.

[0228] Specific examples

[0229] For example, the following prompts can be input to a generative AI model:

[0230] "Generate a dialogue scenario and a list of questions for memory stimulation based on this photo taken at Shonan Beach in the summer of 2005."

[0231] The generated dialogue scenario example is as follows:

[0232] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[0233] question:

[0234] "What's your first memory of seeing this photo?"

[0235] Who did you go to the beach with that day?

[0236] This system allows users to re-experience past memories in an immersive virtual reality space, stimulating and improving memory.

[0237] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0238] Step 1:

[0239] Input: A previous photo selected by the user on their device

[0240] How it works: A user opens an application on their device and selects a photo from their past, such as a family vacation or a special event. The device previews the selected photo, and the user clicks the "Upload" button.

[0241] Output: Photo data ready to upload

[0242] Step 2:

[0243] Input: The "Upload" button the user clicked

[0244] Operation: The device sends the photo data to the server, and uses a business API to display the network connection status and progress.

[0245] Output: Photo data sent to the server

[0246] Step 3:

[0247] Input: Photo data received by the server

[0248] How it works: The server temporarily stores the received photo data and begins analyzing it using an image analysis engine (OpenCV or TensorFlow). Through this analysis, it identifies people and objects in the photo and uses facial recognition technology (Dlib, FaceNet, etc.) to identify people in the photo.

[0249] Output: Analysis results (information on identified people and objects)

[0250] Step 4:

[0251] Input: Analysis results (information on identified people and objects)

[0252] How it works: The server extracts the photo's metadata (date and time of capture, location, etc.) from the EXIF ​​data and retrieves the identified person's information from an internal database.

[0253] Output: Extracted metadata and identified people

[0254] Step 5:

[0255] Input: Extracted metadata and identified people

[0256] How it works: The server generates a memorable episode related to the photo based on the collected metadata. For example, it generates information such as "Memories of a family trip to Shonan Beach in the summer of 2005." It also uses this information to formulate a dialogue scenario and create a list of questions to guide the conversation with the user.

[0257] Output: Generated memory episodes and dialogue scenarios

[0258] Step 6:

[0259] Input: Generated memory episodes and dialogue scenarios

[0260] Operation: The server sends the dialogue scenario and the generated question list to the device and the head-mounted display. Based on the information received from the server, the device prepares to display the dialogue scenario on the user's screen.

[0261] Output: Dialogue scenario and question list sent to the device and head-mounted display

[0262] Step 7:

[0263] Input: Terminal and head-mounted display that received the dialogue scenario and question list

[0264] How it works: The head-mounted display displays the dialogue scenario in a virtual reality space, allowing the user to begin the immersive experience. The device also displays the initial dialogue questions to the user, notifying them that they are ready to begin the dialogue.

[0265] Output: Display of the interaction scenario in a virtual reality space

[0266] Step 8:

[0267] Input: User response (text input or voice input)

[0268] How it works: The device records the user's responses and continuously sends them to the server.

[0269] Output: The user's response sent to the server

[0270] Step 9:

[0271] Input: The user's response sent to the server

[0272] How it works: The server analyzes the user's response using a natural language processing engine (e.g., spaCy) and generates the next question or feedback. The dialogue with the user continues through this cycle.

[0273] Output: Next question or feedback

[0274] Step 10:

[0275] Input: Next question or feedback sent from the server

[0276] Action: The head-mounted display and device present the next question or feedback to the user, and the user continues the interaction.

[0277] Output: The next question or feedback presented to the user

[0278] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0279] The present invention is a system that allows users to use past photos to stimulate their memories and alleviate the early symptoms of dementia. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system aims to optimize the interaction experience and improve the quality of feedback. A specific embodiment of the system is described below.

[0280] System Overview

[0281] This system achieves its functions through the interaction of three parties: the server, the device, and the user. Users upload past photos from their device, and the server analyzes the photos to generate a dialogue scenario. In doing so, it uses an emotion engine to recognize the user's emotions and provide more appropriate questions and feedback.

[0282] System processing flow

[0283] 1. User selects and uploads a photo:

[0284] User: First, launch the application on the device and select some past photos. For example, you might select photos from a family trip or a special event.

[0285] On the device: Shows the user a preview of the selected photo and provides an "Upload" button.

[0286] User: Check the preview and click the "Upload" button.

[0287] 2. The device sends the photo data to the server:

[0288] On the device: Uses a business API to send the selected photo data to the server. Displays the progress of the network connection to the user.

[0289] 3. The server receives the photo and begins analyzing it:

[0290] Server: Temporarily stores the received photo data in storage and launches an image analysis engine, using technologies such as OpenCV and TensorFlow.

[0291] Server: Identifies people and objects in the photo and uses facial recognition technology to identify people in the shot, leveraging libraries like Dlib and FaceNet.

[0292] 4. The server collects the metadata:

[0293] Server: Extracts photo metadata (e.g., EXIF ​​data) and captures the date and time of the photo and its location, which allows us to determine when and where the photo was taken.

[0294] Server: Refers to an internal database and obtains detailed information about the people in the photo (e.g., names, relationships, etc.) from the facial recognition results.

[0295] 5. The server generates the relevant memories:

[0296] Server: Based on the collected metadata, the server creates a story about a memorable episode related to the photo. For example, it creates a story like, "In the summer of 2005, we went on a family trip to Shonan Beach."

[0297] Server: Based on this generated information, a dialogue scenario is formulated and a list of questions is created to guide the conversation with the user.

[0298] 6. Introducing the Emotion Engine:

[0299] Server: Activates the emotion engine and recognizes emotions from the user's voice and text input. This emotion information is used to adjust the dialogue scenario and generate feedback.

[0300] 7. The server sends the dialogue scenario to the device:

[0301] Server: Sends a dialogue scenario and a list of questions based on emotional information to the device, enabling flexible dialogue according to the user's emotions.

[0302] 8. The device presents the dialogue scenario to the user:

[0303] Terminal: The initial dialogue questions are displayed to the user, preparing them to start the dialogue. The questions are displayed taking into account emotional information.

[0304] User: Responds to questions displayed on the screen by typing text or speaking.

[0305] 9. User and system begin interaction:

[0306] Terminal: Records the user's responses and sends them to the server in real time.

[0307] Server: Analyzes the user's response and uses natural language processing technology and an emotion engine to generate the next question or feedback.

[0308] 10. The server analyzes the dialogue and provides feedback:

[0309] Server: Collects interaction logs and evaluates the user's emotional state and memory vividness, adjusting the interaction content and providing personalized feedback.

[0310] Server: Generates personalized feedback tailored to the user's memory impairment and provides it to the user via the device.

[0311] Specific examples

[0312] For example, consider a scenario in which a user selects a photo from a family trip from their childhood and starts a conversation based on the topic of "Memories of having fun with my family at Shonan Beach in the summer of 2005." The user uploads the photo, and the server analyzes it to generate anecdotes and questions like the following:

[0313] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[0314] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[0315] In addition, an emotion engine is used to recognize the user's emotions, and questions that are more in line with the user's emotions are added, such as, "You seem to be having fun. Tell me more about what happened that day."

[0316] This system allows users to enjoy past memories while stimulating and improving their memory. The introduction of an emotion engine makes the interaction more personalized, improving the user experience.

[0317] The processing flow will be explained below.

[0318] Step 1:

[0319] The user launches the application on the device, opens an interface within the application for selecting past photos, and selects a specific photo from the saved photos.

[0320] Step 2:

[0321] The device displays a preview of the selected photo to the user and provides an "Upload" button to confirm that the photo is correct.

[0322] Step 3:

[0323] The user clicks the "Upload" button, which completes preparations for uploading the photo data.

[0324] Step 4:

[0325] The device sends the selected photo data to the server using a business API, and the progress of the network connection is displayed to the user.

[0326] Step 5:

[0327] The server stores the received photo data in temporary storage, then launches the image analysis engine and begins analyzing the photo.

[0328] Step 6:

[0329] The server uses an image analysis engine to identify people and objects in the photo, for example, by using facial recognition technology to identify people, using libraries such as Dlib and FaceNet.

[0330] Step 7:

[0331] The server extracts the photo's metadata (e.g., EXIF ​​data) and obtains the date and time the photo was taken and its location, which identifies when and where the photo was taken.

[0332] Step 8:

[0333] The server references an internal database and obtains detailed information about the people in the photo (e.g., names, relationships, etc.) from the facial recognition results.

[0334] Step 9:

[0335] The server generates information about past memories based on the analysis results and metadata. For example, it creates a story such as "I went on a family trip to Shonan Beach in the summer of 2005."

[0336] Step 10:

[0337] The server creates a dialogue scenario based on the generated memory information, which includes a list of questions to prompt dialogue with the user.

[0338] Step 11:

[0339] The emotion engine is activated. The server extracts emotion data from the user's voice and text and adjusts the dialogue scenario based on this data.

[0340] Step 12:

[0341] The server uses the user's emotional data to generate more personalized questions and feedback. For example, if the user seems to be having fun, it will generate questions like, "What other fun things happened that day?"

[0342] Step 13:

[0343] The server sends a dialogue scenario and a list of questions based on emotion data to the terminal.

[0344] Step 14:

[0345] The terminal prepares to display the dialogue scenario received from the server on the user screen, presents the first question, and notifies the user that the dialogue is ready to begin.

[0346] Step 15:

[0347] The user responds to questions displayed on the screen by text input or voice input, and the device records the user's responses.

[0348] Step 16:

[0349] The device sends the user's responses in real time to the server, which then generates the next question or feedback based on the received data.

[0350] Step 17:

[0351] The server analyzes the user's responses and uses natural language processing techniques and an emotion engine to assess the user's memory state and emotional changes.

[0352] Step 18:

[0353] The server generates the next question and feedback and sends it to the device, which may include personalized questions and suggestions to help the user improve their memory.

[0354] Step 19:

[0355] The device receives new questions and feedback from the server and presents them to the user, allowing the user to continue interacting and stimulating their memory.

[0356] Step 20:

[0357] When the user indicates an intention to end the interactive session (for example, by clicking an "end" button), the terminal notifies the server of this information.

[0358] Step 21:

[0359] The server acknowledges the end of the interaction session and generates a summary and feedback of the interaction, including an assessment of the user's memory state and the progress of the interaction.

[0360] Step 22:

[0361] The device displays feedback from the server to the user, suggests next sessions, saves records, and allows the user to see the results of the interaction.

[0362] Example 2

[0363] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0364] In recent years, with the advancement of an aging society, the prevention and treatment of dementia have become important issues. Memory retention and emotional stability are particularly important issues in the early stages of dementia. Current methods offer limited means for users to stimulate past memories, and more effective methods are needed to recreate and evoke past memories. Furthermore, due to the lack of systems that can adequately grasp users' emotional states and provide feedback through dialogue, there is a need for methods to improve the quality of dialogue.

[0365] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving photo data from a user's terminal, means for analyzing the received photo data using an image analysis engine to identify people and objects in the photo, means for collecting photo metadata based on the analysis results and generating past memory information, means for creating a dialogue scenario based on the generated memory information, means including an emotion recognition engine for detecting the user's emotions and adjusting the dialogue scenario, means for providing flexible dialogue based on the generated dialogue scenario and emotion information, and means for analyzing data obtained from the dialogue and generating and providing individual feedback. This makes it possible to effectively recreate past memories and promote memory retention and emotional stability through dialogue that is in line with the user's emotions.

[0366] "User" means a person who uses the system.

[0367] A "terminal" refers to a device operated by a user, specifically a PC, smartphone, tablet, etc.

[0368] "Photo data" refers to image files imported by the user.

[0369] "Means for receiving" refers to the function for the server to receive data sent from the terminal.

[0370] An "image analysis engine" is software that processes and analyzes image data to extract useful information.

[0371] "Means for identifying people and objects in photographs" refers to the ability to use an image analysis engine to recognize and identify specific elements within a photograph.

[0372] "Metadata" refers to detailed information that accompanies photo data, including the date the photo was taken and location information.

[0373] "Memories" refers to episodes and stories related to past memories generated based on photos and metadata.

[0374] A "dialogue scenario" is a plan that defines the progress of a conversation with a user, constructed based on the generated memory information.

[0375] An "emotion recognition engine" is software that analyzes and recognizes emotions from a user's voice and text data.

[0376] "Means for providing flexible dialogue" refers to a function that adjusts dialogue scenarios based on emotional information and provides questions and feedback that are appropriate for the user.

[0377] "Feedback" refers to responses and advice provided by a system based on interactions with a user.

[0378] "Means for analyzing data" refers to the function of analyzing information obtained from the dialogue and evaluating the user's condition and response.

[0379] "Means for generating and providing individual feedback" refers to a function that creates feedback suited to each individual user based on the analysis results and provides that feedback via the terminal.

[0380] The present invention provides a system that allows users to use past photographs to stimulate their memories and alleviate early symptoms of dementia. This system functions through the interaction of three parties: the user, the terminal, and the server. Specific embodiments are described below.

[0381] System Overview

[0382] Hardware

[0383] Server: The central server that performs the main processing of this system.

[0384] Terminal: A device operated by a user, such as a PC, smartphone, or tablet.

[0385] software

[0386] Image analysis engine: Uses image processing libraries such as OpenCV and TensorFlow.

[0387] Emotion recognition engine: Emotion analysis utilizes technologies such as IBM Watson (registered trademark) and Microsoft (registered trademark) Azure (registered trademark) Emotion API.

[0388] Database: A database for storing metadata and user information.

[0389] Natural language processing engine: Analyzes user input and generates dialogue scenarios.

[0390] System Operation

[0391] 1. Select and upload a photo

[0392] First, the user launches the application on the device. The device displays the user's photo library and allows the user to select memorable photos. For example, the user may select photos from a family trip or a special event. The selected photos are previewed on the device and then sent to the server when the user clicks the "Upload" button.

[0393] 2. Receiving and analyzing photos

[0394] The server receives the photo data sent from the device and temporarily stores it in storage. It then begins analyzing the photo using OpenCV and TensorFlow. The image analysis engine identifies people and objects in the photo and uses facial recognition technology to identify who is in the photo. This process utilizes the Dlib and FaceNet libraries.

[0395] 3. Collecting metadata and generating memories

[0396] The server analyzes the EXIF ​​data of the photo to obtain the date and time the photo was taken and location information. Based on this metadata, it determines when and where the photo was taken. It then combines the results of the image analysis with information from the database to obtain detailed information about the people in the photo (such as their names and relationships). Based on this information, the server then creates a document that describes memorable episodes related to the photo.

[0397] 4. Emotion Recognition and Dialogue Scenario Generation

[0398] Based on the generated memory information, the server formulates a dialogue scenario. It activates an emotion recognition engine to recognize emotions from the user's voice and text input. Based on this emotion information, the dialogue scenario is adjusted to provide more appropriate questions and feedback.

[0399] 5. Providing dialogue scenarios and feedback

[0400] The server sends a list of questions based on the generated dialogue scenario and emotional information to the device. The device displays this to the user, preparing to begin the dialogue. The user responds to the questions displayed on the screen by text input or voice input. The device sends these responses to the server in real time. The server analyzes the responses sent by the user, adjusts the dialogue content, and provides feedback appropriate to each individual.

[0401] Specific examples

[0402] For example, a specific example will be given in which the user selects a photo of "a family trip to Shonan Beach in the summer of 2005."

[0403] Prompt statement:

[0404] "I will upload photos from a family trip to Shonan Beach in the summer of 2005. Please generate a dialogue scenario in which the user talks about their memories of that time."

[0405] Generated episodes:

[0406] "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family enjoyed playing in the sea."

[0407] Generated question:

[0408] "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[0409] Additional input from the emotion engine:

[0410] "Sounds fun. Tell me more about that day."

[0411] In this way, the system stimulates memories while allowing users to enjoy past memories, contributing to the prevention and alleviation of dementia.The introduction of an emotion engine makes the dialogue more personalized, improving the user experience.

[0412] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0413] Step 1:

[0414] User selects and uploads a photo

[0415] How it works: A user launches an application on their device and selects a photo from their photo library, such as a photo from a trip or family event.

[0416] Input: A photo file selected by the user.

[0417] Output: The device displays a preview of the selected photo. An "Upload" button is provided.

[0418] Specific action: The user checks the preview and clicks the "Upload" button.

[0419] Step 2:

[0420] The device sends the photo data to the server.

[0421] Operation: The device uses the business API to send photo data to the server. The progress of the transfer is displayed to the user.

[0422] Input: The photo data selected by the user and uploaded.

[0423] Output: The photo data sent to the server.

[0424] Specific operation: The device sends data to the server via the network.

[0425] Step 3:

[0426] The server receives the photo and begins analyzing it.

[0427] How it works: The server receives the photo data and temporarily stores it in storage. OpenCV and TensorFlow are used for photo analysis.

[0428] Input: Photo data sent from the device.

[0429] Output: As a result of the analysis, specific information about people and objects in the photo.

[0430] Specific operation: The server preprocesses the image and detects human faces. Dlib and FaceNet are used as facial recognition technologies.

[0431] Step 4:

[0432] The server collects the photo metadata.

[0433] How it works: The server analyzes the EXIF ​​data of the photo to obtain the date and time the photo was taken and its location.

[0434] Input: Parsed photo data.

[0435] Output: Date and time of capture, location, and other metadata.

[0436] What it does: The server compares the retrieved metadata with an internal database to determine when and where the photo was taken.

[0437] Step 5:

[0438] The server generates the memory information

[0439] How it works: The server uses collected metadata and facial recognition results to create a document of memorable events related to the photo.

[0440] Input: Metadata, face recognition results.

[0441] Output: A written version of the memory episode.

[0442] Concrete action: For example, generate an episode such as "In the summer of 2005, we went on a family trip to Shonan Beach."

[0443] Step 6:

[0444] The server generates a dialogue scenario

[0445] How it works: The server creates a dialogue scenario with the user based on the generated memory information. It uses a generation engine to create a list of questions to guide the dialogue.

[0446] Input: A written memory episode.

[0447] Output: Dialogue scenario and questionnaire.

[0448] Concrete Actions: Formulate specific questions, such as, "What is your first memory of seeing this photo?"

[0449] Step 7:

[0450] Introducing an emotion recognition engine

[0451] How it works: The server launches an emotion recognition engine to recognize emotions from the user's voice and text input.

[0452] Input: User voice input, text input.

[0453] Output: Emotional information.

[0454] Specific actions: Emotional information is used to adjust the dialogue scenario.

[0455] Step 8:

[0456] The server sends the dialogue scenario to the terminal.

[0457] Operation: The server sends a dialogue scenario and a list of questions based on emotion information to the terminal.

[0458] Input: Dialogue scenario, emotion information.

[0459] Output: The dialogue scenario and questionnaire sent to the terminal.

[0460] Concrete behavior: This enables flexible interaction according to the user's emotions.

[0461] Step 9:

[0462] The device presents a dialogue scenario to the user.

[0463] Operation: The terminal displays the initial dialogue questions to the user and prepares to begin the dialogue.

[0464] Input: Dialogue scenario and question list sent from the server.

[0465] Output: The dialogue questions that are displayed to the user.

[0466] Specific actions: The user responds to questions displayed on the screen by text or voice.

[0467] Step 10:

[0468] The user and the system start the conversation

[0469] How it works: The user responds to questions by typing or speaking.

[0470] Input: The user's response.

[0471] Output: User responses logged to the terminal.

[0472] Specific operation: The terminal sends the response to the server in real time.

[0473] Step 11:

[0474] The server analyzes the conversation and provides feedback

[0475] How it works: The server analyzes the user's responses and adjusts the interaction to provide personalized feedback.

[0476] Input: The user's response sent from the terminal.

[0477] Output: Feedback to the user with next questions or advice.

[0478] Specific operation: The server uses natural language processing technology to analyze the user's response and generate the next question or feedback.

[0479] (Application example 2)

[0480] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0481] Conventional memory stimulation systems do not adequately consider the user's emotions, resulting in a lack of personalized interaction experiences and the inability to effectively stimulate memory or improve skills. Furthermore, it is difficult to evaluate and provide feedback on the user's skills and knowledge in real time.

[0482] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving photo data from a user's terminal, means for analyzing the received photo data using an image analysis engine and identifying objects in the photo, means for collecting photo attribute information and generating past event information based on the analysis results, means for creating a dialogue scenario based on the generated event information, means for recognizing emotions from the user's voice or text input and adjusting the dialogue scenario, means for conducting a dialogue with the user based on the dialogue scenario, and means for evaluating the user's skill and knowledge status and providing feedback. This enables dialogue optimized for individual users and real-time skill evaluation and feedback.

[0483] "User's device" refers to an electronic device used by a user, such as a smartphone, tablet, PC, or wearable device.

[0484] "Photo data" refers to still image data that a user has taken, saved, or selected through a terminal.

[0485] An "image analysis engine" is a software module that analyzes received image data and identifies people and objects within the image.

[0486] "Subjects" are elements that are to be recognized, such as people, animals, objects, and scenery, contained in an image.

[0487] "Attribute information" refers to metadata and context information related to image data, and includes, for example, the date and time of shooting, location information, shooting environment, and subject characteristics.

[0488] "Past event information" is an episode or story related to a specific time or place in the past, generated based on attribute information.

[0489] A "dialogue scenario" is a script of a series of questions and answers for dialogue with a user, constructed based on generated past event information.

[0490] "Emotion" recognition refers to analyzing voice or text input to identify a user's emotional state (e.g., joy, sadness, anger, etc.).

[0491] "Skill and knowledge status" refers to the level of skill and knowledge that a user possesses regarding a specific operation or task.

[0492] The "means for providing feedback" is a software module for providing appropriate advice and evaluation in real time based on the user's answers and actions.

[0493] This invention is a system that aims to alleviate the early symptoms of dementia by stimulating the user's memory using past photographs. This system functions through the interaction of three parties: the server, the terminal, and the user. Specifically, it has the following features:

[0494] 1. Receiving and analyzing photo data

[0495] The device has a function that allows users to select and upload past photos. It is assumed that users will launch an application on the device and select photos from a family trip or a special event. The selected photos are sent to the server by pressing the "upload" button on the device.

[0496] The server temporarily stores the received photo data in storage and launches an image analysis engine (such as OpenCV or TensorFlow). Object recognition technology (such as YOLOv3 or OpenCV) is used to identify the subject in the photo. It also collects photo attribute information (such as EXIF ​​data) and obtains the date and time the photo was taken and its location. Based on the results of this analysis, past event information is generated.

[0497] 2. Dialogue scenario generation and emotion recognition

[0498] The server creates a dialogue scenario based on the generated past event information. It then uses an emotion engine that analyzes voice and text input to recognize the user's emotions and adjust the dialogue scenario accordingly. During this process, questions and feedback based on the emotion information are added.

[0499] 3. User interaction and feedback

[0500] The device presents the dialogue scenario sent from the server to the user and initiates the dialogue. The user responds to questions displayed on the screen by text input or voice input. The user's responses are sent in real time to the server, which then uses natural language processing technology and an emotion engine to generate the next question and feedback.

[0501] 4. Skill evaluation and feedback

[0502] The server collects dialogue logs and evaluates the user's skills and knowledge. Based on this, individual feedback is generated and provided to the user via their device. This feedback evaluates the user's skill and knowledge level for specific operations and tasks, and provides appropriate advice and evaluation in real time.

[0503] Specific examples

[0504] For example, consider a scenario in which a user selects a photo from a family trip from their childhood and starts a conversation based on the topic of "Memories of having fun with my family at Shonan Beach in the summer of 2005." The user uploads the photo, and the server analyzes it to generate anecdotes and questions like the following:

[0505] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[0506] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[0507] In addition, an emotion engine is used to recognize the user's emotions, and questions that are more in line with their emotions are added, such as, "You seem to be having fun. Tell me more about what happened that day."

[0508] Prompt Sentence Examples

[0509] "Please upload photos of your market so we can train you on your business."

[0510] "What is in the uploaded photo? For example, what are the names of machine parts or devices?"

[0511] This system allows users to enjoy past memories while stimulating and improving their memory. The introduction of an emotion engine makes interactions more personalized, improving the user experience.

[0512] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0513] Step 1:

[0514] The device allows the user to select past photos and upload them through the application (input: photo data selected by the user, output: photo data sent to the server). The device displays the selected photos as a preview and provides an upload button. When the user clicks the upload button, the photo data is sent to the server.

[0515] Step 2:

[0516] The server temporarily stores the received photo data in storage and launches an image analysis engine (such as OpenCV or TensorFlow) (input: photo data, output: photo data stored in storage). This engine uses object recognition technology (such as YOLOv3 or OpenCV) to identify objects in the photo.

[0517] Step 3:

[0518] The server analyzes the attribute information of the photo (e.g., EXIF ​​data) and collects the attribute information (input: photo data saved in storage, output: attribute information). This allows the server to identify when and where the photo was taken.

[0519] Step 4:

[0520] The server generates past event information based on the analyzed attribute information (input: attribute information, output: past event information). For example, it generates a specific episode such as "In the summer of 2005, we went on a family trip to Shonan Beach."

[0521] Step 5:

[0522] The server creates a dialogue scenario based on the generated past event information (input: past event information, output: dialogue scenario). Furthermore, it uses an emotion engine that analyzes voice and text input to recognize the user's emotions and adjust the dialogue scenario accordingly.

[0523] Step 6:

[0524] The server creates questions and feedback based on the generated dialogue scenario and emotional information and sends them to the terminal (input: dialogue scenario, emotional information; output: questions and feedback sent to the terminal).

[0525] Step 7:

[0526] The terminal presents the dialogue scenario sent from the server to the user and starts the dialogue (input: questions and feedback sent from the server, output: user's response). The user responds to the questions displayed on the screen by text input or voice input.

[0527] Step 8:

[0528] The terminal records the user's response and transmits it to the server in real time (input: user's response data, output: real-time response data transmitted to the server).

[0529] Step 9:

[0530] The server analyzes the user's response data and generates the next question and feedback using natural language processing technology and an emotion engine (input: user's real-time response data, output: next question and feedback).

[0531] Step 10:

[0532] The server collects dialogue logs and evaluates the user's skills and knowledge (input: dialogue log data, output: user skill evaluation data). Based on this, individual feedback is generated and provided to the user via their device.

[0533] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0534] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0535] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0536] [Second embodiment]

[0537] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0538] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0539] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0540] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0541] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0542] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0543] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0544] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0545] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0546] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0547] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0548] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0549] The present invention provides a system for stimulating a user's memory by using past photographs and alleviating early symptoms of dementia. Specific embodiments of the system will be described below.

[0550] System Overview

[0551] This system works by interacting with the server, the device, and the user. The user uploads past photos from their device to the server, and the server analyzes the photos to generate a dialogue scenario, which is then presented to the user via their device. The user can then have a conversation based on the scenario, allowing them to enjoyably reminisce about their memories.

[0552] System processing flow

[0553] 1. User selects and uploads a photo:

[0554] User: First, open the application on the device and select some old photos, such as photos from a family trip or a special event.

[0555] Terminal: It then displays a preview of the selected photo to the user and provides an "Upload" button.

[0556] User: Check the preview and click the "Upload" button.

[0557] 2. The device sends the photo data to the server:

[0558] Terminal: Uses a business API to send the selected photo data to the server. At this time, the network connection status and progress are displayed to the user.

[0559] 3. The server receives the photo and begins analyzing it:

[0560] Server: Temporarily stores the received photo data in storage and launches an image analysis engine, using technologies such as OpenCV and TensorFlow.

[0561] Server: Identifies people and objects in the photo and uses facial recognition technology to identify people in the shot, leveraging libraries like Dlib and FaceNet.

[0562] 4. The server collects the metadata:

[0563] Server: Extracts photo metadata (such as the date and time of the photo, location information, etc.) from the EXIF ​​data and obtains information about identified people from an internal database. For example, it may collect information such as that the photo was taken at Shonan Beach or that a specific person is a family member.

[0564] 5. The server generates the relevant memories:

[0565] Server: Based on the collected metadata, the server creates text about memorable episodes related to the photo. For example, it generates information such as "Memories of a family trip to Shonan Beach in the summer of 2005."

[0566] Server: Based on this generated information, a dialogue scenario is formulated and a list of questions is created to guide the conversation with the user.

[0567] 6. The server sends the dialogue scenario to the device:

[0568] Server: Sends the dialogue scenario and the generated question list to the terminal.

[0569] Terminal: Prepares to display the dialogue scenario on the user's screen based on the information received from the server.

[0570] 7. The device presents the dialogue scenario to the user:

[0571] Terminal: Displays the initial dialogue questions to the user and notifies them that they are ready to begin the dialogue.

[0572] User: Responds to questions displayed on the device by text input or voice input.

[0573] 8. User and system begin interaction:

[0574] Terminal: Records the user's responses and continuously sends them to the server.

[0575] Server: Analyzes the user's response, generates and selects the next question or feedback, and sends it to the device. Through this cycle, the user can continue the dialogue.

[0576] 9. The server analyzes the dialogue and provides feedback:

[0577] Server: Collects dialogue logs and analyzes user responses using natural language processing technology. Evaluates the user's memory status and emotional changes, and uses this information for the next dialogue.

[0578] Server: Creates individual feedback and follow-up questions based on the user's memory impairment status and provides them to the user via the device.

[0579] Specific examples

[0580] For example, consider a scenario in which a user selects a photo from a family trip from their childhood and starts a conversation based on the topic of "Memories of having fun with my family at Shonan Beach in the summer of 2005." The user uploads the photo, and the server analyzes it to generate anecdotes and questions like the following:

[0581] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[0582] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[0583] As the user answers these questions, the system elicits more detailed information and stimulates the user's memory.

[0584] This system allows users to enjoy memories of the past while stimulating and improving their memory.

[0585] The processing flow will be explained below.

[0586] Step 1:

[0587] The user launches the application on the device, opens an interface within the application for selecting past photos, and selects a specific photo from the saved photos.

[0588] Step 2:

[0589] The device displays a preview of the selected photo to the user and provides an "Upload" button to confirm that the photo is correct.

[0590] Step 3:

[0591] The user clicks the "Upload" button, which completes preparations for uploading the photo data.

[0592] Step 4:

[0593] The device sends the selected photo data to the server using a business API, and the progress of the network connection is displayed to the user.

[0594] Step 5:

[0595] The server stores the received photo data in temporary storage, then launches the image analysis engine and begins analyzing the photo.

[0596] Step 6:

[0597] The server uses an image analysis engine to identify people and objects in the photo, for example, by using facial recognition technology to identify people, using libraries such as Dlib and FaceNet.

[0598] Step 7:

[0599] The server extracts the photo's metadata (e.g., EXIF ​​data) and obtains the date and time the photo was taken and its location, which identifies when and where the photo was taken.

[0600] Step 8:

[0601] The server references an internal database and obtains detailed information about the people in the photo (e.g., names, relationships, etc.) from the facial recognition results.

[0602] Step 9:

[0603] The server generates information about past memories based on the analysis results and metadata. For example, it creates a story such as "I went on a family trip to Shonan Beach in the summer of 2005."

[0604] Step 10:

[0605] The server creates a dialogue scenario based on the generated memory information, which includes a list of questions to prompt dialogue with the user.

[0606] Step 11:

[0607] The server sends the created dialogue scenario to the terminal. The scenario includes episodes and related questions.

[0608] Step 12:

[0609] The terminal prepares to display the dialogue scenario received from the server on the user screen, presents the first question, and notifies the user that the dialogue is ready to begin.

[0610] Step 13:

[0611] The user responds to questions displayed on the screen by text input or voice input, and the device records the user's responses.

[0612] Step 14:

[0613] The device sends the user's responses in real time to the server, which then generates the next question or feedback based on the received data.

[0614] Step 15:

[0615] The server analyzes the user's responses and uses natural language processing techniques (e.g., sentiment analysis) to assess the user's memory state and emotional changes.

[0616] Step 16:

[0617] The server generates the next question and feedback and sends it to the device, which may include personalized questions and suggestions to help the user improve their memory.

[0618] Step 17:

[0619] The device receives new questions and feedback from the server and presents them to the user, allowing the user to continue interacting and stimulating their memory.

[0620] Step 18:

[0621] When the user indicates an intention to end the interactive session (for example, by clicking an "end" button), the terminal notifies the server of this information.

[0622] Step 19:

[0623] The server acknowledges the end of the interaction session and generates a summary and feedback of the interaction, including an assessment of the user's memory state and the progress of the interaction.

[0624] Step 20:

[0625] The device displays feedback from the server to the user, suggests next sessions, saves records, and allows the user to see the results of the interaction.

[0626] Example 1

[0627] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0628] Users in the early stages of dementia have difficulty maintaining and recalling memories in their daily lives, which increases their mental burden. Current systems have limited methods for effectively utilizing users' past photos to stimulate memories, and the lack of an interactive approach reduces opportunities for users to actively reflect on the past. Furthermore, the system lacks mechanisms for monitoring the progress of memory impairment and providing appropriate feedback.

[0629] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0630] In this invention, the server includes a means for receiving image data from the user's information processing device, a means for analyzing the received image data using an image analysis device and identifying objects in the image, a means for collecting additional information about the image based on the analysis results and generating past memory information, a means for creating a dialogue plan based on the generated memory information, a means for engaging in a dialogue with the user based on the dialogue plan, a means for recording the user's responses during the dialogue and transmitting the data, a means for generating dialogue questions and feedback using a generative AI model, and a means for collecting dialogue logs and analyzing them using natural language processing technology. This allows the user to use past photos to stimulate their memories and efficiently recall them. Furthermore, by monitoring the user's memory status through dialogue and providing appropriate feedback, it is possible to maintain and improve cognitive function.

[0631] "User" refers to the person who uses this system, and is particularly targeted at people in the early stages of dementia.

[0632] An "information processing device" is a terminal or device operated by a user, and allows operations such as selecting and uploading photos.

[0633] "Image data" refers to digital images such as photographs and drawings uploaded by users.

[0634] An "image analysis device" is a device that analyzes image data and identifies subjects (people or objects) within the image, and includes an image analysis engine and library.

[0635] "Additional information" refers to metadata such as the image's EXIF ​​data, as well as information related to the subject matter within the image.

[0636] "Memories" refers to past events and episodes that are generated based on the results of image analysis and additional information.

[0637] "Dialogue planning" refers to a scenario or question list that guides a dialogue with the user based on memory information.

[0638] "Generative AI model" refers to an artificial intelligence model used to automatically generate dialogue questions and feedback.

[0639] "Natural language processing technology" refers to technology for analyzing dialogue logs and understanding user responses, and includes technology used for sentence generation and sentiment analysis.

[0640] The present invention provides a system for stimulating past memories in a user and alleviating early symptoms of dementia. Specific processing and embodiments of the program of this system will be described in detail below.

[0641] This system functions through the interaction of three parties: a server, a terminal, and a user. The server is responsible for a wide range of processes, including receiving and analyzing image data, extracting metadata, and generating a dialogue plan. The terminal provides an interface for users to upload photos and initiate dialogue. Users use this interface to select and upload photos and engage in dialogue according to the dialogue scenario provided by the server.

[0642] Server processing

[0643] The server first receives image data from the user. To do this, the server receives image data sent from the device using a RESTful API. The received image data is then temporarily stored in cloud storage. The main technology used is likely to be a cloud service such as Amazon S3.

[0644] The stored image data is analyzed using an image analyzer (e.g., OpenCV or TensorFlow) to identify people and objects in the image and use facial recognition technology (e.g., Dlib or FaceNet). Based on the analysis results, the EXIF ​​data of the photo is extracted and metadata such as the date and time of the photo's capture and location are collected using a library such as ExifTool.

[0645] The collected metadata and analysis results are integrated to generate memory information. Natural language generation technology (generative AI model) is used to create sentences about specific episodes, such as "Memories of a family trip to Shonan Beach in the summer of 2005." Based on the generated memory information, a dialogue plan is also created, and a list of questions is created to guide the conversation with the user.

[0646] The generated dialogue plan and question list are sent to the terminal via encrypted communication.

[0647] About terminal processing

[0648] The device accepts operations from the user and sends them to the server. The user opens the dedicated application, selects and uploads photos. During uploading, a progress bar is displayed to notify the user of the progress. The device displays the dialogue scenario received from the server, allowing the user to start a dialogue. During the dialogue, the device records the user's responses and continuously sends them to the server.

[0649] About User Roles

[0650] The user selects and uploads past photos using the device. Based on the dialogue scenario provided by the server, the user responds by text input or voice input. Through this process, the user's past memories are stimulated, and it is expected that their memory will be improved.

[0651] Specific examples

[0652] For example, if a user uploads a photo of a family trip to Shonan Beach in the summer of 2005, the server processes it as follows: First, it extracts "Date and time of photo: Summer 2005, Location: Shonan Beach" from the photo's EXIF ​​data. Next, it uses facial recognition technology to identify family members and generates anecdotes and questions like the following:

[0653] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[0654] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[0655] As the user answers these questions, the system elicits more detailed information and stimulates the user's memory.

[0656] Prompt Sentence Examples

[0657] "Generate a conversation scenario based on memories from photos of a family trip to Shonan Beach."

[0658] "Create a story related to the person in this photo."

[0659] As described above, this system aims to effectively stimulate the user's memory by linking various hardware and software.

[0660] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0661] Step 1: Select and upload a photo

[0662] Subject: User

[0663] Input: Photo data selected on your device

[0664] Output: Uploaded photo data

[0665] The user opens a dedicated application on their device and selects a past photo. The selected photo is previewed, and when the user clicks the "Upload" button, the photo data is sent to the server.

[0666] Step 2: Receiving and saving photo data

[0667] Subject: Server

[0668] Input: Photo data uploaded by the user

[0669] Output: Saved photo data (cloud storage)

[0670] The server receives the photo data sent from the device and temporarily stores it in cloud storage, using a cloud service such as Amazon S3.

[0671] Step 3: Analyze the photo data

[0672] Subject: Server

[0673] Input: Saved photo data

[0674] Output: Analysis results (specific information about people and objects)

[0675] The server analyzes the stored photo data using image analysis equipment (e.g., OpenCV or TensorFlow). The analysis identifies people and objects in the photos. The analysis results also include information on people identified using facial recognition technology (Dlib or FaceNet).

[0676] Step 4: Collecting Metadata

[0677] Subject: Server

[0678] Input: Analysis results and photo EXIF ​​data

[0679] Output: Collected metadata (date and time of photo, location, etc.)

[0680] The server extracts the EXIF ​​data from the photos using tools such as ExifTool, and collects metadata such as the date and time the photo was taken and the location information. This is then combined with the analysis results to collect information on identified individuals.

[0681] Step 5: Generate memories

[0682] Subject: Server

[0683] Input: Collected metadata and analysis results

[0684] Output: Generated memories

[0685] The server uses natural language generation technology (generative AI model) to generate memory information based on the collected metadata and analysis results. For example, a detailed episode such as "Memories of a family trip to Shonan Beach in the summer of 2005" can be written.

[0686] Step 6: Developing dialogue scenarios

[0687] Subject: Server

[0688] Input: Generated memory information

[0689] Output: Dialogue scenario and questionnaire

[0690] The server formulates a dialogue plan based on the generated memory information and creates a specific list of questions for the user, such as "What is your first memory of seeing this photo?"

[0691] Step 7: Sending the dialogue scenario to the device

[0692] Subject: Server

[0693] Input: Dialogue scenario and questionnaire

[0694] Output: Dialogue scenario and question list sent to the terminal

[0695] The server sends the dialogue scenario and question list to the terminal through encrypted communication.

[0696] Step 8: Present the dialogue scenario

[0697] Subject: Device

[0698] Input: Dialogue scenario and question list received from the server

[0699] Output: The dialogue scenario and question list presented to the user

[0700] The terminal presents the dialogue scenario and question list received from the server to the user, and displays the initial dialogue questions in a pop-up format on the screen to prompt the user to start the dialogue.

[0701] Step 9: User responses and recording

[0702] Subject: User and Device

[0703] Input: User input (text or voice)

[0704] Output: Recorded user responses

[0705] The user responds to questions displayed on the device by text input or voice input, and the device records the user's responses and continuously transmits them to the server.

[0706] Step 10: Continued dialogue and feedback

[0707] Subject: Server

[0708] Input: User response data

[0709] Output: Next dialogue question or feedback

[0710] The server analyzes the user's response data and uses a generative AI model to generate the next question or feedback, allowing the user to continue the dialogue.

[0711] These are the specific processing steps of the system, which aims to effectively stimulate the user's memory and alleviate the early symptoms of dementia.

[0712] (Application example 1)

[0713] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0714] Today, there is a need for effective methods to stimulate memories and alleviate the condition of people in the early stages of dementia. However, existing systems are limited to reviewing old photos and lack the ability to delve deeper into memories or re-experience memories in a more immersive way. Furthermore, they lack the precision to elicit memories through dialogue and the ability to personalize the user experience. Therefore, new technologies are needed to enable users to re-experience past memories in a more realistic way and stimulate and improve memory.

[0715] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0716] In this invention, the server includes means for receiving photo data from a user's device, means for analyzing the received photo data using an image analysis engine and identifying people and objects in the photo, means for collecting photo metadata and generating past memory information based on the analysis results, means for creating a dialogue scenario based on the generated memory information, means for presenting the dialogue scenario to the user and displaying it on a head-mounted display so that the user can re-experience the experience in a virtual reality space, and means for analyzing the user's response using a natural language processing engine and generating the next question and feedback. This allows the user to re-experience memories in a realistic virtual reality space, making it possible to effectively stimulate and improve memory in the early stages of dementia.

[0717] A "user" is an individual who uses the system to re-experience past memories and stimulate their memories.

[0718] "Device" means an electronic device used by a user to upload photo data and receive and display dialogue scenarios.

[0719] "Photo data" refers to image files that have been taken in the past and are uploaded by users.

[0720] The "image analysis engine" is a software component that analyzes received photo data and identifies people and objects in the photo.

[0721] "Metadata" refers to accompanying information included in photo data, such as the date and time of shooting, location information, and information about the shooting device.

[0722] "Memories information" is information about past events generated based on the results of image analysis and photo metadata.

[0723] A "dialogue scenario" is a scenario that includes questions and topics to promote dialogue with the user based on the generated memory information.

[0724] "Virtual reality" is a virtual space generated using computer graphics, in which users can experience a realistic virtual environment.

[0725] A "head-mounted display" is a display device worn by the user on the head to allow the user to visually experience a virtual reality space.

[0726] A "natural language processing engine" is a software component that analyzes a user's dialogue responses and generates the next question or feedback.

[0727] System Overview

[0728] This invention is a system that allows users to use past photographs to stimulate their memories and alleviate the early symptoms of dementia. This system functions through the interaction of a server, a terminal, and a user.

[0729] How to run the program

[0730] 1. User Conduct

[0731] Users use their devices to select past photos, such as photos from a family trip or special event, and upload them to the server.

[0732] 2. Device Operation

[0733] The terminal displays a preview of the selected photo to the user and provides an "Upload" button, which causes the terminal to send the photo data to the server.

[0734] 3. Server Processing

[0735] The server analyzes the received photo data using an image analysis engine (e.g., OpenCV or TensorFlow). It performs facial and object recognition to identify people and objects in the photo. It also recognizes identified people using libraries such as Dlib and FaceNet. It also extracts photo metadata (such as the date and time of the photo shoot and location information) from the EXIF ​​data and obtains information about the identified people from an internal database.

[0736] Next, the server generates a memorable episode related to the photo based on the collected metadata. For example, it writes down information such as "Memories of a family trip to Shonan Beach in the summer of 2005." It then formulates a dialogue scenario based on the generated information and creates a list of questions to encourage conversation with the user.

[0737] 4. Use of head-mounted displays

[0738] The dialogue scenario and question list are sent from the server to the device and head-mounted display, which generates a virtual reality space for the user to re-experience, and presents the dialogue scenario within it.

[0739] 5. Dialogue with users

[0740] The user wears a head-mounted display and responds according to the displayed dialogue scenario. The user's responses are analyzed by a natural language processing engine (e.g., spaCy), which generates the next question or feedback. This allows the dialogue with the user to continue and elicit more memories.

[0741] Specific examples

[0742] For example, the following prompts can be input to a generative AI model:

[0743] "Generate a dialogue scenario and a list of questions for memory stimulation based on this photo taken at Shonan Beach in the summer of 2005."

[0744] The generated dialogue scenario example is as follows:

[0745] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[0746] question:

[0747] "What's your first memory of seeing this photo?"

[0748] Who did you go to the beach with that day?

[0749] This system allows users to re-experience past memories in an immersive virtual reality space, stimulating and improving memory.

[0750] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0751] Step 1:

[0752] Input: A previous photo selected by the user on their device

[0753] How it works: A user opens an application on their device and selects a photo from their past, such as a family vacation or a special event. The device previews the selected photo, and the user clicks the "Upload" button.

[0754] Output: Photo data ready to upload

[0755] Step 2:

[0756] Input: The "Upload" button the user clicked

[0757] Operation: The device sends the photo data to the server, and uses a business API to display the network connection status and progress.

[0758] Output: Photo data sent to the server

[0759] Step 3:

[0760] Input: Photo data received by the server

[0761] How it works: The server temporarily stores the received photo data and begins analyzing it using an image analysis engine (OpenCV or TensorFlow). Through this analysis, it identifies people and objects in the photo and uses facial recognition technology (Dlib, FaceNet, etc.) to identify people in the photo.

[0762] Output: Analysis results (information on identified people and objects)

[0763] Step 4:

[0764] Input: Analysis results (information on identified people and objects)

[0765] How it works: The server extracts the photo's metadata (date and time of capture, location, etc.) from the EXIF ​​data and retrieves the identified person's information from an internal database.

[0766] Output: Extracted metadata and identified people

[0767] Step 5:

[0768] Input: Extracted metadata and identified people

[0769] How it works: The server generates a memorable episode related to the photo based on the collected metadata. For example, it generates information such as "Memories of a family trip to Shonan Beach in the summer of 2005." It also uses this information to formulate a dialogue scenario and create a list of questions to guide the conversation with the user.

[0770] Output: Generated memory episodes and dialogue scenarios

[0771] Step 6:

[0772] Input: Generated memory episodes and dialogue scenarios

[0773] Operation: The server sends the dialogue scenario and the generated question list to the device and the head-mounted display. Based on the information received from the server, the device prepares to display the dialogue scenario on the user's screen.

[0774] Output: Dialogue scenario and question list sent to the device and head-mounted display

[0775] Step 7:

[0776] Input: Terminal and head-mounted display that received the dialogue scenario and question list

[0777] How it works: The head-mounted display displays the dialogue scenario in a virtual reality space, allowing the user to begin the immersive experience. The device also displays the initial dialogue questions to the user, notifying them that they are ready to begin the dialogue.

[0778] Output: Display of the interaction scenario in a virtual reality space

[0779] Step 8:

[0780] Input: User response (text input or voice input)

[0781] How it works: The device records the user's responses and continuously sends them to the server.

[0782] Output: The user's response sent to the server

[0783] Step 9:

[0784] Input: The user's response sent to the server

[0785] How it works: The server analyzes the user's response using a natural language processing engine (e.g., spaCy) and generates the next question or feedback. The dialogue with the user continues through this cycle.

[0786] Output: Next question or feedback

[0787] Step 10:

[0788] Input: Next question or feedback sent from the server

[0789] Action: The head-mounted display and device present the next question or feedback to the user, and the user continues the interaction.

[0790] Output: The next question or feedback presented to the user

[0791] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0792] The present invention is a system that allows users to use past photos to stimulate their memories and alleviate the early symptoms of dementia. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system aims to optimize the interaction experience and improve the quality of feedback. A specific embodiment of the system is described below.

[0793] System Overview

[0794] This system achieves its functions through the interaction of three parties: the server, the device, and the user. Users upload past photos from their device, and the server analyzes the photos to generate a dialogue scenario. In doing so, it uses an emotion engine to recognize the user's emotions and provide more appropriate questions and feedback.

[0795] System processing flow

[0796] 1. User selects and uploads a photo:

[0797] User: First, launch the application on the device and select some past photos. For example, you might select photos from a family trip or a special event.

[0798] On the device: Shows the user a preview of the selected photo and provides an "Upload" button.

[0799] User: Check the preview and click the "Upload" button.

[0800] 2. The device sends the photo data to the server:

[0801] On the device: Uses a business API to send the selected photo data to the server. Displays the progress of the network connection to the user.

[0802] 3. The server receives the photo and begins analyzing it:

[0803] Server: Temporarily stores the received photo data in storage and launches an image analysis engine, using technologies such as OpenCV and TensorFlow.

[0804] Server: Identifies people and objects in the photo and uses facial recognition technology to identify people in the shot, leveraging libraries like Dlib and FaceNet.

[0805] 4. The server collects the metadata:

[0806] Server: Extracts photo metadata (e.g., EXIF ​​data) and captures the date and time of the photo and its location, which allows us to determine when and where the photo was taken.

[0807] Server: Refers to an internal database and obtains detailed information about the people in the photo (e.g., names, relationships, etc.) from the facial recognition results.

[0808] 5. The server generates the relevant memories:

[0809] Server: Based on the collected metadata, the server creates a story about a memorable episode related to the photo. For example, it creates a story like, "In the summer of 2005, we went on a family trip to Shonan Beach."

[0810] Server: Based on this generated information, a dialogue scenario is formulated and a list of questions is created to guide the conversation with the user.

[0811] 6. Introducing the Emotion Engine:

[0812] Server: Activates the emotion engine and recognizes emotions from the user's voice and text input. This emotion information is used to adjust the dialogue scenario and generate feedback.

[0813] 7. The server sends the dialogue scenario to the device:

[0814] Server: Sends a dialogue scenario and a list of questions based on emotional information to the device, enabling flexible dialogue according to the user's emotions.

[0815] 8. The device presents the dialogue scenario to the user:

[0816] Terminal: The initial dialogue questions are displayed to the user, preparing them to start the dialogue. The questions are displayed taking into account emotional information.

[0817] User: Responds to questions displayed on the screen by typing text or speaking.

[0818] 9. User and system begin interaction:

[0819] Terminal: Records the user's responses and sends them to the server in real time.

[0820] Server: Analyzes the user's response and uses natural language processing technology and an emotion engine to generate the next question or feedback.

[0821] 10. The server analyzes the dialogue and provides feedback:

[0822] Server: Collects interaction logs and evaluates the user's emotional state and memory vividness, adjusting the interaction content and providing personalized feedback.

[0823] Server: Generates personalized feedback tailored to the user's memory impairment and provides it to the user via the device.

[0824] Specific examples

[0825] For example, consider a scenario in which a user selects a photo from a family trip from their childhood and starts a conversation based on the topic of "Memories of having fun with my family at Shonan Beach in the summer of 2005." The user uploads the photo, and the server analyzes it to generate anecdotes and questions like the following:

[0826] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[0827] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[0828] In addition, an emotion engine is used to recognize the user's emotions, and questions that are more in line with the user's emotions are added, such as, "You seem to be having fun. Tell me more about what happened that day."

[0829] This system allows users to enjoy past memories while stimulating and improving their memory. The introduction of an emotion engine makes the interaction more personalized, improving the user experience.

[0830] The processing flow will be explained below.

[0831] Step 1:

[0832] The user launches the application on the device, opens an interface within the application for selecting past photos, and selects a specific photo from the saved photos.

[0833] Step 2:

[0834] The device displays a preview of the selected photo to the user and provides an "Upload" button to confirm that the photo is correct.

[0835] Step 3:

[0836] The user clicks the "Upload" button, which completes preparations for uploading the photo data.

[0837] Step 4:

[0838] The device sends the selected photo data to the server using a business API, and the progress of the network connection is displayed to the user.

[0839] Step 5:

[0840] The server stores the received photo data in temporary storage, then launches the image analysis engine and begins analyzing the photo.

[0841] Step 6:

[0842] The server uses an image analysis engine to identify people and objects in the photo, for example, by using facial recognition technology to identify people, using libraries such as Dlib and FaceNet.

[0843] Step 7:

[0844] The server extracts the photo's metadata (e.g., EXIF ​​data) and obtains the date and time the photo was taken and its location, which identifies when and where the photo was taken.

[0845] Step 8:

[0846] The server references an internal database and obtains detailed information about the people in the photo (e.g., names, relationships, etc.) from the facial recognition results.

[0847] Step 9:

[0848] The server generates information about past memories based on the analysis results and metadata. For example, it creates a story such as "I went on a family trip to Shonan Beach in the summer of 2005."

[0849] Step 10:

[0850] The server creates a dialogue scenario based on the generated memory information, which includes a list of questions to prompt dialogue with the user.

[0851] Step 11:

[0852] The emotion engine is activated. The server extracts emotion data from the user's voice and text and adjusts the dialogue scenario based on this data.

[0853] Step 12:

[0854] The server uses the user's emotional data to generate more personalized questions and feedback. For example, if the user seems to be having fun, it will generate questions like, "What other fun things happened that day?"

[0855] Step 13:

[0856] The server sends a dialogue scenario and a list of questions based on emotion data to the terminal.

[0857] Step 14:

[0858] The terminal prepares to display the dialogue scenario received from the server on the user screen, presents the first question, and notifies the user that the dialogue is ready to begin.

[0859] Step 15:

[0860] The user responds to questions displayed on the screen by text input or voice input, and the device records the user's responses.

[0861] Step 16:

[0862] The device sends the user's responses in real time to the server, which then generates the next question or feedback based on the received data.

[0863] Step 17:

[0864] The server analyzes the user's responses and uses natural language processing techniques and an emotion engine to assess the user's memory state and emotional changes.

[0865] Step 18:

[0866] The server generates the next question and feedback and sends it to the device, which may include personalized questions and suggestions to help the user improve their memory.

[0867] Step 19:

[0868] The device receives new questions and feedback from the server and presents them to the user, allowing the user to continue interacting and stimulating their memory.

[0869] Step 20:

[0870] When the user indicates an intention to end the interactive session (for example, by clicking an "end" button), the terminal notifies the server of this information.

[0871] Step 21:

[0872] The server acknowledges the end of the interaction session and generates a summary and feedback of the interaction, including an assessment of the user's memory state and the progress of the interaction.

[0873] Step 22:

[0874] The device displays feedback from the server to the user, suggests next sessions, saves records, and allows the user to see the results of the interaction.

[0875] Example 2

[0876] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0877] In recent years, with the advancement of an aging society, the prevention and treatment of dementia have become important issues. Memory retention and emotional stability are particularly important issues in the early stages of dementia. Current methods offer limited means for users to stimulate past memories, and more effective methods are needed to recreate and evoke past memories. Furthermore, due to the lack of systems that can adequately grasp users' emotional states and provide feedback through dialogue, there is a need for methods to improve the quality of dialogue.

[0878] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving photo data from a user's terminal, means for analyzing the received photo data using an image analysis engine to identify people and objects in the photo, means for collecting photo metadata based on the analysis results and generating past memory information, means for creating a dialogue scenario based on the generated memory information, means including an emotion recognition engine for detecting the user's emotions and adjusting the dialogue scenario, means for providing flexible dialogue based on the generated dialogue scenario and emotion information, and means for analyzing data obtained from the dialogue and generating and providing individual feedback. This makes it possible to effectively recreate past memories and promote memory retention and emotional stability through dialogue that is in line with the user's emotions.

[0879] "User" means a person who uses the system.

[0880] A "terminal" refers to a device operated by a user, specifically a PC, smartphone, tablet, etc.

[0881] "Photo data" refers to image files imported by the user.

[0882] "Means for receiving" refers to the function for the server to receive data sent from the terminal.

[0883] An "image analysis engine" is software that processes and analyzes image data to extract useful information.

[0884] "Means for identifying people and objects in photographs" refers to the ability to use an image analysis engine to recognize and identify specific elements within a photograph.

[0885] "Metadata" refers to detailed information that accompanies photo data, including the date the photo was taken and location information.

[0886] "Memories" refers to episodes and stories related to past memories generated based on photos and metadata.

[0887] A "dialogue scenario" is a plan that defines the progress of a conversation with a user, constructed based on the generated memory information.

[0888] An "emotion recognition engine" is software that analyzes and recognizes emotions from a user's voice and text data.

[0889] "Means for providing flexible dialogue" refers to a function that adjusts dialogue scenarios based on emotional information and provides questions and feedback that are appropriate for the user.

[0890] "Feedback" refers to responses and advice provided by a system based on interactions with a user.

[0891] "Means for analyzing data" refers to the function of analyzing information obtained from the dialogue and evaluating the user's condition and response.

[0892] "Means for generating and providing individual feedback" refers to a function that creates feedback suited to each individual user based on the analysis results and provides that feedback via the terminal.

[0893] The present invention provides a system that allows users to use past photographs to stimulate their memories and alleviate early symptoms of dementia. This system functions through the interaction of three parties: the user, the terminal, and the server. Specific embodiments are described below.

[0894] System Overview

[0895] Hardware

[0896] Server: The central server that performs the main processing of this system.

[0897] Terminal: A device operated by a user, such as a PC, smartphone, or tablet.

[0898] software

[0899] Image analysis engine: Uses image processing libraries such as OpenCV and TensorFlow.

[0900] Emotion recognition engine: Emotion analysis uses technologies such as IBM Watson and Microsoft Azure Emotion API.

[0901] Database: A database for storing metadata and user information.

[0902] Natural language processing engine: Analyzes user input and generates dialogue scenarios.

[0903] System Operation

[0904] 1. Select and upload a photo

[0905] First, the user launches the application on the device. The device displays the user's photo library and allows the user to select memorable photos. For example, the user may select photos from a family trip or a special event. The selected photos are previewed on the device and then sent to the server when the user clicks the "Upload" button.

[0906] 2. Receiving and analyzing photos

[0907] The server receives the photo data sent from the device and temporarily stores it in storage. It then begins analyzing the photo using OpenCV and TensorFlow. The image analysis engine identifies people and objects in the photo and uses facial recognition technology to identify who is in the photo. This process utilizes the Dlib and FaceNet libraries.

[0908] 3. Collecting metadata and generating memories

[0909] The server analyzes the EXIF ​​data of the photo to obtain the date and time the photo was taken and location information. Based on this metadata, it determines when and where the photo was taken. It then combines the results of the image analysis with information from the database to obtain detailed information about the people in the photo (such as their names and relationships). Based on this information, the server then creates a document that describes memorable episodes related to the photo.

[0910] 4. Emotion Recognition and Dialogue Scenario Generation

[0911] Based on the generated memory information, the server formulates a dialogue scenario. It activates an emotion recognition engine to recognize emotions from the user's voice and text input. Based on this emotion information, the dialogue scenario is adjusted to provide more appropriate questions and feedback.

[0912] 5. Providing dialogue scenarios and feedback

[0913] The server sends a list of questions based on the generated dialogue scenario and emotional information to the device. The device displays this to the user, preparing to begin the dialogue. The user responds to the questions displayed on the screen by text input or voice input. The device sends these responses to the server in real time. The server analyzes the responses sent by the user, adjusts the dialogue content, and provides feedback appropriate to each individual.

[0914] Specific examples

[0915] For example, a specific example will be given in which the user selects a photo of "a family trip to Shonan Beach in the summer of 2005."

[0916] Prompt statement:

[0917] "I will upload photos from a family trip to Shonan Beach in the summer of 2005. Please generate a dialogue scenario in which the user talks about their memories of that time."

[0918] Generated episodes:

[0919] "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family enjoyed playing in the sea."

[0920] Generated question:

[0921] "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[0922] Additional input from the emotion engine:

[0923] "Sounds fun. Tell me more about that day."

[0924] In this way, the system stimulates memories while allowing users to enjoy past memories, contributing to the prevention and alleviation of dementia.The introduction of an emotion engine makes the dialogue more personalized, improving the user experience.

[0925] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0926] Step 1:

[0927] User selects and uploads a photo

[0928] How it works: A user launches an application on their device and selects a photo from their photo library, such as a photo from a trip or family event.

[0929] Input: A photo file selected by the user.

[0930] Output: The device displays a preview of the selected photo. An "Upload" button is provided.

[0931] Specific action: The user checks the preview and clicks the "Upload" button.

[0932] Step 2:

[0933] The device sends the photo data to the server.

[0934] Operation: The device uses the business API to send photo data to the server. The progress of the transfer is displayed to the user.

[0935] Input: The photo data selected by the user and uploaded.

[0936] Output: The photo data sent to the server.

[0937] Specific operation: The device sends data to the server via the network.

[0938] Step 3:

[0939] The server receives the photo and begins analyzing it.

[0940] How it works: The server receives the photo data and temporarily stores it in storage. OpenCV and TensorFlow are used for photo analysis.

[0941] Input: Photo data sent from the device.

[0942] Output: As a result of the analysis, specific information about people and objects in the photo.

[0943] Specific operation: The server preprocesses the image and detects human faces. Dlib and FaceNet are used as facial recognition technologies.

[0944] Step 4:

[0945] The server collects the photo metadata.

[0946] How it works: The server analyzes the EXIF ​​data of the photo to obtain the date and time the photo was taken and its location.

[0947] Input: Parsed photo data.

[0948] Output: Date and time of capture, location, and other metadata.

[0949] What it does: The server compares the retrieved metadata with an internal database to determine when and where the photo was taken.

[0950] Step 5:

[0951] The server generates the memory information

[0952] How it works: The server uses collected metadata and facial recognition results to create a document of memorable events related to the photo.

[0953] Input: Metadata, face recognition results.

[0954] Output: A written version of the memory episode.

[0955] Concrete action: For example, generate an episode such as "In the summer of 2005, we went on a family trip to Shonan Beach."

[0956] Step 6:

[0957] The server generates a dialogue scenario

[0958] How it works: The server creates a dialogue scenario with the user based on the generated memory information. It uses a generation engine to create a list of questions to guide the dialogue.

[0959] Input: A written memory episode.

[0960] Output: Dialogue scenario and questionnaire.

[0961] Concrete Actions: Formulate specific questions, such as, "What is your first memory of seeing this photo?"

[0962] Step 7:

[0963] Introducing an emotion recognition engine

[0964] How it works: The server launches an emotion recognition engine to recognize emotions from the user's voice and text input.

[0965] Input: User voice input, text input.

[0966] Output: Emotional information.

[0967] Specific actions: Emotional information is used to adjust the dialogue scenario.

[0968] Step 8:

[0969] The server sends the dialogue scenario to the terminal.

[0970] Operation: The server sends a dialogue scenario and a list of questions based on emotion information to the terminal.

[0971] Input: Dialogue scenario, emotion information.

[0972] Output: The dialogue scenario and questionnaire sent to the terminal.

[0973] Concrete behavior: This enables flexible interaction according to the user's emotions.

[0974] Step 9:

[0975] The device presents a dialogue scenario to the user.

[0976] Operation: The terminal displays the initial dialogue questions to the user and prepares to begin the dialogue.

[0977] Input: Dialogue scenario and question list sent from the server.

[0978] Output: The dialogue questions that are displayed to the user.

[0979] Specific actions: The user responds to questions displayed on the screen by text or voice.

[0980] Step 10:

[0981] The user and the system start the conversation

[0982] How it works: The user responds to questions by typing or speaking.

[0983] Input: The user's response.

[0984] Output: User responses logged to the terminal.

[0985] Specific operation: The terminal sends the response to the server in real time.

[0986] Step 11:

[0987] The server analyzes the conversation and provides feedback

[0988] How it works: The server analyzes the user's responses and adjusts the interaction to provide personalized feedback.

[0989] Input: The user's response sent from the terminal.

[0990] Output: Feedback to the user with next questions or advice.

[0991] Specific operation: The server uses natural language processing technology to analyze the user's response and generate the next question or feedback.

[0992] (Application example 2)

[0993] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0994] Conventional memory stimulation systems do not adequately consider the user's emotions, resulting in a lack of personalized interaction experiences and the inability to effectively stimulate memory or improve skills. Furthermore, it is difficult to evaluate and provide feedback on the user's skills and knowledge in real time.

[0995] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving photo data from a user's terminal, means for analyzing the received photo data using an image analysis engine and identifying objects in the photo, means for collecting photo attribute information and generating past event information based on the analysis results, means for creating a dialogue scenario based on the generated event information, means for recognizing emotions from the user's voice or text input and adjusting the dialogue scenario, means for conducting a dialogue with the user based on the dialogue scenario, and means for evaluating the user's skill and knowledge status and providing feedback. This enables dialogue optimized for individual users and real-time skill evaluation and feedback.

[0996] "User's device" refers to an electronic device used by a user, such as a smartphone, tablet, PC, or wearable device.

[0997] "Photo data" refers to still image data that a user has taken, saved, or selected through a terminal.

[0998] An "image analysis engine" is a software module that analyzes received image data and identifies people and objects within the image.

[0999] "Subjects" are elements that are to be recognized, such as people, animals, objects, and scenery, contained in an image.

[1000] "Attribute information" refers to metadata and context information related to image data, and includes, for example, the date and time of shooting, location information, shooting environment, and subject characteristics.

[1001] "Past event information" is an episode or story related to a specific time or place in the past, generated based on attribute information.

[1002] A "dialogue scenario" is a script of a series of questions and answers for dialogue with a user, constructed based on generated past event information.

[1003] "Emotion" recognition refers to analyzing voice or text input to identify a user's emotional state (e.g., joy, sadness, anger, etc.).

[1004] "Skill and knowledge status" refers to the level of skill and knowledge that a user possesses regarding a specific operation or task.

[1005] The "means for providing feedback" is a software module for providing appropriate advice and evaluation in real time based on the user's answers and actions.

[1006] This invention is a system that aims to alleviate the early symptoms of dementia by stimulating the user's memory using past photographs. This system functions through the interaction of three parties: the server, the terminal, and the user. Specifically, it has the following features:

[1007] 1. Receiving and analyzing photo data

[1008] The device has a function that allows users to select and upload past photos. It is assumed that users will launch an application on the device and select photos from a family trip or a special event. The selected photos are sent to the server by pressing the "upload" button on the device.

[1009] The server temporarily stores the received photo data in storage and launches an image analysis engine (such as OpenCV or TensorFlow). Object recognition technology (such as YOLOv3 or OpenCV) is used to identify the subject in the photo. It also collects photo attribute information (such as EXIF ​​data) and obtains the date and time the photo was taken and its location. Based on the results of this analysis, past event information is generated.

[1010] 2. Dialogue scenario generation and emotion recognition

[1011] The server creates a dialogue scenario based on the generated past event information. It then uses an emotion engine that analyzes voice and text input to recognize the user's emotions and adjust the dialogue scenario accordingly. During this process, questions and feedback based on the emotion information are added.

[1012] 3. User interaction and feedback

[1013] The device presents the dialogue scenario sent from the server to the user and initiates the dialogue. The user responds to questions displayed on the screen by text input or voice input. The user's responses are sent in real time to the server, which then uses natural language processing technology and an emotion engine to generate the next question and feedback.

[1014] 4. Skill evaluation and feedback

[1015] The server collects dialogue logs and evaluates the user's skills and knowledge. Based on this, individual feedback is generated and provided to the user via their device. This feedback evaluates the user's skill and knowledge level for specific operations and tasks, and provides appropriate advice and evaluation in real time.

[1016] Specific examples

[1017] For example, consider a scenario in which a user selects a photo from a family trip from their childhood and starts a conversation based on the topic of "Memories of having fun with my family at Shonan Beach in the summer of 2005." The user uploads the photo, and the server analyzes it to generate anecdotes and questions like the following:

[1018] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[1019] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[1020] In addition, an emotion engine is used to recognize the user's emotions, and questions that are more in line with their emotions are added, such as, "You seem to be having fun. Tell me more about what happened that day."

[1021] Prompt Sentence Examples

[1022] "Please upload photos of your market so we can train you on your business."

[1023] "What is in the uploaded photo? For example, what are the names of machine parts or devices?"

[1024] This system allows users to enjoy past memories while stimulating and improving their memory. The introduction of an emotion engine makes interactions more personalized, improving the user experience.

[1025] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1026] Step 1:

[1027] The device allows the user to select past photos and upload them through the application (input: photo data selected by the user, output: photo data sent to the server). The device displays the selected photos as a preview and provides an upload button. When the user clicks the upload button, the photo data is sent to the server.

[1028] Step 2:

[1029] The server temporarily stores the received photo data in storage and launches an image analysis engine (such as OpenCV or TensorFlow) (input: photo data, output: photo data stored in storage). This engine uses object recognition technology (such as YOLOv3 or OpenCV) to identify objects in the photo.

[1030] Step 3:

[1031] The server analyzes the attribute information of the photo (e.g., EXIF ​​data) and collects the attribute information (input: photo data saved in storage, output: attribute information). This allows the server to identify when and where the photo was taken.

[1032] Step 4:

[1033] The server generates past event information based on the analyzed attribute information (input: attribute information, output: past event information). For example, it generates a specific episode such as "In the summer of 2005, we went on a family trip to Shonan Beach."

[1034] Step 5:

[1035] The server creates a dialogue scenario based on the generated past event information (input: past event information, output: dialogue scenario). Furthermore, it uses an emotion engine that analyzes voice and text input to recognize the user's emotions and adjust the dialogue scenario accordingly.

[1036] Step 6:

[1037] The server creates questions and feedback based on the generated dialogue scenario and emotional information and sends them to the terminal (input: dialogue scenario, emotional information; output: questions and feedback sent to the terminal).

[1038] Step 7:

[1039] The terminal presents the dialogue scenario sent from the server to the user and starts the dialogue (input: questions and feedback sent from the server, output: user's response). The user responds to the questions displayed on the screen by text input or voice input.

[1040] Step 8:

[1041] The terminal records the user's response and transmits it to the server in real time (input: user's response data, output: real-time response data transmitted to the server).

[1042] Step 9:

[1043] The server analyzes the user's response data and generates the next question and feedback using natural language processing technology and an emotion engine (input: user's real-time response data, output: next question and feedback).

[1044] Step 10:

[1045] The server collects dialogue logs and evaluates the user's skills and knowledge (input: dialogue log data, output: user skill evaluation data). Based on this, individual feedback is generated and provided to the user via their device.

[1046] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1047] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1048] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1049] [Third embodiment]

[1050] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1051] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1052] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1053] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1054] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1055] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1056] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1057] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1058] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1059] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1060] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1061] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1062] The present invention provides a system for stimulating a user's memory by using past photographs and alleviating early symptoms of dementia. Specific embodiments of the system will be described below.

[1063] System Overview

[1064] This system works by interacting with the server, the device, and the user. The user uploads past photos from their device to the server, and the server analyzes the photos to generate a dialogue scenario, which is then presented to the user via their device. The user can then have a conversation based on the scenario, allowing them to enjoyably reminisce about their memories.

[1065] System processing flow

[1066] 1. User selects and uploads a photo:

[1067] User: First, open the application on the device and select some old photos, such as photos from a family trip or a special event.

[1068] Terminal: It then displays a preview of the selected photo to the user and provides an "Upload" button.

[1069] User: Check the preview and click the "Upload" button.

[1070] 2. The device sends the photo data to the server:

[1071] Terminal: Uses a business API to send the selected photo data to the server. At this time, the network connection status and progress are displayed to the user.

[1072] 3. The server receives the photo and begins analyzing it:

[1073] Server: Temporarily stores the received photo data in storage and launches an image analysis engine, using technologies such as OpenCV and TensorFlow.

[1074] Server: Identifies people and objects in the photo and uses facial recognition technology to identify people in the shot, leveraging libraries like Dlib and FaceNet.

[1075] 4. The server collects the metadata:

[1076] Server: Extracts photo metadata (such as the date and time of the photo, location information, etc.) from the EXIF ​​data and obtains information about identified people from an internal database. For example, it may collect information such as that the photo was taken at Shonan Beach or that a specific person is a family member.

[1077] 5. The server generates the relevant memories:

[1078] Server: Based on the collected metadata, the server creates text about memorable episodes related to the photo. For example, it generates information such as "Memories of a family trip to Shonan Beach in the summer of 2005."

[1079] Server: Based on this generated information, a dialogue scenario is formulated and a list of questions is created to guide the conversation with the user.

[1080] 6. The server sends the dialogue scenario to the device:

[1081] Server: Sends the dialogue scenario and the generated question list to the terminal.

[1082] Terminal: Prepares to display the dialogue scenario on the user's screen based on the information received from the server.

[1083] 7. The device presents the dialogue scenario to the user:

[1084] Terminal: Displays the initial dialogue questions to the user and notifies them that they are ready to begin the dialogue.

[1085] User: Responds to questions displayed on the device by text input or voice input.

[1086] 8. User and system begin interaction:

[1087] Terminal: Records the user's responses and continuously sends them to the server.

[1088] Server: Analyzes the user's response, generates and selects the next question or feedback, and sends it to the device. Through this cycle, the user can continue the dialogue.

[1089] 9. The server analyzes the dialogue and provides feedback:

[1090] Server: Collects dialogue logs and analyzes user responses using natural language processing technology. Evaluates the user's memory status and emotional changes, and uses this information for the next dialogue.

[1091] Server: Creates individual feedback and follow-up questions based on the user's memory impairment status and provides them to the user via the device.

[1092] Specific examples

[1093] For example, consider a scenario in which a user selects a photo from a family trip from their childhood and starts a conversation based on the topic of "Memories of having fun with my family at Shonan Beach in the summer of 2005." The user uploads the photo, and the server analyzes it to generate anecdotes and questions like the following:

[1094] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[1095] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[1096] As the user answers these questions, the system elicits more detailed information and stimulates the user's memory.

[1097] This system allows users to enjoy memories of the past while stimulating and improving their memory.

[1098] The processing flow will be explained below.

[1099] Step 1:

[1100] The user launches the application on the device, opens an interface within the application for selecting past photos, and selects a specific photo from the saved photos.

[1101] Step 2:

[1102] The device displays a preview of the selected photo to the user and provides an "Upload" button to confirm that the photo is correct.

[1103] Step 3:

[1104] The user clicks the "Upload" button, which completes preparations for uploading the photo data.

[1105] Step 4:

[1106] The device sends the selected photo data to the server using a business API, and the progress of the network connection is displayed to the user.

[1107] Step 5:

[1108] The server stores the received photo data in temporary storage, then launches the image analysis engine and begins analyzing the photo.

[1109] Step 6:

[1110] The server uses an image analysis engine to identify people and objects in the photo, for example, by using facial recognition technology to identify people, using libraries such as Dlib and FaceNet.

[1111] Step 7:

[1112] The server extracts the photo's metadata (e.g., EXIF ​​data) and obtains the date and time the photo was taken and its location, which identifies when and where the photo was taken.

[1113] Step 8:

[1114] The server references an internal database and obtains detailed information about the people in the photo (e.g., names, relationships, etc.) from the facial recognition results.

[1115] Step 9:

[1116] The server generates information about past memories based on the analysis results and metadata. For example, it creates a story such as "I went on a family trip to Shonan Beach in the summer of 2005."

[1117] Step 10:

[1118] The server creates a dialogue scenario based on the generated memory information, which includes a list of questions to prompt dialogue with the user.

[1119] Step 11:

[1120] The server sends the created dialogue scenario to the terminal. The scenario includes episodes and related questions.

[1121] Step 12:

[1122] The terminal prepares to display the dialogue scenario received from the server on the user screen, presents the first question, and notifies the user that the dialogue is ready to begin.

[1123] Step 13:

[1124] The user responds to questions displayed on the screen by text input or voice input, and the device records the user's responses.

[1125] Step 14:

[1126] The device sends the user's responses in real time to the server, which then generates the next question or feedback based on the received data.

[1127] Step 15:

[1128] The server analyzes the user's responses and uses natural language processing techniques (e.g., sentiment analysis) to assess the user's memory state and emotional changes.

[1129] Step 16:

[1130] The server generates the next question and feedback and sends it to the device, which may include personalized questions and suggestions to help the user improve their memory.

[1131] Step 17:

[1132] The device receives new questions and feedback from the server and presents them to the user, allowing the user to continue interacting and stimulating their memory.

[1133] Step 18:

[1134] When the user indicates an intention to end the interactive session (for example, by clicking an "end" button), the terminal notifies the server of this information.

[1135] Step 19:

[1136] The server acknowledges the end of the interaction session and generates a summary and feedback of the interaction, including an assessment of the user's memory state and the progress of the interaction.

[1137] Step 20:

[1138] The device displays feedback from the server to the user, suggests next sessions, saves records, and allows the user to see the results of the interaction.

[1139] Example 1

[1140] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1141] Users in the early stages of dementia have difficulty maintaining and recalling memories in their daily lives, which increases their mental burden. Current systems have limited methods for effectively utilizing users' past photos to stimulate memories, and the lack of an interactive approach reduces opportunities for users to actively reflect on the past. Furthermore, the system lacks mechanisms for monitoring the progress of memory impairment and providing appropriate feedback.

[1142] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1143] In this invention, the server includes a means for receiving image data from the user's information processing device, a means for analyzing the received image data using an image analysis device and identifying objects in the image, a means for collecting additional information about the image based on the analysis results and generating past memory information, a means for creating a dialogue plan based on the generated memory information, a means for engaging in a dialogue with the user based on the dialogue plan, a means for recording the user's responses during the dialogue and transmitting the data, a means for generating dialogue questions and feedback using a generative AI model, and a means for collecting dialogue logs and analyzing them using natural language processing technology. This allows the user to use past photos to stimulate their memories and efficiently recall them. Furthermore, by monitoring the user's memory status through dialogue and providing appropriate feedback, it is possible to maintain and improve cognitive function.

[1144] "User" refers to the person who uses this system, and is particularly targeted at people in the early stages of dementia.

[1145] An "information processing device" is a terminal or device operated by a user, and allows operations such as selecting and uploading photos.

[1146] "Image data" refers to digital images such as photographs and drawings uploaded by users.

[1147] An "image analysis device" is a device that analyzes image data and identifies subjects (people or objects) within the image, and includes an image analysis engine and library.

[1148] "Additional information" refers to metadata such as the image's EXIF ​​data, as well as information related to the subject matter within the image.

[1149] "Memories" refers to past events and episodes that are generated based on the results of image analysis and additional information.

[1150] "Dialogue planning" refers to a scenario or question list that guides a dialogue with the user based on memory information.

[1151] "Generative AI model" refers to an artificial intelligence model used to automatically generate dialogue questions and feedback.

[1152] "Natural language processing technology" refers to technology for analyzing dialogue logs and understanding user responses, and includes technology used for sentence generation and sentiment analysis.

[1153] The present invention provides a system for stimulating past memories in a user and alleviating early symptoms of dementia. Specific processing and embodiments of the program of this system will be described in detail below.

[1154] This system functions through the interaction of three parties: a server, a terminal, and a user. The server is responsible for a wide range of processes, including receiving and analyzing image data, extracting metadata, and generating a dialogue plan. The terminal provides an interface for users to upload photos and initiate dialogue. Users use this interface to select and upload photos and engage in dialogue according to the dialogue scenario provided by the server.

[1155] Server processing

[1156] The server first receives image data from the user. To do this, the server receives image data sent from the device using a RESTful API. The received image data is then temporarily stored in cloud storage. The main technology used is likely to be a cloud service such as Amazon S3.

[1157] The stored image data is analyzed using an image analyzer (e.g., OpenCV or TensorFlow) to identify people and objects in the image and use facial recognition technology (e.g., Dlib or FaceNet). Based on the analysis results, the EXIF ​​data of the photo is extracted and metadata such as the date and time of the photo's capture and location are collected using a library such as ExifTool.

[1158] The collected metadata and analysis results are integrated to generate memory information. Natural language generation technology (generative AI model) is used to create sentences about specific episodes, such as "Memories of a family trip to Shonan Beach in the summer of 2005." Based on the generated memory information, a dialogue plan is also created, and a list of questions is created to guide the conversation with the user.

[1159] The generated dialogue plan and question list are sent to the terminal via encrypted communication.

[1160] About terminal processing

[1161] The device accepts operations from the user and sends them to the server. The user opens the dedicated application, selects and uploads photos. During uploading, a progress bar is displayed to notify the user of the progress. The device displays the dialogue scenario received from the server, allowing the user to start a dialogue. During the dialogue, the device records the user's responses and continuously sends them to the server.

[1162] About User Roles

[1163] The user selects and uploads past photos using the device. Based on the dialogue scenario provided by the server, the user responds by text input or voice input. Through this process, the user's past memories are stimulated, and it is expected that their memory will be improved.

[1164] Specific examples

[1165] For example, if a user uploads a photo of a family trip to Shonan Beach in the summer of 2005, the server processes it as follows: First, it extracts "Date and time of photo: Summer 2005, Location: Shonan Beach" from the photo's EXIF ​​data. Next, it uses facial recognition technology to identify family members and generates anecdotes and questions like the following:

[1166] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[1167] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[1168] As the user answers these questions, the system elicits more detailed information and stimulates the user's memory.

[1169] Prompt Sentence Examples

[1170] "Generate a conversation scenario based on memories from photos of a family trip to Shonan Beach."

[1171] "Create a story related to the person in this photo."

[1172] As described above, this system aims to effectively stimulate the user's memory by linking various hardware and software.

[1173] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1174] Step 1: Select and upload a photo

[1175] Subject: User

[1176] Input: Photo data selected on your device

[1177] Output: Uploaded photo data

[1178] The user opens a dedicated application on their device and selects a past photo. The selected photo is previewed, and when the user clicks the "Upload" button, the photo data is sent to the server.

[1179] Step 2: Receiving and saving photo data

[1180] Subject: Server

[1181] Input: Photo data uploaded by the user

[1182] Output: Saved photo data (cloud storage)

[1183] The server receives the photo data sent from the device and temporarily stores it in cloud storage, using a cloud service such as Amazon S3.

[1184] Step 3: Analyze the photo data

[1185] Subject: Server

[1186] Input: Saved photo data

[1187] Output: Analysis results (specific information about people and objects)

[1188] The server analyzes the stored photo data using image analysis equipment (e.g., OpenCV or TensorFlow). The analysis identifies people and objects in the photos. The analysis results also include information on people identified using facial recognition technology (Dlib or FaceNet).

[1189] Step 4: Collecting Metadata

[1190] Subject: Server

[1191] Input: Analysis results and photo EXIF ​​data

[1192] Output: Collected metadata (date and time of photo, location, etc.)

[1193] The server extracts the EXIF ​​data from the photos using tools such as ExifTool, and collects metadata such as the date and time the photo was taken and the location information. This is then combined with the analysis results to collect information on identified individuals.

[1194] Step 5: Generate memories

[1195] Subject: Server

[1196] Input: Collected metadata and analysis results

[1197] Output: Generated memories

[1198] The server uses natural language generation technology (generative AI model) to generate memory information based on the collected metadata and analysis results. For example, a detailed episode such as "Memories of a family trip to Shonan Beach in the summer of 2005" can be written.

[1199] Step 6: Developing dialogue scenarios

[1200] Subject: Server

[1201] Input: Generated memory information

[1202] Output: Dialogue scenario and questionnaire

[1203] The server formulates a dialogue plan based on the generated memory information and creates a specific list of questions for the user, such as "What is your first memory of seeing this photo?"

[1204] Step 7: Sending the dialogue scenario to the device

[1205] Subject: Server

[1206] Input: Dialogue scenario and questionnaire

[1207] Output: Dialogue scenario and question list sent to the terminal

[1208] The server sends the dialogue scenario and question list to the terminal through encrypted communication.

[1209] Step 8: Present the dialogue scenario

[1210] Subject: Device

[1211] Input: Dialogue scenario and question list received from the server

[1212] Output: The dialogue scenario and question list presented to the user

[1213] The terminal presents the dialogue scenario and question list received from the server to the user, and displays the initial dialogue questions in a pop-up format on the screen to prompt the user to start the dialogue.

[1214] Step 9: User responses and recording

[1215] Subject: User and Device

[1216] Input: User input (text or voice)

[1217] Output: Recorded user responses

[1218] The user responds to questions displayed on the device by text input or voice input, and the device records the user's responses and continuously transmits them to the server.

[1219] Step 10: Continued dialogue and feedback

[1220] Subject: Server

[1221] Input: User response data

[1222] Output: Next dialogue question or feedback

[1223] The server analyzes the user's response data and uses a generative AI model to generate the next question or feedback, allowing the user to continue the dialogue.

[1224] These are the specific processing steps of the system, which aims to effectively stimulate the user's memory and alleviate the early symptoms of dementia.

[1225] (Application example 1)

[1226] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1227] Today, there is a need for effective methods to stimulate memories and alleviate the condition of people in the early stages of dementia. However, existing systems are limited to reviewing old photos and lack the ability to delve deeper into memories or re-experience memories in a more immersive way. Furthermore, they lack the precision to elicit memories through dialogue and the ability to personalize the user experience. Therefore, new technologies are needed to enable users to re-experience past memories in a more realistic way and stimulate and improve memory.

[1228] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1229] In this invention, the server includes means for receiving photo data from a user's device, means for analyzing the received photo data using an image analysis engine and identifying people and objects in the photo, means for collecting photo metadata and generating past memory information based on the analysis results, means for creating a dialogue scenario based on the generated memory information, means for presenting the dialogue scenario to the user and displaying it on a head-mounted display so that the user can re-experience the experience in a virtual reality space, and means for analyzing the user's response using a natural language processing engine and generating the next question and feedback. This allows the user to re-experience memories in a realistic virtual reality space, making it possible to effectively stimulate and improve memory in the early stages of dementia.

[1230] A "user" is an individual who uses the system to re-experience past memories and stimulate their memories.

[1231] "Device" means an electronic device used by a user to upload photo data and receive and display dialogue scenarios.

[1232] "Photo data" refers to image files that have been taken in the past and are uploaded by users.

[1233] The "image analysis engine" is a software component that analyzes received photo data and identifies people and objects in the photo.

[1234] "Metadata" refers to accompanying information included in photo data, such as the date and time of shooting, location information, and information about the shooting device.

[1235] "Memories information" is information about past events generated based on the results of image analysis and photo metadata.

[1236] A "dialogue scenario" is a scenario that includes questions and topics to promote dialogue with the user based on the generated memory information.

[1237] "Virtual reality" is a virtual space generated using computer graphics, in which users can experience a realistic virtual environment.

[1238] A "head-mounted display" is a display device worn by the user on the head to allow the user to visually experience a virtual reality space.

[1239] A "natural language processing engine" is a software component that analyzes a user's dialogue responses and generates the next question or feedback.

[1240] System Overview

[1241] This invention is a system that allows users to use past photographs to stimulate their memories and alleviate the early symptoms of dementia. This system functions through the interaction of a server, a terminal, and a user.

[1242] How to run the program

[1243] 1. User Conduct

[1244] Users use their devices to select past photos, such as photos from a family trip or special event, and upload them to the server.

[1245] 2. Device Operation

[1246] The terminal displays a preview of the selected photo to the user and provides an "Upload" button, which causes the terminal to send the photo data to the server.

[1247] 3. Server Processing

[1248] The server analyzes the received photo data using an image analysis engine (e.g., OpenCV or TensorFlow). It performs facial and object recognition to identify people and objects in the photo. It also recognizes identified people using libraries such as Dlib and FaceNet. It also extracts photo metadata (such as the date and time of the photo shoot and location information) from the EXIF ​​data and obtains information about the identified people from an internal database.

[1249] Next, the server generates a memorable episode related to the photo based on the collected metadata. For example, it writes down information such as "Memories of a family trip to Shonan Beach in the summer of 2005." It then formulates a dialogue scenario based on the generated information and creates a list of questions to encourage conversation with the user.

[1250] 4. Use of head-mounted displays

[1251] The dialogue scenario and question list are sent from the server to the device and head-mounted display, which generates a virtual reality space for the user to re-experience, and presents the dialogue scenario within it.

[1252] 5. Dialogue with users

[1253] The user wears a head-mounted display and responds according to the displayed dialogue scenario. The user's responses are analyzed by a natural language processing engine (e.g., spaCy), which generates the next question or feedback. This allows the dialogue with the user to continue and elicit more memories.

[1254] Specific examples

[1255] For example, the following prompts can be input to a generative AI model:

[1256] "Generate a dialogue scenario and a list of questions for memory stimulation based on this photo taken at Shonan Beach in the summer of 2005."

[1257] The generated dialogue scenario example is as follows:

[1258] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[1259] question:

[1260] "What's your first memory of seeing this photo?"

[1261] Who did you go to the beach with that day?

[1262] This system allows users to re-experience past memories in an immersive virtual reality space, stimulating and improving memory.

[1263] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1264] Step 1:

[1265] Input: A previous photo selected by the user on their device

[1266] How it works: A user opens an application on their device and selects a photo from their past, such as a family vacation or a special event. The device previews the selected photo, and the user clicks the "Upload" button.

[1267] Output: Photo data ready to upload

[1268] Step 2:

[1269] Input: The "Upload" button the user clicked

[1270] Operation: The device sends the photo data to the server, and uses a business API to display the network connection status and progress.

[1271] Output: Photo data sent to the server

[1272] Step 3:

[1273] Input: Photo data received by the server

[1274] How it works: The server temporarily stores the received photo data and begins analyzing it using an image analysis engine (OpenCV or TensorFlow). Through this analysis, it identifies people and objects in the photo and uses facial recognition technology (Dlib, FaceNet, etc.) to identify people in the photo.

[1275] Output: Analysis results (information on identified people and objects)

[1276] Step 4:

[1277] Input: Analysis results (information on identified people and objects)

[1278] How it works: The server extracts the photo's metadata (date and time of capture, location, etc.) from the EXIF ​​data and retrieves the identified person's information from an internal database.

[1279] Output: Extracted metadata and identified people

[1280] Step 5:

[1281] Input: Extracted metadata and identified people

[1282] How it works: The server generates a memorable episode related to the photo based on the collected metadata. For example, it generates information such as "Memories of a family trip to Shonan Beach in the summer of 2005." It also uses this information to formulate a dialogue scenario and create a list of questions to guide the conversation with the user.

[1283] Output: Generated memory episodes and dialogue scenarios

[1284] Step 6:

[1285] Input: Generated memory episodes and dialogue scenarios

[1286] Operation: The server sends the dialogue scenario and the generated question list to the device and the head-mounted display. Based on the information received from the server, the device prepares to display the dialogue scenario on the user's screen.

[1287] Output: Dialogue scenario and question list sent to the device and head-mounted display

[1288] Step 7:

[1289] Input: Terminal and head-mounted display that received the dialogue scenario and question list

[1290] How it works: The head-mounted display displays the dialogue scenario in a virtual reality space, allowing the user to begin the immersive experience. The device also displays the initial dialogue questions to the user, notifying them that they are ready to begin the dialogue.

[1291] Output: Display of the interaction scenario in a virtual reality space

[1292] Step 8:

[1293] Input: User response (text input or voice input)

[1294] How it works: The device records the user's responses and continuously sends them to the server.

[1295] Output: The user's response sent to the server

[1296] Step 9:

[1297] Input: The user's response sent to the server

[1298] How it works: The server analyzes the user's response using a natural language processing engine (e.g., spaCy) and generates the next question or feedback. The dialogue with the user continues through this cycle.

[1299] Output: Next question or feedback

[1300] Step 10:

[1301] Input: Next question or feedback sent from the server

[1302] Action: The head-mounted display and device present the next question or feedback to the user, and the user continues the interaction.

[1303] Output: The next question or feedback presented to the user

[1304] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1305] The present invention is a system that allows users to use past photos to stimulate their memories and alleviate the early symptoms of dementia. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system aims to optimize the interaction experience and improve the quality of feedback. A specific embodiment of the system is described below.

[1306] System Overview

[1307] This system achieves its functions through the interaction of three parties: the server, the device, and the user. Users upload past photos from their device, and the server analyzes the photos to generate a dialogue scenario. In doing so, it uses an emotion engine to recognize the user's emotions and provide more appropriate questions and feedback.

[1308] System processing flow

[1309] 1. User selects and uploads a photo:

[1310] User: First, launch the application on the device and select some past photos. For example, you might select photos from a family trip or a special event.

[1311] On the device: Shows the user a preview of the selected photo and provides an "Upload" button.

[1312] User: Check the preview and click the "Upload" button.

[1313] 2. The device sends the photo data to the server:

[1314] On the device: Uses a business API to send the selected photo data to the server. Displays the progress of the network connection to the user.

[1315] 3. The server receives the photo and begins analyzing it:

[1316] Server: Temporarily stores the received photo data in storage and launches an image analysis engine, using technologies such as OpenCV and TensorFlow.

[1317] Server: Identifies people and objects in the photo and uses facial recognition technology to identify people in the shot, leveraging libraries like Dlib and FaceNet.

[1318] 4. The server collects the metadata:

[1319] Server: Extracts photo metadata (e.g., EXIF ​​data) and captures the date and time of the photo and its location, which allows us to determine when and where the photo was taken.

[1320] Server: Refers to an internal database and obtains detailed information about the people in the photo (e.g., names, relationships, etc.) from the facial recognition results.

[1321] 5. The server generates the relevant memories:

[1322] Server: Based on the collected metadata, the server creates a story about a memorable episode related to the photo. For example, it creates a story like, "In the summer of 2005, we went on a family trip to Shonan Beach."

[1323] Server: Based on this generated information, a dialogue scenario is formulated and a list of questions is created to guide the conversation with the user.

[1324] 6. Introducing the Emotion Engine:

[1325] Server: Activates the emotion engine and recognizes emotions from the user's voice and text input. This emotion information is used to adjust the dialogue scenario and generate feedback.

[1326] 7. The server sends the dialogue scenario to the device:

[1327] Server: Sends a dialogue scenario and a list of questions based on emotional information to the device, enabling flexible dialogue according to the user's emotions.

[1328] 8. The device presents the dialogue scenario to the user:

[1329] Terminal: The initial dialogue questions are displayed to the user, preparing them to start the dialogue. The questions are displayed taking into account emotional information.

[1330] User: Responds to questions displayed on the screen by typing text or speaking.

[1331] 9. User and system begin interaction:

[1332] Terminal: Records the user's responses and sends them to the server in real time.

[1333] Server: Analyzes the user's response and uses natural language processing technology and an emotion engine to generate the next question or feedback.

[1334] 10. The server analyzes the dialogue and provides feedback:

[1335] Server: Collects interaction logs and evaluates the user's emotional state and memory vividness, adjusting the interaction content and providing personalized feedback.

[1336] Server: Generates personalized feedback tailored to the user's memory impairment and provides it to the user via the device.

[1337] Specific examples

[1338] For example, consider a scenario in which a user selects a photo from a family trip from their childhood and starts a conversation based on the topic of "Memories of having fun with my family at Shonan Beach in the summer of 2005." The user uploads the photo, and the server analyzes it to generate anecdotes and questions like the following:

[1339] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[1340] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[1341] In addition, an emotion engine is used to recognize the user's emotions, and questions that are more in line with the user's emotions are added, such as, "You seem to be having fun. Tell me more about what happened that day."

[1342] This system allows users to enjoy past memories while stimulating and improving their memory. The introduction of an emotion engine makes the interaction more personalized, improving the user experience.

[1343] The processing flow will be explained below.

[1344] Step 1:

[1345] The user launches the application on the device, opens an interface within the application for selecting past photos, and selects a specific photo from the saved photos.

[1346] Step 2:

[1347] The device displays a preview of the selected photo to the user and provides an "Upload" button to confirm that the photo is correct.

[1348] Step 3:

[1349] The user clicks the "Upload" button, which completes preparations for uploading the photo data.

[1350] Step 4:

[1351] The device sends the selected photo data to the server using a business API, and the progress of the network connection is displayed to the user.

[1352] Step 5:

[1353] The server stores the received photo data in temporary storage, then launches the image analysis engine and begins analyzing the photo.

[1354] Step 6:

[1355] The server uses an image analysis engine to identify people and objects in the photo, for example, by using facial recognition technology to identify people, using libraries such as Dlib and FaceNet.

[1356] Step 7:

[1357] The server extracts the photo's metadata (e.g., EXIF ​​data) and obtains the date and time the photo was taken and its location, which identifies when and where the photo was taken.

[1358] Step 8:

[1359] The server references an internal database and obtains detailed information about the people in the photo (e.g., names, relationships, etc.) from the facial recognition results.

[1360] Step 9:

[1361] The server generates information about past memories based on the analysis results and metadata. For example, it creates a story such as "I went on a family trip to Shonan Beach in the summer of 2005."

[1362] Step 10:

[1363] The server creates a dialogue scenario based on the generated memory information, which includes a list of questions to prompt dialogue with the user.

[1364] Step 11:

[1365] The emotion engine is activated. The server extracts emotion data from the user's voice and text and adjusts the dialogue scenario based on this data.

[1366] Step 12:

[1367] The server uses the user's emotional data to generate more personalized questions and feedback. For example, if the user seems to be having fun, it will generate questions like, "What other fun things happened that day?"

[1368] Step 13:

[1369] The server sends a dialogue scenario and a list of questions based on emotion data to the terminal.

[1370] Step 14:

[1371] The terminal prepares to display the dialogue scenario received from the server on the user screen, presents the first question, and notifies the user that the dialogue is ready to begin.

[1372] Step 15:

[1373] The user responds to questions displayed on the screen by text input or voice input, and the device records the user's responses.

[1374] Step 16:

[1375] The device sends the user's responses in real time to the server, which then generates the next question or feedback based on the received data.

[1376] Step 17:

[1377] The server analyzes the user's responses and uses natural language processing techniques and an emotion engine to assess the user's memory state and emotional changes.

[1378] Step 18:

[1379] The server generates the next question and feedback and sends it to the device, which may include personalized questions and suggestions to help the user improve their memory.

[1380] Step 19:

[1381] The device receives new questions and feedback from the server and presents them to the user, allowing the user to continue interacting and stimulating their memory.

[1382] Step 20:

[1383] When the user indicates an intention to end the interactive session (for example, by clicking an "end" button), the terminal notifies the server of this information.

[1384] Step 21:

[1385] The server acknowledges the end of the interaction session and generates a summary and feedback of the interaction, including an assessment of the user's memory state and the progress of the interaction.

[1386] Step 22:

[1387] The device displays feedback from the server to the user, suggests next sessions, saves records, and allows the user to see the results of the interaction.

[1388] Example 2

[1389] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1390] In recent years, with the advancement of an aging society, the prevention and treatment of dementia have become important issues. Memory retention and emotional stability are particularly important issues in the early stages of dementia. Current methods offer limited means for users to stimulate past memories, and more effective methods are needed to recreate and evoke past memories. Furthermore, due to the lack of systems that can adequately grasp users' emotional states and provide feedback through dialogue, there is a need for methods to improve the quality of dialogue.

[1391] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving photo data from a user's terminal, means for analyzing the received photo data using an image analysis engine to identify people and objects in the photo, means for collecting photo metadata based on the analysis results and generating past memory information, means for creating a dialogue scenario based on the generated memory information, means including an emotion recognition engine for detecting the user's emotions and adjusting the dialogue scenario, means for providing flexible dialogue based on the generated dialogue scenario and emotion information, and means for analyzing data obtained from the dialogue and generating and providing individual feedback. This makes it possible to effectively recreate past memories and promote memory retention and emotional stability through dialogue that is in line with the user's emotions.

[1392] "User" means a person who uses the system.

[1393] A "terminal" refers to a device operated by a user, specifically a PC, smartphone, tablet, etc.

[1394] "Photo data" refers to image files imported by the user.

[1395] "Means for receiving" refers to the function for the server to receive data sent from the terminal.

[1396] An "image analysis engine" is software that processes and analyzes image data to extract useful information.

[1397] "Means for identifying people and objects in photographs" refers to the ability to use an image analysis engine to recognize and identify specific elements within a photograph.

[1398] "Metadata" refers to detailed information that accompanies photo data, including the date the photo was taken and location information.

[1399] "Memories" refers to episodes and stories related to past memories generated based on photos and metadata.

[1400] A "dialogue scenario" is a plan that defines the progress of a conversation with a user, constructed based on the generated memory information.

[1401] An "emotion recognition engine" is software that analyzes and recognizes emotions from a user's voice and text data.

[1402] "Means for providing flexible dialogue" refers to a function that adjusts dialogue scenarios based on emotional information and provides questions and feedback that are appropriate for the user.

[1403] "Feedback" refers to responses and advice provided by a system based on interactions with a user.

[1404] "Means for analyzing data" refers to the function of analyzing information obtained from the dialogue and evaluating the user's condition and response.

[1405] "Means for generating and providing individual feedback" refers to a function that creates feedback suited to each individual user based on the analysis results and provides that feedback via the terminal.

[1406] The present invention provides a system that allows users to use past photographs to stimulate their memories and alleviate early symptoms of dementia. This system functions through the interaction of three parties: the user, the terminal, and the server. Specific embodiments are described below.

[1407] System Overview

[1408] Hardware

[1409] Server: The central server that performs the main processing of this system.

[1410] Terminal: A device operated by a user, such as a PC, smartphone, or tablet.

[1411] software

[1412] Image analysis engine: Uses image processing libraries such as OpenCV and TensorFlow.

[1413] Emotion recognition engine: Emotion analysis uses technologies such as IBM Watson and Microsoft Azure Emotion API.

[1414] Database: A database for storing metadata and user information.

[1415] Natural language processing engine: Analyzes user input and generates dialogue scenarios.

[1416] System Operation

[1417] 1. Select and upload a photo

[1418] First, the user launches the application on the device. The device displays the user's photo library and allows the user to select memorable photos. For example, the user may select photos from a family trip or a special event. The selected photos are previewed on the device and then sent to the server when the user clicks the "Upload" button.

[1419] 2. Receiving and analyzing photos

[1420] The server receives the photo data sent from the device and temporarily stores it in storage. It then begins analyzing the photo using OpenCV and TensorFlow. The image analysis engine identifies people and objects in the photo and uses facial recognition technology to identify who is in the photo. This process utilizes the Dlib and FaceNet libraries.

[1421] 3. Collecting metadata and generating memories

[1422] The server analyzes the EXIF ​​data of the photo to obtain the date and time the photo was taken and location information. Based on this metadata, it determines when and where the photo was taken. It then combines the results of the image analysis with information from the database to obtain detailed information about the people in the photo (such as their names and relationships). Based on this information, the server then creates a document that describes memorable episodes related to the photo.

[1423] 4. Emotion Recognition and Dialogue Scenario Generation

[1424] Based on the generated memory information, the server formulates a dialogue scenario. It activates an emotion recognition engine to recognize emotions from the user's voice and text input. Based on this emotion information, the dialogue scenario is adjusted to provide more appropriate questions and feedback.

[1425] 5. Providing dialogue scenarios and feedback

[1426] The server sends a list of questions based on the generated dialogue scenario and emotional information to the device. The device displays this to the user, preparing to begin the dialogue. The user responds to the questions displayed on the screen by text input or voice input. The device sends these responses to the server in real time. The server analyzes the responses sent by the user, adjusts the dialogue content, and provides feedback appropriate to each individual.

[1427] Specific examples

[1428] For example, a specific example will be given in which the user selects a photo of "a family trip to Shonan Beach in the summer of 2005."

[1429] Prompt statement:

[1430] "I will upload photos from a family trip to Shonan Beach in the summer of 2005. Please generate a dialogue scenario in which the user talks about their memories of that time."

[1431] Generated episodes:

[1432] "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family enjoyed playing in the sea."

[1433] Generated question:

[1434] "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[1435] Additional input from the emotion engine:

[1436] "Sounds fun. Tell me more about that day."

[1437] In this way, the system stimulates memories while allowing users to enjoy past memories, contributing to the prevention and alleviation of dementia.The introduction of an emotion engine makes the dialogue more personalized, improving the user experience.

[1438] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1439] Step 1:

[1440] User selects and uploads a photo

[1441] How it works: A user launches an application on their device and selects a photo from their photo library, such as a photo from a trip or family event.

[1442] Input: A photo file selected by the user.

[1443] Output: The device displays a preview of the selected photo. An "Upload" button is provided.

[1444] Specific action: The user checks the preview and clicks the "Upload" button.

[1445] Step 2:

[1446] The device sends the photo data to the server.

[1447] Operation: The device uses the business API to send photo data to the server. The progress of the transfer is displayed to the user.

[1448] Input: The photo data selected by the user and uploaded.

[1449] Output: The photo data sent to the server.

[1450] Specific operation: The device sends data to the server via the network.

[1451] Step 3:

[1452] The server receives the photo and begins analyzing it.

[1453] How it works: The server receives the photo data and temporarily stores it in storage. OpenCV and TensorFlow are used for photo analysis.

[1454] Input: Photo data sent from the device.

[1455] Output: As a result of the analysis, specific information about people and objects in the photo.

[1456] Specific operation: The server preprocesses the image and detects human faces. Dlib and FaceNet are used as facial recognition technologies.

[1457] Step 4:

[1458] The server collects the photo metadata.

[1459] How it works: The server analyzes the EXIF ​​data of the photo to obtain the date and time the photo was taken and its location.

[1460] Input: Parsed photo data.

[1461] Output: Date and time of capture, location, and other metadata.

[1462] What it does: The server compares the retrieved metadata with an internal database to determine when and where the photo was taken.

[1463] Step 5:

[1464] The server generates the memory information

[1465] How it works: The server uses collected metadata and facial recognition results to create a document of memorable events related to the photo.

[1466] Input: Metadata, face recognition results.

[1467] Output: A written version of the memory episode.

[1468] Concrete action: For example, generate an episode such as "In the summer of 2005, we went on a family trip to Shonan Beach."

[1469] Step 6:

[1470] The server generates a dialogue scenario

[1471] How it works: The server creates a dialogue scenario with the user based on the generated memory information. It uses a generation engine to create a list of questions to guide the dialogue.

[1472] Input: A written memory episode.

[1473] Output: Dialogue scenario and questionnaire.

[1474] Concrete Actions: Formulate specific questions, such as, "What is your first memory of seeing this photo?"

[1475] Step 7:

[1476] Introducing an emotion recognition engine

[1477] How it works: The server launches an emotion recognition engine to recognize emotions from the user's voice and text input.

[1478] Input: User voice input, text input.

[1479] Output: Emotional information.

[1480] Specific actions: Emotional information is used to adjust the dialogue scenario.

[1481] Step 8:

[1482] The server sends the dialogue scenario to the terminal.

[1483] Operation: The server sends a dialogue scenario and a list of questions based on emotion information to the terminal.

[1484] Input: Dialogue scenario, emotion information.

[1485] Output: The dialogue scenario and questionnaire sent to the terminal.

[1486] Concrete behavior: This enables flexible interaction according to the user's emotions.

[1487] Step 9:

[1488] The device presents a dialogue scenario to the user.

[1489] Operation: The terminal displays the initial dialogue questions to the user and prepares to begin the dialogue.

[1490] Input: Dialogue scenario and question list sent from the server.

[1491] Output: The dialogue questions that are displayed to the user.

[1492] Specific actions: The user responds to questions displayed on the screen by text or voice.

[1493] Step 10:

[1494] The user and the system start the conversation

[1495] How it works: The user responds to questions by typing or speaking.

[1496] Input: The user's response.

[1497] Output: User responses logged to the terminal.

[1498] Specific operation: The terminal sends the response to the server in real time.

[1499] Step 11:

[1500] The server analyzes the conversation and provides feedback

[1501] How it works: The server analyzes the user's responses and adjusts the interaction to provide personalized feedback.

[1502] Input: The user's response sent from the terminal.

[1503] Output: Feedback to the user with next questions or advice.

[1504] Specific operation: The server uses natural language processing technology to analyze the user's response and generate the next question or feedback.

[1505] (Application example 2)

[1506] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1507] Conventional memory stimulation systems do not adequately consider the user's emotions, resulting in a lack of personalized interaction experiences and the inability to effectively stimulate memory or improve skills. Furthermore, it is difficult to evaluate and provide feedback on the user's skills and knowledge in real time.

[1508] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving photo data from a user's terminal, means for analyzing the received photo data using an image analysis engine and identifying objects in the photo, means for collecting photo attribute information and generating past event information based on the analysis results, means for creating a dialogue scenario based on the generated event information, means for recognizing emotions from the user's voice or text input and adjusting the dialogue scenario, means for conducting a dialogue with the user based on the dialogue scenario, and means for evaluating the user's skill and knowledge status and providing feedback. This enables dialogue optimized for individual users and real-time skill evaluation and feedback.

[1509] "User's device" refers to an electronic device used by a user, such as a smartphone, tablet, PC, or wearable device.

[1510] "Photo data" refers to still image data that a user has taken, saved, or selected through a terminal.

[1511] An "image analysis engine" is a software module that analyzes received image data and identifies people and objects within the image.

[1512] "Subjects" are elements that are to be recognized, such as people, animals, objects, and scenery, contained in an image.

[1513] "Attribute information" refers to metadata and context information related to image data, and includes, for example, the date and time of shooting, location information, shooting environment, and subject characteristics.

[1514] "Past event information" is an episode or story related to a specific time or place in the past, generated based on attribute information.

[1515] A "dialogue scenario" is a script of a series of questions and answers for dialogue with a user, constructed based on generated past event information.

[1516] "Emotion" recognition refers to analyzing voice or text input to identify a user's emotional state (e.g., joy, sadness, anger, etc.).

[1517] "Skill and knowledge status" refers to the level of skill and knowledge that a user possesses regarding a specific operation or task.

[1518] The "means for providing feedback" is a software module for providing appropriate advice and evaluation in real time based on the user's answers and actions.

[1519] This invention is a system that aims to alleviate the early symptoms of dementia by stimulating the user's memory using past photographs. This system functions through the interaction of three parties: the server, the terminal, and the user. Specifically, it has the following features:

[1520] 1. Receiving and analyzing photo data

[1521] The device has a function that allows users to select and upload past photos. It is assumed that users will launch an application on the device and select photos from a family trip or a special event. The selected photos are sent to the server by pressing the "upload" button on the device.

[1522] The server temporarily stores the received photo data in storage and launches an image analysis engine (such as OpenCV or TensorFlow). Object recognition technology (such as YOLOv3 or OpenCV) is used to identify the subject in the photo. It also collects photo attribute information (such as EXIF ​​data) and obtains the date and time the photo was taken and its location. Based on the results of this analysis, past event information is generated.

[1523] 2. Dialogue scenario generation and emotion recognition

[1524] The server creates a dialogue scenario based on the generated past event information. It then uses an emotion engine that analyzes voice and text input to recognize the user's emotions and adjust the dialogue scenario accordingly. During this process, questions and feedback based on the emotion information are added.

[1525] 3. User interaction and feedback

[1526] The device presents the dialogue scenario sent from the server to the user and initiates the dialogue. The user responds to questions displayed on the screen by text input or voice input. The user's responses are sent in real time to the server, which then uses natural language processing technology and an emotion engine to generate the next question and feedback.

[1527] 4. Skill evaluation and feedback

[1528] The server collects dialogue logs and evaluates the user's skills and knowledge. Based on this, individual feedback is generated and provided to the user via their device. This feedback evaluates the user's skill and knowledge level for specific operations and tasks, and provides appropriate advice and evaluation in real time.

[1529] Specific examples

[1530] For example, consider a scenario in which a user selects a photo from a family trip from their childhood and starts a conversation based on the topic of "Memories of having fun with my family at Shonan Beach in the summer of 2005." The user uploads the photo, and the server analyzes it to generate anecdotes and questions like the following:

[1531] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[1532] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[1533] In addition, an emotion engine is used to recognize the user's emotions, and questions that are more in line with their emotions are added, such as, "You seem to be having fun. Tell me more about what happened that day."

[1534] Prompt Sentence Examples

[1535] "Please upload photos of your market so we can train you on your business."

[1536] "What is in the uploaded photo? For example, what are the names of machine parts or devices?"

[1537] This system allows users to enjoy past memories while stimulating and improving their memory. The introduction of an emotion engine makes interactions more personalized, improving the user experience.

[1538] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1539] Step 1:

[1540] The device allows the user to select past photos and upload them through the application (input: photo data selected by the user, output: photo data sent to the server). The device displays the selected photos as a preview and provides an upload button. When the user clicks the upload button, the photo data is sent to the server.

[1541] Step 2:

[1542] The server temporarily stores the received photo data in storage and launches an image analysis engine (such as OpenCV or TensorFlow) (input: photo data, output: photo data stored in storage). This engine uses object recognition technology (such as YOLOv3 or OpenCV) to identify objects in the photo.

[1543] Step 3:

[1544] The server analyzes the attribute information of the photo (e.g., EXIF ​​data) and collects the attribute information (input: photo data saved in storage, output: attribute information). This allows the server to identify when and where the photo was taken.

[1545] Step 4:

[1546] The server generates past event information based on the analyzed attribute information (input: attribute information, output: past event information). For example, it generates a specific episode such as "In the summer of 2005, we went on a family trip to Shonan Beach."

[1547] Step 5:

[1548] The server creates a dialogue scenario based on the generated past event information (input: past event information, output: dialogue scenario). Furthermore, it uses an emotion engine that analyzes voice and text input to recognize the user's emotions and adjust the dialogue scenario accordingly.

[1549] Step 6:

[1550] The server creates questions and feedback based on the generated dialogue scenario and emotional information and sends them to the terminal (input: dialogue scenario, emotional information; output: questions and feedback sent to the terminal).

[1551] Step 7:

[1552] The terminal presents the dialogue scenario sent from the server to the user and starts the dialogue (input: questions and feedback sent from the server, output: user's response). The user responds to the questions displayed on the screen by text input or voice input.

[1553] Step 8:

[1554] The terminal records the user's response and transmits it to the server in real time (input: user's response data, output: real-time response data transmitted to the server).

[1555] Step 9:

[1556] The server analyzes the user's response data and generates the next question and feedback using natural language processing technology and an emotion engine (input: user's real-time response data, output: next question and feedback).

[1557] Step 10:

[1558] The server collects dialogue logs and evaluates the user's skills and knowledge (input: dialogue log data, output: user skill evaluation data). Based on this, individual feedback is generated and provided to the user via their device.

[1559] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1560] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1561] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1562] [Fourth embodiment]

[1563] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1564] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1565] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1566] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1567] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1568] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1569] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1570] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1571] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1572] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1573] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1574] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1575] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1576] The present invention provides a system for stimulating a user's memory by using past photographs and alleviating early symptoms of dementia. Specific embodiments of the system will be described below.

[1577] System Overview

[1578] This system works by interacting with the server, the device, and the user. The user uploads past photos from their device to the server, and the server analyzes the photos to generate a dialogue scenario, which is then presented to the user via their device. The user can then have a conversation based on the scenario, allowing them to enjoyably reminisce about their memories.

[1579] System processing flow

[1580] 1. User selects and uploads a photo:

[1581] User: First, open the application on the device and select some old photos, such as photos from a family trip or a special event.

[1582] Terminal: It then displays a preview of the selected photo to the user and provides an "Upload" button.

[1583] User: Check the preview and click the "Upload" button.

[1584] 2. The device sends the photo data to the server:

[1585] Terminal: Uses a business API to send the selected photo data to the server. At this time, the network connection status and progress are displayed to the user.

[1586] 3. The server receives the photo and begins analyzing it:

[1587] Server: Temporarily stores the received photo data in storage and launches an image analysis engine, using technologies such as OpenCV and TensorFlow.

[1588] Server: Identifies people and objects in the photo and uses facial recognition technology to identify people in the shot, leveraging libraries like Dlib and FaceNet.

[1589] 4. The server collects the metadata:

[1590] Server: Extracts photo metadata (such as the date and time of the photo, location information, etc.) from the EXIF ​​data and obtains information about identified people from an internal database. For example, it may collect information such as that the photo was taken at Shonan Beach or that a specific person is a family member.

[1591] 5. The server generates the relevant memories:

[1592] Server: Based on the collected metadata, the server creates text about memorable episodes related to the photo. For example, it generates information such as "Memories of a family trip to Shonan Beach in the summer of 2005."

[1593] Server: Based on this generated information, a dialogue scenario is formulated and a list of questions is created to guide the conversation with the user.

[1594] 6. The server sends the dialogue scenario to the device:

[1595] Server: Sends the dialogue scenario and the generated question list to the terminal.

[1596] Terminal: Prepares to display the dialogue scenario on the user's screen based on the information received from the server.

[1597] 7. The device presents the dialogue scenario to the user:

[1598] Terminal: Displays the initial dialogue questions to the user and notifies them that they are ready to begin the dialogue.

[1599] User: Responds to questions displayed on the device by text input or voice input.

[1600] 8. User and system begin interaction:

[1601] Terminal: Records the user's responses and continuously sends them to the server.

[1602] Server: Analyzes the user's response, generates and selects the next question or feedback, and sends it to the device. Through this cycle, the user can continue the dialogue.

[1603] 9. The server analyzes the dialogue and provides feedback:

[1604] Server: Collects dialogue logs and analyzes user responses using natural language processing technology. Evaluates the user's memory status and emotional changes, and uses this information for the next dialogue.

[1605] Server: Creates individual feedback and follow-up questions based on the user's memory impairment status and provides them to the user via the device.

[1606] Specific examples

[1607] For example, consider a scenario in which a user selects a photo from a family trip from their childhood and starts a conversation based on the topic of "Memories of having fun with my family at Shonan Beach in the summer of 2005." The user uploads the photo, and the server analyzes it to generate anecdotes and questions like the following:

[1608] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[1609] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[1610] As the user answers these questions, the system elicits more detailed information and stimulates the user's memory.

[1611] This system allows users to enjoy memories of the past while stimulating and improving their memory.

[1612] The processing flow will be explained below.

[1613] Step 1:

[1614] The user launches the application on the device, opens an interface within the application for selecting past photos, and selects a specific photo from the saved photos.

[1615] Step 2:

[1616] The device displays a preview of the selected photo to the user and provides an "Upload" button to confirm that the photo is correct.

[1617] Step 3:

[1618] The user clicks the "Upload" button, which completes preparations for uploading the photo data.

[1619] Step 4:

[1620] The device sends the selected photo data to the server using a business API, and the progress of the network connection is displayed to the user.

[1621] Step 5:

[1622] The server stores the received photo data in temporary storage, then launches the image analysis engine and begins analyzing the photo.

[1623] Step 6:

[1624] The server uses an image analysis engine to identify people and objects in the photo, for example, by using facial recognition technology to identify people, using libraries such as Dlib and FaceNet.

[1625] Step 7:

[1626] The server extracts the photo's metadata (e.g., EXIF ​​data) and obtains the date and time the photo was taken and its location, which identifies when and where the photo was taken.

[1627] Step 8:

[1628] The server references an internal database and obtains detailed information about the people in the photo (e.g., names, relationships, etc.) from the facial recognition results.

[1629] Step 9:

[1630] The server generates information about past memories based on the analysis results and metadata. For example, it creates a story such as "I went on a family trip to Shonan Beach in the summer of 2005."

[1631] Step 10:

[1632] The server creates a dialogue scenario based on the generated memory information, which includes a list of questions to prompt dialogue with the user.

[1633] Step 11:

[1634] The server sends the created dialogue scenario to the terminal. The scenario includes episodes and related questions.

[1635] Step 12:

[1636] The terminal prepares to display the dialogue scenario received from the server on the user screen, presents the first question, and notifies the user that the dialogue is ready to begin.

[1637] Step 13:

[1638] The user responds to questions displayed on the screen by text input or voice input, and the device records the user's responses.

[1639] Step 14:

[1640] The device sends the user's responses in real time to the server, which then generates the next question or feedback based on the received data.

[1641] Step 15:

[1642] The server analyzes the user's responses and uses natural language processing techniques (e.g., sentiment analysis) to assess the user's memory state and emotional changes.

[1643] Step 16:

[1644] The server generates the next question and feedback and sends it to the device, which may include personalized questions and suggestions to help the user improve their memory.

[1645] Step 17:

[1646] The device receives new questions and feedback from the server and presents them to the user, allowing the user to continue interacting and stimulating their memory.

[1647] Step 18:

[1648] When the user indicates an intention to end the interactive session (for example, by clicking an "end" button), the terminal notifies the server of this information.

[1649] Step 19:

[1650] The server acknowledges the end of the interaction session and generates a summary and feedback of the interaction, including an assessment of the user's memory state and the progress of the interaction.

[1651] Step 20:

[1652] The device displays feedback from the server to the user, suggests next sessions, saves records, and allows the user to see the results of the interaction.

[1653] Example 1

[1654] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1655] Users in the early stages of dementia have difficulty maintaining and recalling memories in their daily lives, which increases their mental burden. Current systems have limited methods for effectively utilizing users' past photos to stimulate memories, and the lack of an interactive approach reduces opportunities for users to actively reflect on the past. Furthermore, the system lacks mechanisms for monitoring the progress of memory impairment and providing appropriate feedback.

[1656] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1657] In this invention, the server includes a means for receiving image data from the user's information processing device, a means for analyzing the received image data using an image analysis device and identifying objects in the image, a means for collecting additional information about the image based on the analysis results and generating past memory information, a means for creating a dialogue plan based on the generated memory information, a means for engaging in a dialogue with the user based on the dialogue plan, a means for recording the user's responses during the dialogue and transmitting the data, a means for generating dialogue questions and feedback using a generative AI model, and a means for collecting dialogue logs and analyzing them using natural language processing technology. This allows the user to use past photos to stimulate their memories and efficiently recall them. Furthermore, by monitoring the user's memory status through dialogue and providing appropriate feedback, it is possible to maintain and improve cognitive function.

[1658] "User" refers to the person who uses this system, and is particularly targeted at people in the early stages of dementia.

[1659] An "information processing device" is a terminal or device operated by a user, and allows operations such as selecting and uploading photos.

[1660] "Image data" refers to digital images such as photographs and drawings uploaded by users.

[1661] An "image analysis device" is a device that analyzes image data and identifies subjects (people or objects) within the image, and includes an image analysis engine and library.

[1662] "Additional information" refers to metadata such as the image's EXIF ​​data, as well as information related to the subject matter within the image.

[1663] "Memories" refers to past events and episodes that are generated based on the results of image analysis and additional information.

[1664] "Dialogue planning" refers to a scenario or question list that guides a dialogue with the user based on memory information.

[1665] "Generative AI model" refers to an artificial intelligence model used to automatically generate dialogue questions and feedback.

[1666] "Natural language processing technology" refers to technology for analyzing dialogue logs and understanding user responses, and includes technology used for sentence generation and sentiment analysis.

[1667] The present invention provides a system for stimulating past memories in a user and alleviating early symptoms of dementia. Specific processing and embodiments of the program of this system will be described in detail below.

[1668] This system functions through the interaction of three parties: a server, a terminal, and a user. The server is responsible for a wide range of processes, including receiving and analyzing image data, extracting metadata, and generating a dialogue plan. The terminal provides an interface for users to upload photos and initiate dialogue. Users use this interface to select and upload photos and engage in dialogue according to the dialogue scenario provided by the server.

[1669] Server processing

[1670] The server first receives image data from the user. To do this, the server receives image data sent from the device using a RESTful API. The received image data is then temporarily stored in cloud storage. The main technology used is likely to be a cloud service such as Amazon S3.

[1671] The stored image data is analyzed using an image analyzer (e.g., OpenCV or TensorFlow) to identify people and objects in the image and use facial recognition technology (e.g., Dlib or FaceNet). Based on the analysis results, the EXIF ​​data of the photo is extracted and metadata such as the date and time of the photo's capture and location are collected using a library such as ExifTool.

[1672] The collected metadata and analysis results are integrated to generate memory information. Natural language generation technology (generative AI model) is used to create sentences about specific episodes, such as "Memories of a family trip to Shonan Beach in the summer of 2005." Based on the generated memory information, a dialogue plan is also created, and a list of questions is created to guide the conversation with the user.

[1673] The generated dialogue plan and question list are sent to the terminal via encrypted communication.

[1674] About terminal processing

[1675] The device accepts operations from the user and sends them to the server. The user opens the dedicated application, selects and uploads photos. During uploading, a progress bar is displayed to notify the user of the progress. The device displays the dialogue scenario received from the server, allowing the user to start a dialogue. During the dialogue, the device records the user's responses and continuously sends them to the server.

[1676] About User Roles

[1677] The user selects and uploads past photos using the device. Based on the dialogue scenario provided by the server, the user responds by text input or voice input. Through this process, the user's past memories are stimulated, and it is expected that their memory will be improved.

[1678] Specific examples

[1679] For example, if a user uploads a photo of a family trip to Shonan Beach in the summer of 2005, the server processes it as follows: First, it extracts "Date and time of photo: Summer 2005, Location: Shonan Beach" from the photo's EXIF ​​data. Next, it uses facial recognition technology to identify family members and generates anecdotes and questions like the following:

[1680] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[1681] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[1682] As the user answers these questions, the system elicits more detailed information and stimulates the user's memory.

[1683] Prompt Sentence Examples

[1684] "Generate a conversation scenario based on memories from photos of a family trip to Shonan Beach."

[1685] "Create a story related to the person in this photo."

[1686] As described above, this system aims to effectively stimulate the user's memory by linking various hardware and software.

[1687] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1688] Step 1: Select and upload a photo

[1689] Subject: User

[1690] Input: Photo data selected on your device

[1691] Output: Uploaded photo data

[1692] The user opens a dedicated application on their device and selects a past photo. The selected photo is previewed, and when the user clicks the "Upload" button, the photo data is sent to the server.

[1693] Step 2: Receiving and saving photo data

[1694] Subject: Server

[1695] Input: Photo data uploaded by the user

[1696] Output: Saved photo data (cloud storage)

[1697] The server receives the photo data sent from the device and temporarily stores it in cloud storage, using a cloud service such as Amazon S3.

[1698] Step 3: Analyze the photo data

[1699] Subject: Server

[1700] Input: Saved photo data

[1701] Output: Analysis results (specific information about people and objects)

[1702] The server analyzes the stored photo data using image analysis equipment (e.g., OpenCV or TensorFlow). The analysis identifies people and objects in the photos. The analysis results also include information on people identified using facial recognition technology (Dlib or FaceNet).

[1703] Step 4: Collecting Metadata

[1704] Subject: Server

[1705] Input: Analysis results and photo EXIF ​​data

[1706] Output: Collected metadata (date and time of photo, location, etc.)

[1707] The server extracts the EXIF ​​data from the photos using tools such as ExifTool, and collects metadata such as the date and time the photo was taken and the location information. This is then combined with the analysis results to collect information on identified individuals.

[1708] Step 5: Generate memories

[1709] Subject: Server

[1710] Input: Collected metadata and analysis results

[1711] Output: Generated memories

[1712] The server uses natural language generation technology (generative AI model) to generate memory information based on the collected metadata and analysis results. For example, a detailed episode such as "Memories of a family trip to Shonan Beach in the summer of 2005" can be written.

[1713] Step 6: Developing dialogue scenarios

[1714] Subject: Server

[1715] Input: Generated memory information

[1716] Output: Dialogue scenario and questionnaire

[1717] The server formulates a dialogue plan based on the generated memory information and creates a specific list of questions for the user, such as "What is your first memory of seeing this photo?"

[1718] Step 7: Sending the dialogue scenario to the device

[1719] Subject: Server

[1720] Input: Dialogue scenario and questionnaire

[1721] Output: Dialogue scenario and question list sent to the terminal

[1722] The server sends the dialogue scenario and question list to the terminal through encrypted communication.

[1723] Step 8: Present the dialogue scenario

[1724] Subject: Device

[1725] Input: Dialogue scenario and question list received from the server

[1726] Output: The dialogue scenario and question list presented to the user

[1727] The terminal presents the dialogue scenario and question list received from the server to the user, and displays the initial dialogue questions in a pop-up format on the screen to prompt the user to start the dialogue.

[1728] Step 9: User responses and recording

[1729] Subject: User and Device

[1730] Input: User input (text or voice)

[1731] Output: Recorded user responses

[1732] The user responds to questions displayed on the device by text input or voice input, and the device records the user's responses and continuously transmits them to the server.

[1733] Step 10: Continued dialogue and feedback

[1734] Subject: Server

[1735] Input: User response data

[1736] Output: Next dialogue question or feedback

[1737] The server analyzes the user's response data and uses a generative AI model to generate the next question or feedback, allowing the user to continue the dialogue.

[1738] These are the specific processing steps of the system, which aims to effectively stimulate the user's memory and alleviate the early symptoms of dementia.

[1739] (Application example 1)

[1740] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1741] Today, there is a need for effective methods to stimulate memories and alleviate the condition of people in the early stages of dementia. However, existing systems are limited to reviewing old photos and lack the ability to delve deeper into memories or re-experience memories in a more immersive way. Furthermore, they lack the precision to elicit memories through dialogue and the ability to personalize the user experience. Therefore, new technologies are needed to enable users to re-experience past memories in a more realistic way and stimulate and improve memory.

[1742] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1743] In this invention, the server includes means for receiving photo data from a user's device, means for analyzing the received photo data using an image analysis engine and identifying people and objects in the photo, means for collecting photo metadata and generating past memory information based on the analysis results, means for creating a dialogue scenario based on the generated memory information, means for presenting the dialogue scenario to the user and displaying it on a head-mounted display so that the user can re-experience the experience in a virtual reality space, and means for analyzing the user's response using a natural language processing engine and generating the next question and feedback. This allows the user to re-experience memories in a realistic virtual reality space, making it possible to effectively stimulate and improve memory in the early stages of dementia.

[1744] A "user" is an individual who uses the system to re-experience past memories and stimulate their memories.

[1745] "Device" means an electronic device used by a user to upload photo data and receive and display dialogue scenarios.

[1746] "Photo data" refers to image files that have been taken in the past and are uploaded by users.

[1747] The "image analysis engine" is a software component that analyzes received photo data and identifies people and objects in the photo.

[1748] "Metadata" refers to accompanying information included in photo data, such as the date and time of shooting, location information, and information about the shooting device.

[1749] "Memories information" is information about past events generated based on the results of image analysis and photo metadata.

[1750] A "dialogue scenario" is a scenario that includes questions and topics to promote dialogue with the user based on the generated memory information.

[1751] "Virtual reality" is a virtual space generated using computer graphics, in which users can experience a realistic virtual environment.

[1752] A "head-mounted display" is a display device worn by the user on the head to allow the user to visually experience a virtual reality space.

[1753] A "natural language processing engine" is a software component that analyzes a user's dialogue responses and generates the next question or feedback.

[1754] System Overview

[1755] This invention is a system that allows users to use past photographs to stimulate their memories and alleviate the early symptoms of dementia. This system functions through the interaction of a server, a terminal, and a user.

[1756] How to run the program

[1757] 1. User Conduct

[1758] Users use their devices to select past photos, such as photos from a family trip or special event, and upload them to the server.

[1759] 2. Device Operation

[1760] The terminal displays a preview of the selected photo to the user and provides an "Upload" button, which causes the terminal to send the photo data to the server.

[1761] 3. Server Processing

[1762] The server analyzes the received photo data using an image analysis engine (e.g., OpenCV or TensorFlow). It performs facial and object recognition to identify people and objects in the photo. It also recognizes identified people using libraries such as Dlib and FaceNet. It also extracts photo metadata (such as the date and time of the photo shoot and location information) from the EXIF ​​data and obtains information about the identified people from an internal database.

[1763] Next, the server generates a memorable episode related to the photo based on the collected metadata. For example, it writes down information such as "Memories of a family trip to Shonan Beach in the summer of 2005." It then formulates a dialogue scenario based on the generated information and creates a list of questions to encourage conversation with the user.

[1764] 4. Use of head-mounted displays

[1765] The dialogue scenario and question list are sent from the server to the device and head-mounted display, which generates a virtual reality space for the user to re-experience, and presents the dialogue scenario within it.

[1766] 5. Dialogue with users

[1767] The user wears a head-mounted display and responds according to the displayed dialogue scenario. The user's responses are analyzed by a natural language processing engine (e.g., spaCy), which generates the next question or feedback. This allows the dialogue with the user to continue and elicit more memories.

[1768] Specific examples

[1769] For example, the following prompts can be input to a generative AI model:

[1770] "Generate a dialogue scenario and a list of questions for memory stimulation based on this photo taken at Shonan Beach in the summer of 2005."

[1771] The generated dialogue scenario example is as follows:

[1772] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[1773] question:

[1774] "What's your first memory of seeing this photo?"

[1775] Who did you go to the beach with that day?

[1776] This system allows users to re-experience past memories in an immersive virtual reality space, stimulating and improving memory.

[1777] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1778] Step 1:

[1779] Input: A previous photo selected by the user on their device

[1780] How it works: A user opens an application on their device and selects a photo from their past, such as a family vacation or a special event. The device previews the selected photo, and the user clicks the "Upload" button.

[1781] Output: Photo data ready to upload

[1782] Step 2:

[1783] Input: The "Upload" button the user clicked

[1784] Operation: The device sends the photo data to the server, and uses a business API to display the network connection status and progress.

[1785] Output: Photo data sent to the server

[1786] Step 3:

[1787] Input: Photo data received by the server

[1788] How it works: The server temporarily stores the received photo data and begins analyzing it using an image analysis engine (OpenCV or TensorFlow). Through this analysis, it identifies people and objects in the photo and uses facial recognition technology (Dlib, FaceNet, etc.) to identify people in the photo.

[1789] Output: Analysis results (information on identified people and objects)

[1790] Step 4:

[1791] Input: Analysis results (information on identified people and objects)

[1792] How it works: The server extracts the photo's metadata (date and time of capture, location, etc.) from the EXIF ​​data and retrieves the identified person's information from an internal database.

[1793] Output: Extracted metadata and identified people

[1794] Step 5:

[1795] Input: Extracted metadata and identified people

[1796] How it works: The server generates a memorable episode related to the photo based on the collected metadata. For example, it generates information such as "Memories of a family trip to Shonan Beach in the summer of 2005." It also uses this information to formulate a dialogue scenario and create a list of questions to guide the conversation with the user.

[1797] Output: Generated memory episodes and dialogue scenarios

[1798] Step 6:

[1799] Input: Generated memory episodes and dialogue scenarios

[1800] Operation: The server sends the dialogue scenario and the generated question list to the device and the head-mounted display. Based on the information received from the server, the device prepares to display the dialogue scenario on the user's screen.

[1801] Output: Dialogue scenario and question list sent to the device and head-mounted display

[1802] Step 7:

[1803] Input: Terminal and head-mounted display that received the dialogue scenario and question list

[1804] How it works: The head-mounted display displays the dialogue scenario in a virtual reality space, allowing the user to begin the immersive experience. The device also displays the initial dialogue questions to the user, notifying them that they are ready to begin the dialogue.

[1805] Output: Display of the interaction scenario in a virtual reality space

[1806] Step 8:

[1807] Input: User response (text input or voice input)

[1808] How it works: The device records the user's responses and continuously sends them to the server.

[1809] Output: The user's response sent to the server

[1810] Step 9:

[1811] Input: The user's response sent to the server

[1812] How it works: The server analyzes the user's response using a natural language processing engine (e.g., spaCy) and generates the next question or feedback. The dialogue with the user continues through this cycle.

[1813] Output: Next question or feedback

[1814] Step 10:

[1815] Input: Next question or feedback sent from the server

[1816] Action: The head-mounted display and device present the next question or feedback to the user, and the user continues the interaction.

[1817] Output: The next question or feedback presented to the user

[1818] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1819] The present invention is a system that allows users to use past photos to stimulate their memories and alleviate the early symptoms of dementia. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system aims to optimize the interaction experience and improve the quality of feedback. A specific embodiment of the system is described below.

[1820] System Overview

[1821] This system achieves its functions through the interaction of three parties: the server, the device, and the user. Users upload past photos from their device, and the server analyzes the photos to generate a dialogue scenario. In doing so, it uses an emotion engine to recognize the user's emotions and provide more appropriate questions and feedback.

[1822] System processing flow

[1823] 1. User selects and uploads a photo:

[1824] User: First, launch the application on the device and select some past photos. For example, you might select photos from a family trip or a special event.

[1825] On the device: Shows the user a preview of the selected photo and provides an "Upload" button.

[1826] User: Check the preview and click the "Upload" button.

[1827] 2. The device sends the photo data to the server:

[1828] On the device: Uses a business API to send the selected photo data to the server. Displays the progress of the network connection to the user.

[1829] 3. The server receives the photo and begins analyzing it:

[1830] Server: Temporarily stores the received photo data in storage and launches an image analysis engine, using technologies such as OpenCV and TensorFlow.

[1831] Server: Identifies people and objects in the photo and uses facial recognition technology to identify people in the shot, leveraging libraries like Dlib and FaceNet.

[1832] 4. The server collects the metadata:

[1833] Server: Extracts photo metadata (e.g., EXIF ​​data) and captures the date and time of the photo and its location, which allows us to determine when and where the photo was taken.

[1834] Server: Refers to an internal database and obtains detailed information about the people in the photo (e.g., names, relationships, etc.) from the facial recognition results.

[1835] 5. The server generates the relevant memories:

[1836] Server: Based on the collected metadata, the server creates a story about a memorable episode related to the photo. For example, it creates a story like, "In the summer of 2005, we went on a family trip to Shonan Beach."

[1837] Server: Based on this generated information, a dialogue scenario is formulated and a list of questions is created to guide the conversation with the user.

[1838] 6. Introducing the Emotion Engine:

[1839] Server: Activates the emotion engine and recognizes emotions from the user's voice and text input. This emotion information is used to adjust the dialogue scenario and generate feedback.

[1840] 7. The server sends the dialogue scenario to the device:

[1841] Server: Sends a dialogue scenario and a list of questions based on emotional information to the device, enabling flexible dialogue according to the user's emotions.

[1842] 8. The device presents the dialogue scenario to the user:

[1843] Terminal: The initial dialogue questions are displayed to the user, preparing them to start the dialogue. The questions are displayed taking into account emotional information.

[1844] User: Responds to questions displayed on the screen by typing text or speaking.

[1845] 9. User and system begin interaction:

[1846] Terminal: Records the user's responses and sends them to the server in real time.

[1847] Server: Analyzes the user's response and uses natural language processing technology and an emotion engine to generate the next question or feedback.

[1848] 10. The server analyzes the dialogue and provides feedback:

[1849] Server: Collects interaction logs and evaluates the user's emotional state and memory vividness, adjusting the interaction content and providing personalized feedback.

[1850] Server: Generates personalized feedback tailored to the user's memory impairment and provides it to the user via the device.

[1851] Specific examples

[1852] For example, consider a scenario in which a user selects a photo from a family trip from their childhood and starts a conversation based on the topic of "Memories of having fun with my family at Shonan Beach in the summer of 2005." The user uploads the photo, and the server analyzes it to generate anecdotes and questions like the following:

[1853] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[1854] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[1855] In addition, an emotion engine is used to recognize the user's emotions, and questions that are more in line with the user's emotions are added, such as, "You seem to be having fun. Tell me more about what happened that day."

[1856] This system allows users to enjoy past memories while stimulating and improving their memory. The introduction of an emotion engine makes the interaction more personalized, improving the user experience.

[1857] The processing flow will be explained below.

[1858] Step 1:

[1859] The user launches the application on the device, opens an interface within the application for selecting past photos, and selects a specific photo from the saved photos.

[1860] Step 2:

[1861] The device displays a preview of the selected photo to the user and provides an "Upload" button to confirm that the photo is correct.

[1862] Step 3:

[1863] The user clicks the "Upload" button, which completes preparations for uploading the photo data.

[1864] Step 4:

[1865] The device sends the selected photo data to the server using a business API, and the progress of the network connection is displayed to the user.

[1866] Step 5:

[1867] The server stores the received photo data in temporary storage, then launches the image analysis engine and begins analyzing the photo.

[1868] Step 6:

[1869] The server uses an image analysis engine to identify people and objects in the photo, for example, by using facial recognition technology to identify people, using libraries such as Dlib and FaceNet.

[1870] Step 7:

[1871] The server extracts the photo's metadata (e.g., EXIF ​​data) and obtains the date and time the photo was taken and its location, which identifies when and where the photo was taken.

[1872] Step 8:

[1873] The server references an internal database and obtains detailed information about the people in the photo (e.g., names, relationships, etc.) from the facial recognition results.

[1874] Step 9:

[1875] The server generates information about past memories based on the analysis results and metadata. For example, it creates a story such as "I went on a family trip to Shonan Beach in the summer of 2005."

[1876] Step 10:

[1877] The server creates a dialogue scenario based on the generated memory information, which includes a list of questions to prompt dialogue with the user.

[1878] Step 11:

[1879] The emotion engine is activated. The server extracts emotion data from the user's voice and text and adjusts the dialogue scenario based on this data.

[1880] Step 12:

[1881] The server uses the user's emotional data to generate more personalized questions and feedback. For example, if the user seems to be having fun, it will generate questions like, "What other fun things happened that day?"

[1882] Step 13:

[1883] The server sends a dialogue scenario and a list of questions based on emotion data to the terminal.

[1884] Step 14:

[1885] The terminal prepares to display the dialogue scenario received from the server on the user screen, presents the first question, and notifies the user that the dialogue is ready to begin.

[1886] Step 15:

[1887] The user responds to questions displayed on the screen by text input or voice input, and the device records the user's responses.

[1888] Step 16:

[1889] The device sends the user's responses in real time to the server, which then generates the next question or feedback based on the received data.

[1890] Step 17:

[1891] The server analyzes the user's responses and uses natural language processing techniques and an emotion engine to assess the user's memory state and emotional changes.

[1892] Step 18:

[1893] The server generates the next question and feedback and sends it to the device, which may include personalized questions and suggestions to help the user improve their memory.

[1894] Step 19:

[1895] The device receives new questions and feedback from the server and presents them to the user, allowing the user to continue interacting and stimulating their memory.

[1896] Step 20:

[1897] When the user indicates an intention to end the interactive session (for example, by clicking an "end" button), the terminal notifies the server of this information.

[1898] Step 21:

[1899] The server acknowledges the end of the interaction session and generates a summary and feedback of the interaction, including an assessment of the user's memory state and the progress of the interaction.

[1900] Step 22:

[1901] The device displays feedback from the server to the user, suggests next sessions, saves records, and allows the user to see the results of the interaction.

[1902] Example 2

[1903] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1904] In recent years, with the advancement of an aging society, the prevention and treatment of dementia have become important issues. Memory retention and emotional stability are particularly important issues in the early stages of dementia. Current methods offer limited means for users to stimulate past memories, and more effective methods are needed to recreate and evoke past memories. Furthermore, due to the lack of systems that can adequately grasp users' emotional states and provide feedback through dialogue, there is a need for methods to improve the quality of dialogue.

[1905] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving photo data from a user's terminal, means for analyzing the received photo data using an image analysis engine to identify people and objects in the photo, means for collecting photo metadata based on the analysis results and generating past memory information, means for creating a dialogue scenario based on the generated memory information, means including an emotion recognition engine for detecting the user's emotions and adjusting the dialogue scenario, means for providing flexible dialogue based on the generated dialogue scenario and emotion information, and means for analyzing data obtained from the dialogue and generating and providing individual feedback. This makes it possible to effectively recreate past memories and promote memory retention and emotional stability through dialogue that is in line with the user's emotions.

[1906] "User" means a person who uses the system.

[1907] A "terminal" refers to a device operated by a user, specifically a PC, smartphone, tablet, etc.

[1908] "Photo data" refers to image files imported by the user.

[1909] "Means for receiving" refers to the function for the server to receive data sent from the terminal.

[1910] An "image analysis engine" is software that processes and analyzes image data to extract useful information.

[1911] "Means for identifying people and objects in photographs" refers to the ability to use an image analysis engine to recognize and identify specific elements within a photograph.

[1912] "Metadata" refers to detailed information that accompanies photo data, including the date the photo was taken and location information.

[1913] "Memories" refers to episodes and stories related to past memories generated based on photos and metadata.

[1914] A "dialogue scenario" is a plan that defines the progress of a conversation with a user, constructed based on the generated memory information.

[1915] An "emotion recognition engine" is software that analyzes and recognizes emotions from a user's voice and text data.

[1916] "Means for providing flexible dialogue" refers to a function that adjusts dialogue scenarios based on emotional information and provides questions and feedback that are appropriate for the user.

[1917] "Feedback" refers to responses and advice provided by a system based on interactions with a user.

[1918] "Means for analyzing data" refers to the function of analyzing information obtained from the dialogue and evaluating the user's condition and response.

[1919] "Means for generating and providing individual feedback" refers to a function that creates feedback suited to each individual user based on the analysis results and provides that feedback via the terminal.

[1920] The present invention provides a system that allows users to use past photographs to stimulate their memories and alleviate early symptoms of dementia. This system functions through the interaction of three parties: the user, the terminal, and the server. Specific embodiments are described below.

[1921] System Overview

[1922] Hardware

[1923] Server: The central server that performs the main processing of this system.

[1924] Terminal: A device operated by a user, such as a PC, smartphone, or tablet.

[1925] software

[1926] Image analysis engine: Uses image processing libraries such as OpenCV and TensorFlow.

[1927] Emotion recognition engine: Emotion analysis uses technologies such as IBM Watson and Microsoft Azure Emotion API.

[1928] Database: A database for storing metadata and user information.

[1929] Natural language processing engine: Analyzes user input and generates dialogue scenarios.

[1930] System Operation

[1931] 1. Select and upload a photo

[1932] First, the user launches the application on the device. The device displays the user's photo library and allows the user to select memorable photos. For example, the user may select photos from a family trip or a special event. The selected photos are previewed on the device and then sent to the server when the user clicks the "Upload" button.

[1933] 2. Receiving and analyzing photos

[1934] The server receives the photo data sent from the device and temporarily stores it in storage. It then begins analyzing the photo using OpenCV and TensorFlow. The image analysis engine identifies people and objects in the photo and uses facial recognition technology to identify who is in the photo. This process utilizes the Dlib and FaceNet libraries.

[1935] 3. Collecting metadata and generating memories

[1936] The server analyzes the EXIF ​​data of the photo to obtain the date and time the photo was taken and location information. Based on this metadata, it determines when and where the photo was taken. It then combines the results of the image analysis with information from the database to obtain detailed information about the people in the photo (such as their names and relationships). Based on this information, the server then creates a document that describes memorable episodes related to the photo.

[1937] 4. Emotion Recognition and Dialogue Scenario Generation

[1938] Based on the generated memory information, the server formulates a dialogue scenario. It activates an emotion recognition engine to recognize emotions from the user's voice and text input. Based on this emotion information, the dialogue scenario is adjusted to provide more appropriate questions and feedback.

[1939] 5. Providing dialogue scenarios and feedback

[1940] The server sends a list of questions based on the generated dialogue scenario and emotional information to the device. The device displays this to the user, preparing to begin the dialogue. The user responds to the questions displayed on the screen by text input or voice input. The device sends these responses to the server in real time. The server analyzes the responses sent by the user, adjusts the dialogue content, and provides feedback appropriate to each individual.

[1941] Specific examples

[1942] For example, a specific example will be given in which the user selects a photo of "a family trip to Shonan Beach in the summer of 2005."

[1943] Prompt statement:

[1944] "I will upload photos from a family trip to Shonan Beach in the summer of 2005. Please generate a dialogue scenario in which the user talks about their memories of that time."

[1945] Generated episodes:

[1946] "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family enjoyed playing in the sea."

[1947] Generated question:

[1948] "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[1949] Additional input from the emotion engine:

[1950] "Sounds fun. Tell me more about that day."

[1951] In this way, the system stimulates memories while allowing users to enjoy past memories, contributing to the prevention and alleviation of dementia.The introduction of an emotion engine makes the dialogue more personalized, improving the user experience.

[1952] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1953] Step 1:

[1954] User selects and uploads a photo

[1955] How it works: A user launches an application on their device and selects a photo from their photo library, such as a photo from a trip or family event.

[1956] Input: A photo file selected by the user.

[1957] Output: The device displays a preview of the selected photo. An "Upload" button is provided.

[1958] Specific action: The user checks the preview and clicks the "Upload" button.

[1959] Step 2:

[1960] The device sends the photo data to the server.

[1961] Operation: The device uses the business API to send photo data to the server. The progress of the transfer is displayed to the user.

[1962] Input: The photo data selected by the user and uploaded.

[1963] Output: The photo data sent to the server.

[1964] Specific operation: The device sends data to the server via the network.

[1965] Step 3:

[1966] The server receives the photo and begins analyzing it.

[1967] How it works: The server receives the photo data and temporarily stores it in storage. OpenCV and TensorFlow are used for photo analysis.

[1968] Input: Photo data sent from the device.

[1969] Output: As a result of the analysis, specific information about people and objects in the photo.

[1970] Specific operation: The server preprocesses the image and detects human faces. Dlib and FaceNet are used as facial recognition technologies.

[1971] Step 4:

[1972] The server collects the photo metadata.

[1973] How it works: The server analyzes the EXIF ​​data of the photo to obtain the date and time the photo was taken and its location.

[1974] Input: Parsed photo data.

[1975] Output: Date and time of capture, location, and other metadata.

[1976] What it does: The server compares the retrieved metadata with an internal database to determine when and where the photo was taken.

[1977] Step 5:

[1978] The server generates the memory information

[1979] How it works: The server uses collected metadata and facial recognition results to create a document of memorable events related to the photo.

[1980] Input: Metadata, face recognition results.

[1981] Output: A written version of the memory episode.

[1982] Concrete action: For example, generate an episode such as "In the summer of 2005, we went on a family trip to Shonan Beach."

[1983] Step 6:

[1984] The server generates a dialogue scenario

[1985] How it works: The server creates a dialogue scenario with the user based on the generated memory information. It uses a generation engine to create a list of questions to guide the dialogue.

[1986] Input: A written memory episode.

[1987] Output: Dialogue scenario and questionnaire.

[1988] Concrete Actions: Formulate specific questions, such as, "What is your first memory of seeing this photo?"

[1989] Step 7:

[1990] Introducing an emotion recognition engine

[1991] How it works: The server launches an emotion recognition engine to recognize emotions from the user's voice and text input.

[1992] Input: User voice input, text input.

[1993] Output: Emotional information.

[1994] Specific actions: Emotional information is used to adjust the dialogue scenario.

[1995] Step 8:

[1996] The server sends the dialogue scenario to the terminal.

[1997] Operation: The server sends a dialogue scenario and a list of questions based on emotion information to the terminal.

[1998] Input: Dialogue scenario, emotion information.

[1999] Output: The dialogue scenario and questionnaire sent to the terminal.

[2000] Concrete behavior: This enables flexible interaction according to the user's emotions.

[2001] Step 9:

[2002] The device presents a dialogue scenario to the user.

[2003] Operation: The terminal displays the initial dialogue questions to the user and prepares to begin the dialogue.

[2004] Input: Dialogue scenario and question list sent from the server.

[2005] Output: The dialogue questions that are displayed to the user.

[2006] Specific actions: The user responds to questions displayed on the screen by text or voice.

[2007] Step 10:

[2008] The user and the system start the conversation

[2009] How it works: The user responds to questions by typing or speaking.

[2010] Input: The user's response.

[2011] Output: User responses logged to the terminal.

[2012] Specific operation: The terminal sends the response to the server in real time.

[2013] Step 11:

[2014] The server analyzes the conversation and provides feedback

[2015] How it works: The server analyzes the user's responses and adjusts the interaction to provide personalized feedback.

[2016] Input: The user's response sent from the terminal.

[2017] Output: Feedback to the user with next questions or advice.

[2018] Specific operation: The server uses natural language processing technology to analyze the user's response and generate the next question or feedback.

[2019] (Application example 2)

[2020] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2021] Conventional memory stimulation systems do not adequately consider the user's emotions, resulting in a lack of personalized interaction experiences and the inability to effectively stimulate memory or improve skills. Furthermore, it is difficult to evaluate and provide feedback on the user's skills and knowledge in real time.

[2022] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving photo data from a user's terminal, means for analyzing the received photo data using an image analysis engine and identifying objects in the photo, means for collecting photo attribute information and generating past event information based on the analysis results, means for creating a dialogue scenario based on the generated event information, means for recognizing emotions from the user's voice or text input and adjusting the dialogue scenario, means for conducting a dialogue with the user based on the dialogue scenario, and means for evaluating the user's skill and knowledge status and providing feedback. This enables dialogue optimized for individual users and real-time skill evaluation and feedback.

[2023] "User's device" refers to an electronic device used by a user, such as a smartphone, tablet, PC, or wearable device.

[2024] "Photo data" refers to still image data that a user has taken, saved, or selected through a terminal.

[2025] An "image analysis engine" is a software module that analyzes received image data and identifies people and objects within the image.

[2026] "Subjects" are elements that are to be recognized, such as people, animals, objects, and scenery, contained in an image.

[2027] "Attribute information" refers to metadata and context information related to image data, and includes, for example, the date and time of shooting, location information, shooting environment, and subject characteristics.

[2028] "Past event information" is an episode or story related to a specific time or place in the past, generated based on attribute information.

[2029] A "dialogue scenario" is a script of a series of questions and answers for dialogue with a user, constructed based on generated past event information.

[2030] "Emotion" recognition refers to analyzing voice or text input to identify a user's emotional state (e.g., joy, sadness, anger, etc.).

[2031] "Skill and knowledge status" refers to the level of skill and knowledge that a user possesses regarding a specific operation or task.

[2032] The "means for providing feedback" is a software module for providing appropriate advice and evaluation in real time based on the user's answers and actions.

[2033] This invention is a system that aims to alleviate the early symptoms of dementia by stimulating the user's memory using past photographs. This system functions through the interaction of three parties: the server, the terminal, and the user. Specifically, it has the following features:

[2034] 1. Receiving and analyzing photo data

[2035] The device has a function that allows users to select and upload past photos. It is assumed that users will launch an application on the device and select photos from a family trip or a special event. The selected photos are sent to the server by pressing the "upload" button on the device.

[2036] The server temporarily stores the received photo data in storage and launches an image analysis engine (such as OpenCV or TensorFlow). Object recognition technology (such as YOLOv3 or OpenCV) is used to identify the subject in the photo. It also collects photo attribute information (such as EXIF ​​data) and obtains the date and time the photo was taken and its location. Based on the results of this analysis, past event information is generated.

[2037] 2. Dialogue scenario generation and emotion recognition

[2038] The server creates a dialogue scenario based on the generated past event information. It then uses an emotion engine that analyzes voice and text input to recognize the user's emotions and adjust the dialogue scenario accordingly. During this process, questions and feedback based on the emotion information are added.

[2039] 3. User interaction and feedback

[2040] The device presents the dialogue scenario sent from the server to the user and initiates the dialogue. The user responds to questions displayed on the screen by text input or voice input. The user's responses are sent in real time to the server, which then uses natural language processing technology and an emotion engine to generate the next question and feedback.

[2041] 4. Skill evaluation and feedback

[2042] The server collects dialogue logs and evaluates the user's skills and knowledge. Based on this, individual feedback is generated and provided to the user via their device. This feedback evaluates the user's skill and knowledge level for specific operations and tasks, and provides appropriate advice and evaluation in real time.

[2043] Specific examples

[2044] For example, consider a scenario in which a user selects a photo from a family trip from their childhood and starts a conversation based on the topic of "Memories of having fun with my family at Shonan Beach in the summer of 2005." The user uploads the photo, and the server analyzes it to generate anecdotes and questions like the following:

[2045] Storyline: "This photo was taken at Shonan Beach in the summer of 2005. It was a beautiful sunny day, and the whole family was playing in the sea."

[2046] Questions: "What is your first memory of seeing this photo?" "Who did you go to the beach with on this day?"

[2047] In addition, an emotion engine is used to recognize the user's emotions, and questions that are more in line with their emotions are added, such as, "You seem to be having fun. Tell me more about what happened that day."

[2048] Prompt Sentence Examples

[2049] "Please upload photos of your market so we can train you on your business."

[2050] "What is in the uploaded photo? For example, what are the names of machine parts or devices?"

[2051] This system allows users to enjoy past memories while stimulating and improving their memory. The introduction of an emotion engine makes interactions more personalized, improving the user experience.

[2052] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2053] Step 1:

[2054] The device allows the user to select past photos and upload them through the application (input: photo data selected by the user, output: photo data sent to the server). The device displays the selected photos as a preview and provides an upload button. When the user clicks the upload button, the photo data is sent to the server.

[2055] Step 2:

[2056] The server temporarily stores the received photo data in storage and launches an image analysis engine (such as OpenCV or TensorFlow) (input: photo data, output: photo data stored in storage). This engine uses object recognition technology (such as YOLOv3 or OpenCV) to identify objects in the photo.

[2057] Step 3:

[2058] The server analyzes the attribute information of the photo (e.g., EXIF ​​data) and collects the attribute information (input: photo data saved in storage, output: attribute information). This allows the server to identify when and where the photo was taken.

[2059] Step 4:

[2060] The server generates past event information based on the analyzed attribute information (input: attribute information, output: past event information). For example, it generates a specific episode such as "In the summer of 2005, we went on a family trip to Shonan Beach."

[2061] Step 5:

[2062] The server creates a dialogue scenario based on the generated past event information (input: past event information, output: dialogue scenario). Furthermore, it uses an emotion engine that analyzes voice and text input to recognize the user's emotions and adjust the dialogue scenario accordingly.

[2063] Step 6:

[2064] The server creates questions and feedback based on the generated dialogue scenario and emotional information and sends them to the terminal (input: dialogue scenario, emotional information; output: questions and feedback sent to the terminal).

[2065] Step 7:

[2066] The terminal presents the dialogue scenario sent from the server to the user and starts the dialogue (input: questions and feedback sent from the server, output: user's response). The user responds to the questions displayed on the screen by text input or voice input.

[2067] Step 8:

[2068] The terminal records the user's response and transmits it to the server in real time (input: user's response data, output: real-time response data transmitted to the server).

[2069] Step 9:

[2070] The server analyzes the user's response data and generates the next question and feedback using natural language processing technology and an emotion engine (input: user's real-time response data, output: next question and feedback).

[2071] Step 10:

[2072] The server collects dialogue logs and evaluates the user's skills and knowledge (input: dialogue log data, output: user skill evaluation data). Based on this, individual feedback is generated and provided to the user via their device.

[2073] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2074] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2075] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2076] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2077] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2078] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2079] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2080] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2081] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2082] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2083] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2084] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2085] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2086] Alternatively, the specific processing program...

Claims

1. means for receiving photo data from a user's terminal; A means for analyzing the received photo data using an image analysis engine and identifying people and objects in the photo; A means for collecting metadata of the photos and generating past memory information based on the analysis results; A means for creating a dialogue scenario based on the generated memory information; A system including a means for conducting a dialogue with a user based on a dialogue scenario.

2. 10. The system of claim 1, further comprising means for monitoring the user's memory impairment status and providing interaction feedback.

3. 10. The system of claim 1, further comprising means for identifying people in photographs using facial recognition technology.

Images