system

Abstract

Provide a system. 【Solution means】 In the terminal, means for the user to select past visual information stored in the storage device, means for analyzing the feature information of the selected visual information, means for generating an inquiry based on the analyzed feature information, means for presenting the generated inquiry to the user and receiving the user's response, means for analyzing the user's response, generating an additional inquiry based on the response, and presenting it to the user again, means for storing the conversation with the user in the recording device, means characterized in that the terminal is a portable information terminal, means for automatically generating a natural language form inquiry using an AI model generated based on the analyzed visual information, means for analyzing the user's response by natural language processing technology and automatically generating related additional inquiries, A system including.

Description

Technical Field

[0001] The technology of this disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, with the increase in the number of elderly people living alone, delaying the onset and progression of dementia has become an important issue. In particular, the decrease in the opportunity for conversation in daily life is regarded as one of the factors leading to the decline of cognitive function. However, in reality, there is a problem that it is not easy for elderly people to find someone to talk to on a daily basis. For this reason, a mechanism that can utilize past memories and stimulate the cognitive function of elderly people through natural conversation is required.

Means for Solving the Problems

[0005] To address the aforementioned problems, this invention provides an interactive system that utilizes stored past images, aiming to meet the needs of elderly people regarding a lack of conversation partners and dementia prevention. This system allows users to select images from a storage device using a terminal, analyzes the characteristic information of the selected image, and generates and presents questions to the elderly person based on that analysis. Furthermore, it has the function of receiving responses from the elderly person and analyzing those responses to naturally continue the conversation. Through this series of dialogue processes, the system aims to maintain and improve cognitive function by providing an environment in which elderly people can utilize their own memories and engage in daily conversations.

[0006] A "terminal" is an electronic device that users operate to access and use this system.

[0007] A "storage device" is a storage medium or database used to store saved data or images.

[0008] "Image" refers to photographs and paintings that contain visual information stored as digital data.

[0009] "Users" refers to individuals who operate this system, and is particularly intended for elderly people.

[0010] "Feature information" refers to data about the content and characteristics of an image, and includes information that contains the elements to be analyzed.

[0011] "Analyzing" refers to the process of deriving meaning or patterns based on specific information or data.

[0012] "Generating questions" means creating questions in natural language to facilitate interaction with users.

[0013] "To present" refers to the act of conveying information to a user through display or audio.

[0014] A "response" is an opinion or answer that a user gives in response to a question presented to them.

[0015] A "recording device" is a medium or system for storing conversations and data with users.

Brief Explanation of Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention is an interactive system aimed at preventing dementia in the elderly, and consists of a terminal, a server, and a storage device. The embodiments thereof are described in detail below.

[0038] First, the user accesses the system using a terminal. The user can then select past images stored in the device's interface. These selected images are used as triggers to evoke the user's past experiences and memories.

[0039] The server is responsible for analyzing image data. It obtains characteristic information about the image using metadata associated with the image and image analysis techniques. For example, if a specific landmark or person is depicted in the image, that information is identified.

[0040] Next, the server automatically generates questions to present to the user based on this analyzed information. Natural language processing technology is used for this question generation, and the questions are output in a format that is easy for the user to answer naturally.

[0041] The device presents the generated questions to the user in either audio or text format. The user answers the questions, initiating a dialogue. The user's responses are then analyzed by the server, generating additional relevant questions or comments. This allows the dialogue to continue naturally and provides the user with opportunities to activate their memory.

[0042] Finally, the server saves the content of the interaction with the user to a recording device. This record allows the user to review past interactions at a later date, which is expected to facilitate long-term learning and memory enhancement.

[0043] For example, when a user selects a travel photo, the server generates questions based on the location and time of the photo, such as "Who did you visit this place with?" or "What was the most enjoyable part of this trip?", deepening the conversation with the user. In this way, the present invention contributes to elderly people reliving past experiences and stimulating their brains through conversation.

[0044] The following describes the processing flow.

[0045] Step 1:

[0046] The user accesses the photo storage service using their device. Through the interface, the user views a list of past images stored on the storage device and selects a specific image they wish to reminisce about.

[0047] Step 2:

[0048] The selected image is sent from the terminal to the server. The server uses metadata associated with the image (e.g., date and time of capture, location) and image analysis techniques to analyze the image's characteristic information (e.g., specific landmarks or people).

[0049] Step 3:

[0050] The server generates questions related to the image based on the analysis results. The question generation engine uses natural language processing technology to prepare multiple questions in a format that is easy for the user to answer.

[0051] Step 4:

[0052] The terminal presents the user with a question sent from the server, either verbally or as text. The user responds to the question from the terminal using either verbal or text input.

[0053] Step 5:

[0054] The user's response is sent from the terminal to the server. The server uses speech recognition and natural language processing technologies to analyze the response and generate additional questions or comments related to its content.

[0055] Step 6:

[0056] The device presents the user with additional questions or comments, continuing the conversation. By answering the questions, the user can share more detailed memories.

[0057] Step 7:

[0058] The server saves the content of the conversation with the user as a log to a recording device. The conversation log is made available for the user to review at a later date.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] In preventing dementia in the elderly, it is necessary to smoothly carry out a series of processes that effectively recall past memories and activate the brain through dialogue. However, existing systems have difficulty naturally continuing dialogue with users, and the activation of memories through dialogue is not currently sufficient.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes means for a user to select past visual data stored in a storage medium on an information terminal, means for an information processing device that analyzes the characteristic information of the selected visual data, and means for generating questions using a generative artificial intelligence model. This makes it possible to effectively evoke memories using visual data as a trigger and to naturally continue the dialogue with the user.

[0064] An "information terminal" is an electronic device that allows users to perform operations such as selecting visual data or responding to questions via an interface.

[0065] A "storage medium" is a recording device used to store data such as visual data and dialogue content.

[0066] "Visual data" refers to past images and photographs used to evoke past experiences and memories.

[0067] An "information processing device" is a computer used to analyze visual data and extract feature information.

[0068] "Feature information" refers to identifiable information such as landmarks and people contained in visual data.

[0069] A "generative artificial intelligence model" is a machine learning-based model used to automatically generate questions for users.

[0070] "Natural language processing technology" is a computer language processing technology used to analyze user responses and generate natural-sounding dialogue.

[0071] Modes for carrying out the invention

[0072] This invention is an interactive system aimed at preventing dementia in the elderly, and consists of an information terminal, a server, and a storage medium. A specific example of this system is described below.

[0073] Users access the system using an information terminal. This terminal is a general-purpose computer such as a tablet or smartphone, and it provides an interface for selecting past visual data (images and photographs) stored on a storage medium. This selected visual data is used as a trigger to evoke the user's past experiences and memories.

[0074] The server is responsible for analyzing visual data. Using visual data analysis technologies such as OpenCV and visual data recognition APIs, the server extracts feature information from image data. This feature information includes identifiable elements such as landmarks and people.

[0075] Next, the server uses a generative AI model to automatically generate questions to present to the user based on the extracted feature information. This utilizes natural language processing technology to output the questions in a format that is easy for the user to answer. For example, possible generated prompts might be "Who did you visit this place with?" or "What was the most memorable event of this trip?"

[0076] The terminal presents the user with generated questions, and the user responds. The server analyzes the user's responses and generates additional questions or comments based on them, continuing the dialogue. This process helps to evoke the user's memory and enables a more natural and richer conversation.

[0077] For example, if a user selects a photo from a family trip, the server will generate questions such as, "Who are the people in this photo?" or "What was the most enjoyable part of this trip?" This stimulates the user's memory and contributes to brain health.

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] The user accesses the system using an information terminal and selects past visual data from a storage medium. The input is the visual data selected by the user, and the output is the selected image data. Specifically, the terminal displays a user interface and presents the user with a list of images. The user selects a specific photograph while browsing using touch controls.

[0081] Step 2:

[0082] The server analyzes the selected image data received from the user. The input for this step is the selected image data, and the output is feature information about that image. The server uses visual data analysis techniques (e.g., OpenCV) to extract features such as landmarks and people. Specifically, the server inputs the image data into a machine learning model to analyze geometric shapes, colors, and structural features.

[0083] Step 3:

[0084] The server automatically generates questions based on extracted feature information. The input is the feature information, and the output is the generated question. The server uses a generative AI model to create questions in a natural language format. Specifically, the server inputs the feature information as a prompt into the generative AI model, resulting in a question such as, "Who did you visit this place with?"

[0085] Step 4:

[0086] The terminal presents the user with a question received from the server. The input is the generated question, and the output is the user's response. Specifically, the terminal displays the question in audio or text format and collects the user's answer as audio or text input.

[0087] Step 5:

[0088] The server analyzes the user's response and generates additional questions. Here, the input is the user's response, and the output is the additional questions. The server uses natural language processing techniques to analyze the user's intent and information to derive relevant questions. Specifically, the server runs an algorithm that analyzes the user's answer and generates new questions related to that response.

[0089] Step 6:

[0090] The server saves the entire dialogue session to a storage medium. The input is the dialogue log data, and the output is the saved data. Specifically, the server records the dialogue content as structured data in a database and manages it so that the user can access it later.

[0091] (Application Example 1)

[0092] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0093] The problem that this invention aims to solve is to provide an effective conversational support system that contributes to maintaining and improving the cognitive function of the elderly. This problem stems from the limited opportunities to recall memories in daily life and the lack of a constant companion to activate memories through conversation. In particular, there is a need for technology that generates natural conversations based on visual information.

[0094] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0095] In this invention, the server includes means for analyzing past visual information stored in a memory device to obtain feature information, means for automatically generating queries in natural language form using a generative AI model based on the analyzed feature information, and means for analyzing the generated questions and the user's responses and automatically generating additional related queries using natural language processing technology. This enables the user to activate their memory and maintain cognitive function through interactive dialogue triggered by their past visual information.

[0096] A "terminal" is a portable information device used by a user to access storage and select past visual information.

[0097] A "memory device" is a data storage unit that stores past visual information that the user can select.

[0098] "Visual information" refers to image data used to evoke past experiences and memories of the user.

[0099] "Feature information" refers to identifying information such as landmarks and people extracted from analyzed visual information.

[0100] A "generative AI model" is an artificial intelligence technology that automatically generates queries in natural language format based on feature information.

[0101] An "inquiry" is a question in natural language format that is generated by a generative AI model based on visual information and presented to the user.

[0102] "Natural language processing technology" refers to language analysis techniques used to analyze user responses and generate related additional queries.

[0103] A "conversation" refers to the exchange between generated inquiries and user responses, and is the process of interaction that is stored in a recording device.

[0104] This invention is a system that activates cognitive functions in elderly individuals by enabling them to engage in dialogue based on their past visual information. This system consists of a portable information device, a server, and a storage device.

[0105] The user accesses the system using a terminal and selects past visual information stored in the storage device. The selected visual information is sent from the terminal to the server. The server receives the visual information and uses image analysis technology, such as Google® Cloud Vision API, to obtain feature information (landmarks, people, etc.).

[0106] Based on the analyzed feature information, the server automatically generates queries in natural language format using generative AI models such as OpenAI's GPT model. These queries are then presented to the user via the terminal. The user responds to the presented queries.

[0107] User responses are analyzed by a server using speech recognition and natural language processing technologies such as Amazon Transcribe. Based on the analysis results, the server generates relevant additional queries and presents them to the user again. This entire process enables continuous interaction between the user and the system, effectively retrieving the user's memory.

[0108] For example, if a user selects a past "family trip photo," the server generates questions such as "Where was this photo taken?" or "What was the most memorable event from this trip?" based on the feature information extracted from the photo.

[0109] An example of a prompt for a generative AI model is: "Image analysis result: Landmark 'Historical building', Person 'Relative'. Question generation: Generate a question that will help the user recall memories." Such prompts allow the system to provide a dialogue that makes it easy for the user to recall past experiences.

[0110] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0111] Step 1:

[0112] The user accesses the system using a portable information device and selects past visual information stored in its memory. The input is the user's selection, and the output is the selected visual information. Based on the user's selection, the terminal sends the corresponding visual information to the server.

[0113] Step 2:

[0114] The server uses image analysis technology to extract feature information from the received visual information. The input is selected visual information, and the output is feature information such as landmarks and people in the image. In this process, the server analyzes the feature information using the Google Cloud Vision API, among others.

[0115] Step 3:

[0116] The server automatically generates natural language queries using a generative AI model based on the acquired feature information. The input is the feature information, and the output is the generated query. The server generates the question using OpenAI's GPT model and sends it to the terminal.

[0117] Step 4:

[0118] The terminal presents a generated query to the user. The user responds to it. The input is the generated query, and the output is the user's response. The terminal presents the query in voice or text format and receives the user's response.

[0119] Step 5:

[0120] The server analyzes the user's response and generates additional relevant queries using natural language processing techniques. The input is the user's response, and the output is the additional queries. The server converts speech to text using tools like Amazon Transcribe and then applies natural language processing.

[0121] Step 6:

[0122] The terminal then presents the user with additional queries to facilitate continued interaction. The input is the additional queries, and the output is the continuation of the interaction. The terminal saves this interaction to a recording device, which the user can review later.

[0123] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0124] This invention is a system that combines an emotion engine with image analysis technology and natural language processing technology, with the aim of stimulating the cognitive functions of the elderly and enriching their conversations.

[0125] The user first operates the device to select images from their memory to recall past experiences. This image selection serves as a means of stimulating the user's memories and emotions.

[0126] The server receives the selected image and extracts its feature information using image analysis technology. This feature information includes the date and time the image was taken, the location, and information about objects and people in the image. Based on this, the server uses natural language processing technology to generate questions that are easy for the user to answer naturally and presents them to the user through the terminal.

[0127] When a user responds to a presented question via voice or text, the device sends the response to the server. The emotion engine on the server analyzes the tone and content of the user's voice to recognize the user's emotional state. This recognition of emotional state is used, for example, to determine whether the user is feeling nostalgic or amused after viewing an image.

[0128] The emotion data obtained by the emotion engine is used by the server to generate further questions and comments. In this process, the tone and content of the dialogue are adjusted according to the user's emotions. For example, if the user is feeling happy, more positive questions can be asked.

[0129] Furthermore, the user's responses and emotional changes are recorded by a recording device. This recording is used by the user to review their emotional changes and conversation history at a later date.

[0130] For example, if a user selects a photo of a fun family gathering, the server will generate questions such as "How did you feel about this gathering?" based on the facial recognition results obtained from the photo, providing a dialogue that elicits positive emotions from the user.

[0131] Thus, the present invention is a system that effectively promotes the activation of cognitive functions by retrieving the user's past memories through images and adjusting responses using an emotion engine.

[0132] The following describes the processing flow.

[0133] Step 1:

[0134] The user displays a list of images saved from their device's storage and selects a specific image. Through this image selection, the user attempts to reminisce about their past memories.

[0135] Step 2:

[0136] The device sends the selected image to the server. The server uses image analysis techniques to extract feature information from the image. This includes information such as face recognition, landmarks, and date and time within the image. Using this analysis result, the server prepares to generate appropriate questions relevant to the user.

[0137] Step 3:

[0138] The server generates appropriate questions using natural language processing techniques based on the extracted feature information. The generated questions are crafted in a format that makes it easy for the user to answer naturally. These questions are sent to the device, which then presents them to the user via voice or text.

[0139] Step 4:

[0140] The user responds to questions presented by the device using voice or text. The device sends this response to the server in real time. In the case of a voice response, it is transmitted as audio data.

[0141] Step 5:

[0142] The server analyzes the response received from the user using an emotion engine. The emotion engine recognizes the user's emotional state based on the tone of voice and the content of what is being said. For example, if the user's voice is cheerful, it detects a positive emotion.

[0143] Step 6:

[0144] The server considers the recognized emotional state and generates the next questions and comments. At this stage, adjustments are made according to the user's emotions. If the emotional state is positive, questions and comments are generated to continue the positive conversation. The generated content is then sent back to the device.

[0145] Step 7:

[0146] The device presents the user with newly generated questions and comments, facilitating the smooth continuation of the dialogue. The user responds, and the dialogue cycle continues.

[0147] Step 8:

[0148] The server saves data on the user's responses and emotional changes to a recording device. This allows the user to review the conversation history later and reflect on the changes in their emotions.

[0149] (Example 2)

[0150] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0151] To improve cognitive function in older adults, a system is needed that stimulates past memories and guides conversations according to emotions. However, conventional systems have shortcomings in recognizing emotions and generating appropriate dialogues, making it difficult to provide natural and effective cognitive stimulation for older adults.

[0152] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0153] In this invention, the server includes means for the user to select past images stored in information memory, means for analyzing the features of the selected images using image analysis technology, means for generating questions based on the analyzed features using natural language processing technology, and means for using emotion analysis technology to recognize the emotional state based on the generated questions. This enables the provision of interactive dialogue tailored to the user's emotions and memories, and activates the user's cognitive functions.

[0154] An "information processing device" refers to any electronic device used for inputting, processing, storing, and outputting data, which can be operated by the user through an interface.

[0155] "Information storage" refers to data storage that stores digital information and allows it to be retrieved as needed.

[0156] "Image analysis technology" refers to all technologies used to extract features and patterns from digital images and perform classification and recognition.

[0157] "Natural language processing technology" refers to the technology that enables computers to understand human language and process it automatically.

[0158] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate natural language sentences from input data.

[0159] "Emotional analysis technology" refers to technology that automatically analyzes and classifies a speaker's emotional state from audio and text information.

[0160] "Information recording" refers to the process of saving conversations and data so that they can be referenced later.

[0161] The following configuration and procedure are necessary to implement the invention. This invention is a system for stimulating the cognitive functions of the elderly and enriching their conversations, and its details are described below.

[0162] First, the user uses the terminal to select past images stored in the information memory. The terminal is equipped with input devices such as a touchscreen, keyboard, and mouse, and the user can operate it through the interface. The selected images are saved in standard formats such as JPEG and PNG.

[0163] Next, the terminal sends the selected image data to the server. The server uses image analysis technology to extract features from this image. This analysis uses AI-based image processing libraries (e.g., OpenCV, TENSORFLOW®, etc.). The feature information includes the date and time of capture, location, and information about objects and people in the image.

[0164] The server uses a generative AI model based on the extracted feature information and applies natural language processing techniques (e.g., the GPT model) to generate questions for the user. These questions are then presented to the user via the terminal.

[0165] For example, if a user selects a photo from a family trip, the server will generate a question such as, "Tell us about the most enjoyable moment from your family trip," based on the information obtained from that image. This prompt helps the user recall their memories.

[0166] The user responds to the generated questions using voice or text. The terminal sends the response to the server. The server uses sentiment analysis technology (e.g., IBM Watson® Tone Analyzer) to analyze the user's tone of voice and text content to recognize the user's emotional state.

[0167] To provide symmetrical dialogue that reflects emotions, the server can adjust the dialogue based on the user's emotions and generate additional questions or comments. This allows the system to provide a valuable interactive experience for the user and support the activation of cognitive functions.

[0168] Furthermore, the history of the conversation and data on the user's emotional changes are stored in the information log, which the user can use to look back on it at a later date.

[0169] Thus, the present invention provides a system that supports the cognitive function of the elderly by combining image analysis technology, natural language processing technology, and sentiment analysis technology.

[0170] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0171] Step 1:

[0172] The user operates the device to select past images stored in the information memory. This selection is made via touch or mouse clicks, and the input image data is displayed on the device's screen. The images selected by the user are used as basic data for subsequent processing.

[0173] Step 2:

[0174] The terminal sends the selected image data to the server. The images are in formats such as JPEG or PNG, and this becomes the input data for the server's analysis system. After receiving the data, the server prepares to pass the images to the analysis tool.

[0175] Step 3:

[0176] The server extracts features from the received image data using image analysis techniques. Specifically, AI algorithms (e.g., OpenCV, TensorFlow) are applied to obtain information such as the date and time of capture, location, and objects or people within the image. This feature data forms the basis for the next questions generated.

[0177] Step 4:

[0178] The server uses natural language processing techniques with feature information obtained from image analysis as input. It utilizes a generative AI model (e.g., GPT model) to generate questions for the user. During this process, the output is adjusted to ensure it is easy for the user to answer. The generated questions are then sent to the terminal.

[0179] Step 5:

[0180] The terminal presents the user with a question sent from the server. The question is displayed as text on the screen and may be read aloud if necessary. At this stage, the user receives the question on the screen.

[0181] Step 6:

[0182] The user responds to the presented questions either verbally or in text. This response is recorded as input data on the device and, if verbal, is converted to text using speech recognition technology. This text data forms the basis for the next analysis step.

[0183] Step 7:

[0184] The terminal sends the user's response text to the server. Upon receiving this transmitted data, the server prepares to perform the next analysis.

[0185] Step 8:

[0186] The server applies sentiment analysis technology to analyze the emotional state of the user's responses. This analysis considers tone information obtained from the speech and the content of the text. The analysis results become important input data for generating the next dialogue.

[0187] Step 9:

[0188] The server uses the results of the emotional state analysis as input and generates adaptive additional questions and comments using a generative AI model. In this step, the generated content is adjusted according to the user's emotional state. The results are sent to the terminal, and the interactive dialogue with the user continues.

[0189] (Application Example 2)

[0190] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0191] Cognitive decline in older adults is often exacerbated by a lack of interpersonal interaction and stimulation of memories based on past experiences. Furthermore, eliciting emotions through conversation is difficult, and providing dialogue that responds to those emotions is even more challenging. Current technology does not adequately provide solutions that can appropriately understand a user's emotional state and provide dialogue accordingly. Therefore, there is a need for new methods that effectively stimulate cognitive function and richly evoke past memories and emotions.

[0192] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0193] In this invention, the server includes means for the user to select past images stored in a memory device, means for analyzing the characteristic information of the selected images, and means for adjusting the dialogue content based on the analyzed emotions. This makes it possible to stimulate the past memories of elderly people, understand their emotions, and provide more appropriate and enriching dialogue.

[0194] A "terminal" is a device used by a user to operate, and is equipped with storage and communication functions.

[0195] A "storage device" is a device for holding data, and is a medium for storing information such as images and text.

[0196] "Image analysis" is a technique for extracting feature information from selected images, and is the process of recognizing objects and photographic information within an image.

[0197] "Feature information" refers to data obtained through image analysis, including information about people, objects, time, and location within the image.

[0198] "Question generation" is the process of creating questions to present to users based on analyzed feature information.

[0199] "User response" refers to the user's reply to a presented question, which is information sent to the server as audio or text.

[0200] "Emotional analysis" is a technology that evaluates a user's emotional state based on their responses and tone of voice.

[0201] "Dialogue adjustment" is the process of dynamically changing the tone and content of a dialogue based on analyzed emotions.

[0202] "Providing interaction" is the process of offering additional dialogue and feedback tailored to the user's emotional state.

[0203] A "recording device" is a device for saving the content of conversations with users and is a medium for making past logs accessible.

[0204] The system that realizes this invention includes several main components. First, the terminal allows the user to select an image from its storage device to reminisce about past memories. The selected image is sent to a server, which uses image analysis technology to extract detailed feature information from the image. This feature information includes people and objects in the image, the date and time the image was taken, and the location where it was taken.

[0205] The server then uses natural language processing techniques based on the extracted feature information to generate questions that are easy for the user to answer. The generated questions are presented to the user through the terminal, and the user's voice or text response is sent back to the server.

[0206] Sentiment analysis is performed by an emotion engine based on the user's responses and tone of voice. This evaluates the user's emotional state, and the tone and content of the conversation are adjusted accordingly. For example, if the user is enjoying looking at an image, more positive questions and comments will be provided.

[0207] Furthermore, past conversations and emotional changes are saved in the recording device, allowing users to later review their emotional progression and conversation history. This supports the long-term activation of cognitive functions.

[0208] As a concrete example, when a user selects a family photo, the server asks a question such as, "What is the most memorable event from this photo?" The emotion engine analyzes the user's cheerful response and then asks follow-up questions such as, "Could you tell us more about that memory?"

[0209] An example of a prompt for a generative AI model is the question, "Output questions to guide the conversation about related events and emotions so that the model can describe the user's emotions associated with the image."

[0210] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0211] Step 1:

[0212] The user operates the device and selects a past image from its storage. The input is the user's instructions, and the output is the selected image. The user's action is to tap the image through the device's interface.

[0213] Step 2:

[0214] The terminal sends the selected image to the server. The input is the selected image, and the output is the completion of the transfer of the image data to the server. The terminal uploads the image data to the server using the network.

[0215] Step 3:

[0216] The server extracts feature information from received images using image analysis techniques. The input is image data, and the output is feature information (people, objects, date and time, location, etc.). The server applies image processing algorithms to identify important elements within the image.

[0217] Step 4:

[0218] The server analyzes the extracted feature information and uses natural language processing techniques to generate questions that are easy for the user to answer. The input is feature information, and the output is the generated questions. The server uses a generative AI model to construct highly relevant questions.

[0219] Step 5:

[0220] The generated question is sent to the device and presented to the user. The input is the generated question, and the output is the display of the question to the user. The device displays the question in the UI and prompts the user for a voice or text response.

[0221] Step 6:

[0222] The user responds to questions using voice or text. Input is the user's response, and output is the response data input to the terminal. The user inputs their answers using a microphone or keyboard.

[0223] Step 7:

[0224] The terminal sends the user's response to the server. The input is the response data from the user, and the output is the completion of data transmission to the server. The terminal processes the response data and forwards it to the server.

[0225] Step 8:

[0226] The server analyzes the user's response using sentiment analysis technology to recognize their emotional state. The input is the response data, and the output is the recognized emotional state. The server evaluates the text and tone of the response and uses an emotion engine to identify the emotion.

[0227] Step 9:

[0228] The server adjusts the dialogue content based on the recognized emotional state and generates new questions and comments. The input is the emotional state, and the output is the content of the next dialogue. The server then uses the generative AI model again to design an appropriate dialogue.

[0229] Step 10:

[0230] The refined dialogue is sent to the terminal to support further interaction with the user. The input is the newly generated dialogue, and the output is what is presented to the user. The terminal provides the user with a continuous conversation, facilitating communication.

[0231] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0232] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0233] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0234] [Second Embodiment]

[0235] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0236] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0237] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0238] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0239] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0240] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0241] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0242] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0243] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0244] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0245] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0246] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0247] This invention is an interactive system aimed at preventing dementia in the elderly, and consists of a terminal, a server, and a storage device. The embodiments thereof are described in detail below.

[0248] First, the user accesses the system using a terminal. The user can then select past images stored in the device's interface. These selected images are used as triggers to evoke the user's past experiences and memories.

[0249] The server is responsible for analyzing image data. It obtains characteristic information about the image using metadata associated with the image and image analysis techniques. For example, if a specific landmark or person is depicted in the image, that information is identified.

[0250] Next, the server automatically generates questions to present to the user based on this analyzed information. Natural language processing technology is used for this question generation, and the questions are output in a format that is easy for the user to answer naturally.

[0251] The device presents the generated questions to the user in either audio or text format. The user answers the questions, initiating a dialogue. The user's responses are then analyzed by the server, generating additional relevant questions or comments. This allows the dialogue to continue naturally and provides the user with opportunities to activate their memory.

[0252] Finally, the server saves the content of the interaction with the user to a recording device. This record allows the user to review past interactions at a later date, which is expected to facilitate long-term learning and memory enhancement.

[0253] For example, when a user selects a travel photo, the server generates questions based on the location and time of the photo, such as "Who did you visit this place with?" or "What was the most enjoyable part of this trip?", deepening the conversation with the user. In this way, the present invention contributes to elderly people reliving past experiences and stimulating their brains through conversation.

[0254] The following describes the processing flow.

[0255] Step 1:

[0256] The user accesses the photo storage service using their device. Through the interface, the user views a list of past images stored on the storage device and selects a specific image they wish to reminisce about.

[0257] Step 2:

[0258] The selected image is sent from the terminal to the server. The server uses metadata associated with the image (e.g., date and time of capture, location) and image analysis techniques to analyze the image's characteristic information (e.g., specific landmarks or people).

[0259] Step 3:

[0260] The server generates questions related to the image based on the analysis results. The question generation engine uses natural language processing technology to prepare multiple questions in a format that is easy for the user to answer.

[0261] Step 4:

[0262] The terminal presents the user with a question sent from the server, either verbally or as text. The user responds to the question from the terminal using either verbal or text input.

[0263] Step 5:

[0264] The user's response is sent from the terminal to the server. The server uses speech recognition and natural language processing technologies to analyze the response and generate additional questions or comments related to its content.

[0265] Step 6:

[0266] The device presents the user with additional questions or comments, continuing the conversation. By answering the questions, the user can share more detailed memories.

[0267] Step 7:

[0268] The server saves the content of the conversation with the user as a log to a recording device. The conversation log is made available for the user to review at a later date.

[0269] (Example 1)

[0270] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0271] In preventing dementia in the elderly, it is necessary to smoothly carry out a series of processes that effectively recall past memories and activate the brain through dialogue. However, existing systems have difficulty naturally continuing dialogue with users, and the activation of memories through dialogue is not currently sufficient.

[0272] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0273] In this invention, the server includes means for a user to select past visual data stored in a storage medium on an information terminal, means for an information processing device that analyzes the characteristic information of the selected visual data, and means for generating questions using a generative artificial intelligence model. This makes it possible to effectively evoke memories using visual data as a trigger and to naturally continue the dialogue with the user.

[0274] An "information terminal" is an electronic device that allows users to perform operations such as selecting visual data or responding to questions via an interface.

[0275] A "storage medium" is a recording device used to store data such as visual data and dialogue content.

[0276] "Visual data" refers to past images and photographs used to evoke past experiences and memories.

[0277] An "information processing device" is a computer used to analyze visual data and extract feature information.

[0278] "Feature information" refers to identifiable information such as landmarks and people contained in visual data.

[0279] The "generative AI model" is a machine learning-based model used to automatically generate questions for users.

[0280] The "natural language processing technology" is a computer language processing technology used to analyze users' responses and generate natural conversations.

[0281] Mode for Carrying Out the Invention

[0282] The present invention is an interactive system aimed at preventing dementia in the elderly and is composed of an information terminal, a server, and a storage medium. Specific embodiments of this system will be described below.

[0283] The user accesses the system using an information terminal. As the information terminal, general-purpose computers such as tablets and smartphones are used, and an interface is provided for selecting past visual data (images and photos) stored in the storage medium. This selected visual data is used as a trigger to evoke the user's past experiences and memories.

[0284] The server is responsible for analyzing visual data. The server uses visual data analysis technologies such as OpenCV and visual data recognition APIs to extract feature information from image data. This feature information includes identifiable elements such as landmarks and people.

[0285] Next, the server uses the generative AI model to automatically generate questions to present to the user based on the extracted feature information. This makes full use of natural language processing technology and is output in a format that is easy for the user to answer. For example, generated prompts such as "Who did you visit this place with?" and "What is the event that particularly impressed you during this trip?" can be considered.

[0286] The terminal presents the generated questions to the user, and the user responds to them. The server analyzes the user's responses, generates additional questions or comments based on them, and continues the conversation. This process evokes the user's memory and enables a more natural and rich conversation.

[0287] As a specific example, when the user selects a photo of a family trip, the server generates questions such as "Who are the people in this photo?" or "What was the most enjoyable thing about this trip?" This activates the user's memory and contributes to brain health.

[0288] The flow of the specific process in Example 1 will be described using FIG. 11.

[0289] Step 1:

[0290] The user accesses the system using an information terminal and selects past visual data from a storage medium. The input is the visual data selected by the user, and the output is the selected image data. As a specific operation, the terminal displays a user interface and presents an image list to the user. The user selects a specific photo while browsing by touch operation.

[0291] Step 2:

[0292] The server analyzes the selected image data received from the user. The input for this step is the selected image data, and the output is the feature information regarding that image. The server uses visual data analysis technology (e.g., OpenCV) to extract features such as landmarks and people. As a specific operation, the server inputs the image data into a machine learning model and analyzes geometric shapes, colors, and structural features.

[0293] Step 3:

[0294] The server automatically generates questions based on extracted feature information. The input is the feature information, and the output is the generated question. The server uses a generative AI model to create questions in a natural language format. Specifically, the server inputs the feature information as a prompt into the generative AI model, resulting in a question such as, "Who did you visit this place with?"

[0295] Step 4:

[0296] The terminal presents the user with a question received from the server. The input is the generated question, and the output is the user's response. Specifically, the terminal displays the question in audio or text format and collects the user's answer as audio or text input.

[0297] Step 5:

[0298] The server analyzes the user's response and generates additional questions. Here, the input is the user's response, and the output is the additional questions. The server uses natural language processing techniques to analyze the user's intent and information to derive relevant questions. Specifically, the server runs an algorithm that analyzes the user's answer and generates new questions related to that response.

[0299] Step 6:

[0300] The server saves the entire dialogue session to a storage medium. The input is the dialogue log data, and the output is the saved data. Specifically, the server records the dialogue content as structured data in a database and manages it so that the user can access it later.

[0301] (Application Example 1)

[0302] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0303] The problem to be solved by the present invention is to provide an effective dialogue-form support system for maintaining and improving the cognitive function of the elderly. This problem is caused by the limited opportunities to evoke memory conversations in daily life and the inability to always obtain a companion for memory activation through dialogue. In particular, there is a need for a technology that generates natural conversations based on visual information.

[0304] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0305] In this invention, the server includes means for analyzing past visual information stored in a storage device to obtain feature information, means for automatically generating a natural language form of inquiry using a generated AI model based on the analyzed feature information, and means for analyzing the generated question and the user's response and automatically generating related additional inquiries using natural language processing technology. As a result, the user can activate memory and maintain cognitive function through an interactive dialogue triggered by their own past visual information.

[0306] The "terminal" is a portable information device used by the user to access the storage device and select past visual information.

[0307] The "storage device" is a data storage unit that stores past visual information that can be selected by the user.

[0308] The "visual information" is image data used to evoke the user's past experiences and memories.

[0309] The "feature information" is identification information such as landmarks and people extracted from the analyzed visual information.

[0310] The "generated AI model" is an artificial intelligence technology that automatically generates a natural language form of inquiry based on feature information.

[0311] An "inquiry" is a question in natural language format that is generated by a generative AI model based on visual information and presented to the user.

[0312] "Natural language processing technology" refers to language analysis techniques used to analyze user responses and generate related additional queries.

[0313] A "conversation" refers to the exchange between generated inquiries and user responses, and is the process of interaction that is stored in a recording device.

[0314] This invention is a system that activates cognitive functions in elderly individuals by enabling them to engage in dialogue based on their past visual information. This system consists of a portable information device, a server, and a storage device.

[0315] The user accesses the system using a terminal and selects past visual information stored in the storage device. The selected visual information is sent from the terminal to the server. The server receives the visual information and uses image analysis technology, such as the Google Cloud Vision API, to obtain feature information (landmarks, people, etc.).

[0316] Based on the analyzed feature information, the server automatically generates natural language queries using generative AI models such as OpenAI's GPT model. These queries are then presented to the user via the terminal. The user responds to the presented queries.

[0317] User responses are analyzed by a server using speech recognition and natural language processing technologies such as Amazon Transcribe. Based on the analysis results, the server generates relevant additional queries and presents them to the user again. This entire process enables continuous interaction between the user and the system, effectively retrieving the user's memory.

[0318] For example, if a user selects a past "family trip photo," the server generates questions such as "Where was this photo taken?" or "What was the most memorable event from this trip?" based on the feature information extracted from the photo.

[0319] An example of a prompt for a generative AI model is: "Image analysis result: Landmark 'Historical building', Person 'Relative'. Question generation: Generate a question that will help the user recall memories." Such prompts allow the system to provide a dialogue that makes it easy for the user to recall past experiences.

[0320] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0321] Step 1:

[0322] The user accesses the system using a portable information device and selects past visual information stored in its memory. The input is the user's selection, and the output is the selected visual information. Based on the user's selection, the terminal sends the corresponding visual information to the server.

[0323] Step 2:

[0324] The server uses image analysis technology to extract feature information from the received visual information. The input is selected visual information, and the output is feature information such as landmarks and people in the image. In this process, the server analyzes the feature information using the Google Cloud Vision API, among others.

[0325] Step 3:

[0326] The server automatically generates natural language queries using a generative AI model based on the acquired feature information. The input is the feature information, and the output is the generated query. The server generates the question using OpenAI's GPT model and sends it to the terminal.

[0327] Step 4:

[0328] The terminal presents a generated query to the user. The user responds to it. The input is the generated query, and the output is the user's response. The terminal presents the query in voice or text format and receives the user's response.

[0329] Step 5:

[0330] The server analyzes the user's response and generates additional relevant queries using natural language processing techniques. The input is the user's response, and the output is the additional queries. The server converts speech to text using tools like Amazon Transcribe and then applies natural language processing.

[0331] Step 6:

[0332] The terminal then presents the user with additional queries to facilitate continued interaction. The input is the additional queries, and the output is the continuation of the interaction. The terminal saves this interaction to a recording device, which the user can review later.

[0333] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0334] This invention is a system that combines an emotion engine with image analysis technology and natural language processing technology, with the aim of stimulating the cognitive functions of the elderly and enriching their conversations.

[0335] The user first operates the device to select images from their memory to recall past experiences. This image selection serves as a means of stimulating the user's memories and emotions.

[0336] The server receives the selected image and extracts its feature information using image analysis technology. This feature information includes the date and time the image was taken, the location, and information about objects and people in the image. Based on this, the server uses natural language processing technology to generate questions that are easy for the user to answer naturally and presents them to the user through the terminal.

[0337] When a user responds to a presented question via voice or text, the device sends the response to the server. The emotion engine on the server analyzes the tone and content of the user's voice to recognize the user's emotional state. This recognition of emotional state is used, for example, to determine whether the user is feeling nostalgic or amused after viewing an image.

[0338] The emotion data obtained by the emotion engine is used by the server to generate further questions and comments. In this process, the tone and content of the dialogue are adjusted according to the user's emotions. For example, if the user is feeling happy, more positive questions can be asked.

[0339] Furthermore, the user's responses and emotional changes are recorded by a recording device. This recording is used by the user to review their emotional changes and conversation history at a later date.

[0340] For example, if a user selects a photo of a fun family gathering, the server will generate questions such as "How did you feel about this gathering?" based on the facial recognition results obtained from the photo, providing a dialogue that elicits positive emotions from the user.

[0341] Thus, the present invention is a system that effectively promotes the activation of cognitive functions by retrieving the user's past memories through images and adjusting responses using an emotion engine.

[0342] The following describes the processing flow.

[0343] Step 1:

[0344] The user displays a list of images saved from their device's storage and selects a specific image. Through this image selection, the user attempts to reminisce about their past memories.

[0345] Step 2:

[0346] The device sends the selected image to the server. The server uses image analysis techniques to extract feature information from the image. This includes information such as face recognition, landmarks, and date and time within the image. Using this analysis result, the server prepares to generate appropriate questions relevant to the user.

[0347] Step 3:

[0348] The server generates appropriate questions using natural language processing techniques based on the extracted feature information. The generated questions are crafted in a format that makes it easy for the user to answer naturally. These questions are sent to the device, which then presents them to the user via voice or text.

[0349] Step 4:

[0350] The user responds to questions presented by the device using voice or text. The device sends this response to the server in real time. In the case of a voice response, it is transmitted as audio data.

[0351] Step 5:

[0352] The server analyzes the response received from the user using an emotion engine. The emotion engine recognizes the user's emotional state based on the tone of voice and the content of what is being said. For example, if the user's voice is cheerful, it detects a positive emotion.

[0353] Step 6:

[0354] The server considers the recognized emotional state and generates the next questions and comments. At this stage, adjustments are made according to the user's emotions. If the emotional state is positive, questions and comments are generated to continue the positive conversation. The generated content is then sent back to the device.

[0355] Step 7:

[0356] The device presents the user with newly generated questions and comments, facilitating the smooth continuation of the dialogue. The user responds, and the dialogue cycle continues.

[0357] Step 8:

[0358] The server saves data on the user's responses and emotional changes to a recording device. This allows the user to review the conversation history later and reflect on the changes in their emotions.

[0359] (Example 2)

[0360] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0361] To improve cognitive function in older adults, a system is needed that stimulates past memories and guides conversations according to emotions. However, conventional systems have shortcomings in recognizing emotions and generating appropriate dialogues, making it difficult to provide natural and effective cognitive stimulation for older adults.

[0362] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0363] In this invention, the server includes means for the user to select past images stored in information memory, means for analyzing the features of the selected images using image analysis technology, means for generating questions based on the analyzed features using natural language processing technology, and means for using emotion analysis technology to recognize the emotional state based on the generated questions. This enables the provision of interactive dialogue tailored to the user's emotions and memories, and activates the user's cognitive functions.

[0364] An "information processing device" refers to any electronic device used for inputting, processing, storing, and outputting data, which can be operated by the user through an interface.

[0365] "Information storage" refers to data storage that stores digital information and allows it to be retrieved as needed.

[0366] "Image analysis technology" refers to all technologies used to extract features and patterns from digital images and perform classification and recognition.

[0367] "Natural language processing technology" refers to the technology that enables computers to understand human language and process it automatically.

[0368] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate natural language sentences from input data.

[0369] "Emotional analysis technology" refers to technology that automatically analyzes and classifies a speaker's emotional state from audio and text information.

[0370] "Information recording" refers to the process of saving conversations and data so that they can be referenced later.

[0371] The following configuration and procedure are necessary to implement the invention. This invention is a system for stimulating the cognitive functions of the elderly and enriching their conversations, and its details are described below.

[0372] First, the user uses the terminal to select past images stored in the information memory. The terminal is equipped with input devices such as a touchscreen, keyboard, and mouse, and the user can operate it through the interface. The selected images are saved in standard formats such as JPEG and PNG.

[0373] Next, the terminal sends the selected image data to the server. The server uses image analysis techniques to extract features from this image. This analysis utilizes AI-based image processing libraries (e.g., OpenCV, TensorFlow). The feature information includes the date and time of capture, location, and information about objects and people in the image.

[0374] The server uses a generative AI model based on the extracted feature information and applies natural language processing techniques (e.g., the GPT model) to generate questions for the user. These questions are then presented to the user via the terminal.

[0375] For example, if a user selects a photo from a family trip, the server will generate a question such as, "Tell us about the most enjoyable moment from your family trip," based on the information obtained from that image. This prompt helps the user recall their memories.

[0376] The user responds to the generated questions using voice or text. The terminal sends the response to the server. The server uses sentiment analysis technology (e.g., IBM Watson Tone Analyzer) to analyze the user's tone of voice and text content to recognize the user's emotional state.

[0377] To provide symmetrical dialogue that reflects emotions, the server can adjust the dialogue based on the user's emotions and generate additional questions or comments. This allows the system to provide a valuable interactive experience for the user and support the activation of cognitive functions.

[0378] Furthermore, the history of the conversation and data on the user's emotional changes are stored in the information log, which the user can use to look back on it at a later date.

[0379] Thus, the present invention provides a system that supports the cognitive function of the elderly by combining image analysis technology, natural language processing technology, and sentiment analysis technology.

[0380] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0381] Step 1:

[0382] The user operates the device to select past images stored in the information memory. This selection is made via touch or mouse clicks, and the input image data is displayed on the device's screen. The images selected by the user are used as basic data for subsequent processing.

[0383] Step 2:

[0384] The terminal sends the selected image data to the server. The images are in formats such as JPEG or PNG, and this becomes the input data for the server's analysis system. After receiving the data, the server prepares to pass the images to the analysis tool.

[0385] Step 3:

[0386] The server extracts features from the received image data using image analysis techniques. Specifically, AI algorithms (e.g., OpenCV, TensorFlow) are applied to obtain information such as the date and time of capture, location, and objects or people within the image. This feature data forms the basis for the next questions generated.

[0387] Step 4:

[0388] The server uses natural language processing techniques with feature information obtained from image analysis as input. It utilizes a generative AI model (e.g., GPT model) to generate questions for the user. During this process, the output is adjusted to ensure it is easy for the user to answer. The generated questions are then sent to the terminal.

[0389] Step 5:

[0390] The terminal presents the user with a question sent from the server. The question is displayed as text on the screen and may be read aloud if necessary. At this stage, the user receives the question on the screen.

[0391] Step 6:

[0392] The user responds to the presented questions either verbally or in text. This response is recorded as input data on the device and, if verbal, is converted to text using speech recognition technology. This text data forms the basis for the next analysis step.

[0393] Step 7:

[0394] The terminal sends the user's response text to the server. Upon receiving this transmitted data, the server prepares to perform the next analysis.

[0395] Step 8:

[0396] The server applies sentiment analysis technology to analyze the emotional state of the user's responses. This analysis considers tone information obtained from the speech and the content of the text. The analysis results become important input data for generating the next dialogue.

[0397] Step 9:

[0398] The server uses the results of the emotional state analysis as input and generates adaptive additional questions and comments using a generative AI model. In this step, the generated content is adjusted according to the user's emotional state. The results are sent to the terminal, and the interactive dialogue with the user continues.

[0399] (Application Example 2)

[0400] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0401] Cognitive decline in older adults is often exacerbated by a lack of interpersonal interaction and stimulation of memories based on past experiences. Furthermore, eliciting emotions through conversation is difficult, and providing dialogue that responds to those emotions is even more challenging. Current technology does not adequately provide solutions that can appropriately understand a user's emotional state and provide dialogue accordingly. Therefore, there is a need for new methods that effectively stimulate cognitive function and richly evoke past memories and emotions.

[0402] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0403] In this invention, the server includes means for the user to select past images stored in a memory device, means for analyzing the characteristic information of the selected images, and means for adjusting the dialogue content based on the analyzed emotions. This makes it possible to stimulate the past memories of elderly people, understand their emotions, and provide more appropriate and enriching dialogue.

[0404] A "terminal" is a device used by a user to operate, and is equipped with storage and communication functions.

[0405] A "storage device" is a device for holding data, and is a medium for storing information such as images and text.

[0406] "Image analysis" is a technique for extracting feature information from selected images, and is the process of recognizing objects and photographic information within an image.

[0407] "Feature information" refers to data obtained through image analysis, including information about people, objects, time, and location within the image.

[0408] "Question generation" is the process of creating questions to present to users based on analyzed feature information.

[0409] "User response" refers to the user's reply to a presented question, which is information sent to the server as audio or text.

[0410] "Emotional analysis" is a technology that evaluates a user's emotional state based on their responses and tone of voice.

[0411] "Dialogue adjustment" is the process of dynamically changing the tone and content of a dialogue based on analyzed emotions.

[0412] "Providing interaction" is the process of offering additional dialogue and feedback tailored to the user's emotional state.

[0413] A "recording device" is a device for saving the content of conversations with users and is a medium for making past logs accessible.

[0414] The system that realizes this invention includes several main components. First, the terminal allows the user to select an image from its storage device to reminisce about past memories. The selected image is sent to a server, which uses image analysis technology to extract detailed feature information from the image. This feature information includes people and objects in the image, the date and time the image was taken, and the location where it was taken.

[0415] The server then uses natural language processing techniques based on the extracted feature information to generate questions that are easy for the user to answer. The generated questions are presented to the user through the terminal, and the user's voice or text response is sent back to the server.

[0416] Sentiment analysis is performed by an emotion engine based on the user's responses and tone of voice. This evaluates the user's emotional state, and the tone and content of the conversation are adjusted accordingly. For example, if the user is enjoying looking at an image, more positive questions and comments will be provided.

[0417] Furthermore, past conversations and emotional changes are saved in the recording device, allowing users to later review their emotional progression and conversation history. This supports the long-term activation of cognitive functions.

[0418] As a concrete example, when a user selects a family photo, the server asks a question such as, "What is the most memorable event from this photo?" The emotion engine analyzes the user's cheerful response and then asks follow-up questions such as, "Could you tell us more about that memory?"

[0419] An example of a prompt for a generative AI model is the question, "Output questions to guide the conversation about related events and emotions so that the model can describe the user's emotions associated with the image."

[0420] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0421] Step 1:

[0422] The user operates the device and selects a past image from its storage. The input is the user's instructions, and the output is the selected image. The user's action is to tap the image through the device's interface.

[0423] Step 2:

[0424] The terminal sends the selected image to the server. The input is the selected image, and the output is the completion of the transfer of the image data to the server. The terminal uploads the image data to the server using the network.

[0425] Step 3:

[0426] The server extracts feature information from received images using image analysis techniques. The input is image data, and the output is feature information (people, objects, date and time, location, etc.). The server applies image processing algorithms to identify important elements within the image.

[0427] Step 4:

[0428] The server analyzes the extracted feature information and uses natural language processing techniques to generate questions that are easy for the user to answer. The input is feature information, and the output is the generated questions. The server uses a generative AI model to construct highly relevant questions.

[0429] Step 5:

[0430] The generated question is sent to the device and presented to the user. The input is the generated question, and the output is the display of the question to the user. The device displays the question in the UI and prompts the user for a voice or text response.

[0431] Step 6:

[0432] The user responds to questions using voice or text. Input is the user's response, and output is the response data input to the terminal. The user inputs their answers using a microphone or keyboard.

[0433] Step 7:

[0434] The terminal sends the user's response to the server. The input is the response data from the user, and the output is the completion of data transmission to the server. The terminal processes the response data and forwards it to the server.

[0435] Step 8:

[0436] The server analyzes the user's response using sentiment analysis technology to recognize their emotional state. The input is the response data, and the output is the recognized emotional state. The server evaluates the text and tone of the response and uses an emotion engine to identify the emotion.

[0437] Step 9:

[0438] The server adjusts the dialogue content based on the recognized emotional state and generates new questions and comments. The input is the emotional state, and the output is the content of the next dialogue. The server then uses the generative AI model again to design an appropriate dialogue.

[0439] Step 10:

[0440] The refined dialogue is sent to the terminal to support further interaction with the user. The input is the newly generated dialogue, and the output is what is presented to the user. The terminal provides the user with a continuous conversation, facilitating communication.

[0441] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0442] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0443] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0444] [Third Embodiment]

[0445] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0446] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0447] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0448] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0449] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0450] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0451] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0452] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0453] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0454] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0455] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0456] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0457] This invention is an interactive system aimed at preventing dementia in the elderly, and consists of a terminal, a server, and a storage device. The embodiments thereof are described in detail below.

[0458] First, the user accesses the system using a terminal. The user can then select past images stored in the device's interface. These selected images are used as triggers to evoke the user's past experiences and memories.

[0459] The server is responsible for analyzing image data. It obtains characteristic information about the image using metadata associated with the image and image analysis techniques. For example, if a specific landmark or person is depicted in the image, that information is identified.

[0460] Next, the server automatically generates questions to present to the user based on this analyzed information. Natural language processing technology is used for this question generation, and the questions are output in a format that is easy for the user to answer naturally.

[0461] The device presents the generated questions to the user in either audio or text format. The user answers the questions, initiating a dialogue. The user's responses are then analyzed by the server, generating additional relevant questions or comments. This allows the dialogue to continue naturally and provides the user with opportunities to activate their memory.

[0462] Finally, the server saves the content of the interaction with the user to a recording device. This record allows the user to review past interactions at a later date, which is expected to facilitate long-term learning and memory enhancement.

[0463] For example, when a user selects a travel photo, the server generates questions based on the location and time of the photo, such as "Who did you visit this place with?" or "What was the most enjoyable part of this trip?", deepening the conversation with the user. In this way, the present invention contributes to elderly people reliving past experiences and stimulating their brains through conversation.

[0464] The following describes the processing flow.

[0465] Step 1:

[0466] The user accesses the photo storage service using their device. Through the interface, the user views a list of past images stored on the storage device and selects a specific image they wish to reminisce about.

[0467] Step 2:

[0468] The selected image is sent from the terminal to the server. The server uses metadata associated with the image (e.g., date and time of capture, location) and image analysis techniques to analyze the image's characteristic information (e.g., specific landmarks or people).

[0469] Step 3:

[0470] The server generates questions related to the image based on the analysis results. The question generation engine uses natural language processing technology to prepare multiple questions in a format that is easy for the user to answer.

[0471] Step 4:

[0472] The terminal presents the user with a question sent from the server, either verbally or as text. The user responds to the question from the terminal using either verbal or text input.

[0473] Step 5:

[0474] The user's response is sent from the terminal to the server. The server uses speech recognition and natural language processing technologies to analyze the response and generate additional questions or comments related to its content.

[0475] Step 6:

[0476] The device presents the user with additional questions or comments, continuing the conversation. By answering the questions, the user can share more detailed memories.

[0477] Step 7:

[0478] The server saves the content of the conversation with the user as a log to a recording device. The conversation log is made available for the user to review at a later date.

[0479] (Example 1)

[0480] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0481] In preventing dementia in the elderly, it is necessary to smoothly carry out a series of processes that effectively recall past memories and activate the brain through dialogue. However, existing systems have difficulty naturally continuing dialogue with users, and the activation of memories through dialogue is not currently sufficient.

[0482] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0483] In this invention, the server includes means for a user to select past visual data stored in a storage medium on an information terminal, means for an information processing device that analyzes the characteristic information of the selected visual data, and means for generating questions using a generative artificial intelligence model. This makes it possible to effectively evoke memories using visual data as a trigger and to naturally continue the dialogue with the user.

[0484] An "information terminal" is an electronic device that allows users to perform operations such as selecting visual data or responding to questions via an interface.

[0485] A "storage medium" is a recording device used to store data such as visual data and dialogue content.

[0486] "Visual data" refers to past images and photographs used to evoke past experiences and memories.

[0487] An "information processing device" is a computer used to analyze visual data and extract feature information.

[0488] "Feature information" refers to identifiable information such as landmarks and people contained in visual data.

[0489] A "generative artificial intelligence model" is a machine learning-based model used to automatically generate questions for users.

[0490] "Natural language processing technology" is a computer language processing technology used to analyze user responses and generate natural-sounding dialogue.

[0491] Modes for carrying out the invention

[0492] This invention is an interactive system aimed at preventing dementia in the elderly, and consists of an information terminal, a server, and a storage medium. A specific example of this system is described below.

[0493] Users access the system using an information terminal. This terminal is a general-purpose computer such as a tablet or smartphone, and it provides an interface for selecting past visual data (images and photographs) stored on a storage medium. This selected visual data is used as a trigger to evoke the user's past experiences and memories.

[0494] The server is responsible for analyzing visual data. Using visual data analysis technologies such as OpenCV and visual data recognition APIs, the server extracts feature information from image data. This feature information includes identifiable elements such as landmarks and people.

[0495] Next, the server uses a generative AI model to automatically generate questions to present to the user based on the extracted feature information. This utilizes natural language processing technology to output the questions in a format that is easy for the user to answer. For example, possible generated prompts might be "Who did you visit this place with?" or "What was the most memorable event of this trip?"

[0496] The terminal presents the user with generated questions, and the user responds. The server analyzes the user's responses and generates additional questions or comments based on them, continuing the dialogue. This process helps to evoke the user's memory and enables a more natural and richer conversation.

[0497] For example, if a user selects a photo from a family trip, the server will generate questions such as, "Who are the people in this photo?" or "What was the most enjoyable part of this trip?" This stimulates the user's memory and contributes to brain health.

[0498] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0499] Step 1:

[0500] The user accesses the system using an information terminal and selects past visual data from a storage medium. The input is the visual data selected by the user, and the output is the selected image data. Specifically, the terminal displays a user interface and presents the user with a list of images. The user selects a specific photograph while browsing using touch controls.

[0501] Step 2:

[0502] The server analyzes the selected image data received from the user. The input for this step is the selected image data, and the output is feature information about that image. The server uses visual data analysis techniques (e.g., OpenCV) to extract features such as landmarks and people. Specifically, the server inputs the image data into a machine learning model to analyze geometric shapes, colors, and structural features.

[0503] Step 3:

[0504] The server automatically generates questions based on extracted feature information. The input is the feature information, and the output is the generated question. The server uses a generative AI model to create questions in a natural language format. Specifically, the server inputs the feature information as a prompt into the generative AI model, resulting in a question such as, "Who did you visit this place with?"

[0505] Step 4:

[0506] The terminal presents the user with a question received from the server. The input is the generated question, and the output is the user's response. Specifically, the terminal displays the question in audio or text format and collects the user's answer as audio or text input.

[0507] Step 5:

[0508] The server analyzes the user's response and generates additional questions. Here, the input is the user's response, and the output is the additional questions. The server uses natural language processing techniques to analyze the user's intent and information to derive relevant questions. Specifically, the server runs an algorithm that analyzes the user's answer and generates new questions related to that response.

[0509] Step 6:

[0510] The server saves the entire dialogue session to a storage medium. The input is the dialogue log data, and the output is the saved data. Specifically, the server records the dialogue content as structured data in a database and manages it so that the user can access it later.

[0511] (Application Example 1)

[0512] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0513] The problem that this invention aims to solve is to provide an effective conversational support system that contributes to maintaining and improving the cognitive function of the elderly. This problem stems from the limited opportunities to recall memories in daily life and the lack of a constant companion to activate memories through conversation. In particular, there is a need for technology that generates natural conversations based on visual information.

[0514] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0515] In this invention, the server includes means for analyzing past visual information stored in a memory device to obtain feature information, means for automatically generating queries in natural language form using a generative AI model based on the analyzed feature information, and means for analyzing the generated questions and the user's responses and automatically generating additional related queries using natural language processing technology. This enables the user to activate their memory and maintain cognitive function through interactive dialogue triggered by their past visual information.

[0516] A "terminal" is a portable information device used by a user to access storage and select past visual information.

[0517] A "memory device" is a data storage unit that stores past visual information that the user can select.

[0518] "Visual information" refers to image data used to evoke past experiences and memories of the user.

[0519] "Feature information" refers to identifying information such as landmarks and people extracted from analyzed visual information.

[0520] A "generative AI model" is an artificial intelligence technology that automatically generates queries in natural language format based on feature information.

[0521] An "inquiry" is a question in natural language format that is generated by a generative AI model based on visual information and presented to the user.

[0522] "Natural language processing technology" refers to language analysis techniques used to analyze user responses and generate related additional queries.

[0523] A "conversation" refers to the exchange between generated inquiries and user responses, and is the process of interaction that is stored in a recording device.

[0524] This invention is a system that activates cognitive functions in elderly individuals by enabling them to engage in dialogue based on their past visual information. This system consists of a portable information device, a server, and a storage device.

[0525] The user accesses the system using a terminal and selects past visual information stored in the storage device. The selected visual information is sent from the terminal to the server. The server receives the visual information and uses image analysis technology, such as the Google Cloud Vision API, to obtain feature information (landmarks, people, etc.).

[0526] Based on the analyzed feature information, the server automatically generates natural language queries using generative AI models such as OpenAI's GPT model. These queries are then presented to the user via the terminal. The user responds to the presented queries.

[0527] User responses are analyzed by a server using speech recognition and natural language processing technologies such as Amazon Transcribe. Based on the analysis results, the server generates relevant additional queries and presents them to the user again. This entire process enables continuous interaction between the user and the system, effectively retrieving the user's memory.

[0528] For example, if a user selects a past "family trip photo," the server generates questions such as "Where was this photo taken?" or "What was the most memorable event from this trip?" based on the feature information extracted from the photo.

[0529] An example of a prompt for a generative AI model is: "Image analysis result: Landmark 'Historical building', Person 'Relative'. Question generation: Generate a question that will help the user recall memories." Such prompts allow the system to provide a dialogue that makes it easy for the user to recall past experiences.

[0530] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0531] Step 1:

[0532] The user accesses the system using a portable information device and selects past visual information stored in its memory. The input is the user's selection, and the output is the selected visual information. Based on the user's selection, the terminal sends the corresponding visual information to the server.

[0533] Step 2:

[0534] The server uses image analysis technology to extract feature information from the received visual information. The input is selected visual information, and the output is feature information such as landmarks and people in the image. In this process, the server analyzes the feature information using the Google Cloud Vision API, among others.

[0535] Step 3:

[0536] The server automatically generates natural language queries using a generative AI model based on the acquired feature information. The input is the feature information, and the output is the generated query. The server generates the question using OpenAI's GPT model and sends it to the terminal.

[0537] Step 4:

[0538] The terminal presents a generated query to the user. The user responds to it. The input is the generated query, and the output is the user's response. The terminal presents the query in voice or text format and receives the user's response.

[0539] Step 5:

[0540] The server analyzes the user's response and generates additional relevant queries using natural language processing techniques. The input is the user's response, and the output is the additional queries. The server converts speech to text using tools like Amazon Transcribe and then applies natural language processing.

[0541] Step 6:

[0542] The terminal then presents the user with additional queries to facilitate continued interaction. The input is the additional queries, and the output is the continuation of the interaction. The terminal saves this interaction to a recording device, which the user can review later.

[0543] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0544] This invention is a system that combines an emotion engine with image analysis technology and natural language processing technology, with the aim of stimulating the cognitive functions of the elderly and enriching their conversations.

[0545] The user first operates the device to select images from their memory to recall past experiences. This image selection serves as a means of stimulating the user's memories and emotions.

[0546] The server receives the selected image and extracts its feature information using image analysis technology. This feature information includes the date and time the image was taken, the location, and information about objects and people in the image. Based on this, the server uses natural language processing technology to generate questions that are easy for the user to answer naturally and presents them to the user through the terminal.

[0547] When a user responds to a presented question via voice or text, the device sends the response to the server. The emotion engine on the server analyzes the tone and content of the user's voice to recognize the user's emotional state. This recognition of emotional state is used, for example, to determine whether the user is feeling nostalgic or amused after viewing an image.

[0548] The emotion data obtained by the emotion engine is used by the server to generate further questions and comments. In this process, the tone and content of the dialogue are adjusted according to the user's emotions. For example, if the user is feeling happy, more positive questions can be asked.

[0549] Furthermore, the user's responses and emotional changes are recorded by a recording device. This recording is used by the user to review their emotional changes and conversation history at a later date.

[0550] For example, if a user selects a photo of a fun family gathering, the server will generate questions such as "How did you feel about this gathering?" based on the facial recognition results obtained from the photo, providing a dialogue that elicits positive emotions from the user.

[0551] Thus, the present invention is a system that effectively promotes the activation of cognitive functions by retrieving the user's past memories through images and adjusting responses using an emotion engine.

[0552] The following describes the processing flow.

[0553] Step 1:

[0554] The user displays a list of images saved from their device's storage and selects a specific image. Through this image selection, the user attempts to reminisce about their past memories.

[0555] Step 2:

[0556] The device sends the selected image to the server. The server uses image analysis techniques to extract feature information from the image. This includes information such as face recognition, landmarks, and date and time within the image. Using this analysis result, the server prepares to generate appropriate questions relevant to the user.

[0557] Step 3:

[0558] The server generates appropriate questions using natural language processing techniques based on the extracted feature information. The generated questions are crafted in a format that makes it easy for the user to answer naturally. These questions are sent to the device, which then presents them to the user via voice or text.

[0559] Step 4:

[0560] The user responds to questions presented by the device using voice or text. The device sends this response to the server in real time. In the case of a voice response, it is transmitted as audio data.

[0561] Step 5:

[0562] The server analyzes the response received from the user using an emotion engine. The emotion engine recognizes the user's emotional state based on the tone of voice and the content of what is being said. For example, if the user's voice is cheerful, it detects a positive emotion.

[0563] Step 6:

[0564] The server considers the recognized emotional state and generates the next questions and comments. At this stage, adjustments are made according to the user's emotions. If the emotional state is positive, questions and comments are generated to continue the positive conversation. The generated content is then sent back to the device.

[0565] Step 7:

[0566] The device presents the user with newly generated questions and comments, facilitating the smooth continuation of the dialogue. The user responds, and the dialogue cycle continues.

[0567] Step 8:

[0568] The server saves data on the user's responses and emotional changes to a recording device. This allows the user to review the conversation history later and reflect on the changes in their emotions.

[0569] (Example 2)

[0570] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0571] To improve cognitive function in older adults, a system is needed that stimulates past memories and guides conversations according to emotions. However, conventional systems have shortcomings in recognizing emotions and generating appropriate dialogues, making it difficult to provide natural and effective cognitive stimulation for older adults.

[0572] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0573] In this invention, the server includes means for the user to select past images stored in information memory, means for analyzing the features of the selected images using image analysis technology, means for generating questions based on the analyzed features using natural language processing technology, and means for using emotion analysis technology to recognize the emotional state based on the generated questions. This enables the provision of interactive dialogue tailored to the user's emotions and memories, and activates the user's cognitive functions.

[0574] An "information processing device" refers to any electronic device used for inputting, processing, storing, and outputting data, which can be operated by the user through an interface.

[0575] "Information storage" refers to data storage that stores digital information and allows it to be retrieved as needed.

[0576] "Image analysis technology" refers to all technologies used to extract features and patterns from digital images and perform classification and recognition.

[0577] "Natural language processing technology" refers to the technology that enables computers to understand human language and process it automatically.

[0578] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate natural language sentences from input data.

[0579] "Emotional analysis technology" refers to technology that automatically analyzes and classifies a speaker's emotional state from audio and text information.

[0580] "Information recording" refers to the process of saving conversations and data so that they can be referenced later.

[0581] The following configuration and procedure are necessary to implement the invention. This invention is a system for stimulating the cognitive functions of the elderly and enriching their conversations, and its details are described below.

[0582] First, the user uses the terminal to select past images stored in the information memory. The terminal is equipped with input devices such as a touchscreen, keyboard, and mouse, and the user can operate it through the interface. The selected images are saved in standard formats such as JPEG and PNG.

[0583] Next, the terminal sends the selected image data to the server. The server uses image analysis techniques to extract features from this image. This analysis utilizes AI-based image processing libraries (e.g., OpenCV, TensorFlow). The feature information includes the date and time of capture, location, and information about objects and people in the image.

[0584] The server uses a generative AI model based on the extracted feature information and applies natural language processing techniques (e.g., the GPT model) to generate questions for the user. These questions are then presented to the user via the terminal.

[0585] For example, if a user selects a photo from a family trip, the server will generate a question such as, "Tell us about the most enjoyable moment from your family trip," based on the information obtained from that image. This prompt helps the user recall their memories.

[0586] The user responds to the generated questions using voice or text. The terminal sends the response to the server. The server uses sentiment analysis technology (e.g., IBM Watson Tone Analyzer) to analyze the user's tone of voice and text content to recognize the user's emotional state.

[0587] To provide symmetrical dialogue that reflects emotions, the server can adjust the dialogue based on the user's emotions and generate additional questions or comments. This allows the system to provide a valuable interactive experience for the user and support the activation of cognitive functions.

[0588] Furthermore, the history of the conversation and data on the user's emotional changes are stored in the information log, which the user can use to look back on it at a later date.

[0589] Thus, the present invention provides a system that supports the cognitive function of the elderly by combining image analysis technology, natural language processing technology, and sentiment analysis technology.

[0590] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0591] Step 1:

[0592] The user operates the device to select past images stored in the information memory. This selection is made via touch or mouse clicks, and the input image data is displayed on the device's screen. The images selected by the user are used as basic data for subsequent processing.

[0593] Step 2:

[0594] The terminal sends the selected image data to the server. The images are in formats such as JPEG or PNG, and this becomes the input data for the server's analysis system. After receiving the data, the server prepares to pass the images to the analysis tool.

[0595] Step 3:

[0596] The server extracts features from the received image data using image analysis techniques. Specifically, AI algorithms (e.g., OpenCV, TensorFlow) are applied to obtain information such as the date and time of capture, location, and objects or people within the image. This feature data forms the basis for the next questions generated.

[0597] Step 4:

[0598] The server uses natural language processing techniques with feature information obtained from image analysis as input. It utilizes a generative AI model (e.g., GPT model) to generate questions for the user. During this process, the output is adjusted to ensure it is easy for the user to answer. The generated questions are then sent to the terminal.

[0599] Step 5:

[0600] The terminal presents the user with a question sent from the server. The question is displayed as text on the screen and may be read aloud if necessary. At this stage, the user receives the question on the screen.

[0601] Step 6:

[0602] The user responds to the presented questions either verbally or in text. This response is recorded as input data on the device and, if verbal, is converted to text using speech recognition technology. This text data forms the basis for the next analysis step.

[0603] Step 7:

[0604] The terminal sends the user's response text to the server. Upon receiving this transmitted data, the server prepares to perform the next analysis.

[0605] Step 8:

[0606] The server applies sentiment analysis technology to analyze the emotional state of the user's responses. This analysis considers tone information obtained from the speech and the content of the text. The analysis results become important input data for generating the next dialogue.

[0607] Step 9:

[0608] The server uses the results of the emotional state analysis as input and generates adaptive additional questions and comments using a generative AI model. In this step, the generated content is adjusted according to the user's emotional state. The results are sent to the terminal, and the interactive dialogue with the user continues.

[0609] (Application Example 2)

[0610] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0611] Cognitive decline in older adults is often exacerbated by a lack of interpersonal interaction and stimulation of memories based on past experiences. Furthermore, eliciting emotions through conversation is difficult, and providing dialogue that responds to those emotions is even more challenging. Current technology does not adequately provide solutions that can appropriately understand a user's emotional state and provide dialogue accordingly. Therefore, there is a need for new methods that effectively stimulate cognitive function and richly evoke past memories and emotions.

[0612] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0613] In this invention, the server includes means for the user to select past images stored in a memory device, means for analyzing the characteristic information of the selected images, and means for adjusting the dialogue content based on the analyzed emotions. This makes it possible to stimulate the past memories of elderly people, understand their emotions, and provide more appropriate and enriching dialogue.

[0614] A "terminal" is a device used by a user to operate, and is equipped with storage and communication functions.

[0615] A "storage device" is a device for holding data, and is a medium for storing information such as images and text.

[0616] "Image analysis" is a technique for extracting feature information from selected images, and is the process of recognizing objects and photographic information within an image.

[0617] "Feature information" refers to data obtained through image analysis, including information about people, objects, time, and location within the image.

[0618] "Question generation" is the process of creating questions to present to users based on analyzed feature information.

[0619] "User response" refers to the user's reply to a presented question, which is information sent to the server as audio or text.

[0620] "Emotional analysis" is a technology that evaluates a user's emotional state based on their responses and tone of voice.

[0621] "Dialogue adjustment" is the process of dynamically changing the tone and content of a dialogue based on analyzed emotions.

[0622] "Providing interaction" is the process of offering additional dialogue and feedback tailored to the user's emotional state.

[0623] A "recording device" is a device for saving the content of conversations with users and is a medium for making past logs accessible.

[0624] The system that realizes this invention includes several main components. First, the terminal allows the user to select an image from its storage device to reminisce about past memories. The selected image is sent to a server, which uses image analysis technology to extract detailed feature information from the image. This feature information includes people and objects in the image, the date and time the image was taken, and the location where it was taken.

[0625] The server then uses natural language processing techniques based on the extracted feature information to generate questions that are easy for the user to answer. The generated questions are presented to the user through the terminal, and the user's voice or text response is sent back to the server.

[0626] Sentiment analysis is performed by an emotion engine based on the user's responses and tone of voice. This evaluates the user's emotional state, and the tone and content of the conversation are adjusted accordingly. For example, if the user is enjoying looking at an image, more positive questions and comments will be provided.

[0627] Furthermore, past conversations and emotional changes are saved in the recording device, allowing users to later review their emotional progression and conversation history. This supports the long-term activation of cognitive functions.

[0628] As a concrete example, when a user selects a family photo, the server asks a question such as, "What is the most memorable event from this photo?" The emotion engine analyzes the user's cheerful response and then asks follow-up questions such as, "Could you tell us more about that memory?"

[0629] An example of a prompt for a generative AI model is the question, "Output questions to guide the conversation about related events and emotions so that the model can describe the user's emotions associated with the image."

[0630] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0631] Step 1:

[0632] The user operates the device and selects a past image from its storage. The input is the user's instructions, and the output is the selected image. The user's action is to tap the image through the device's interface.

[0633] Step 2:

[0634] The terminal sends the selected image to the server. The input is the selected image, and the output is the completion of the transfer of the image data to the server. The terminal uploads the image data to the server using the network.

[0635] Step 3:

[0636] The server extracts feature information from received images using image analysis techniques. The input is image data, and the output is feature information (people, objects, date and time, location, etc.). The server applies image processing algorithms to identify important elements within the image.

[0637] Step 4:

[0638] The server analyzes the extracted feature information and uses natural language processing techniques to generate questions that are easy for the user to answer. The input is feature information, and the output is the generated questions. The server uses a generative AI model to construct highly relevant questions.

[0639] Step 5:

[0640] The generated question is sent to the device and presented to the user. The input is the generated question, and the output is the display of the question to the user. The device displays the question in the UI and prompts the user for a voice or text response.

[0641] Step 6:

[0642] The user responds to questions using voice or text. Input is the user's response, and output is the response data input to the terminal. The user inputs their answers using a microphone or keyboard.

[0643] Step 7:

[0644] The terminal sends the user's response to the server. The input is the response data from the user, and the output is the completion of data transmission to the server. The terminal processes the response data and forwards it to the server.

[0645] Step 8:

[0646] The server analyzes the user's response using sentiment analysis technology to recognize their emotional state. The input is the response data, and the output is the recognized emotional state. The server evaluates the text and tone of the response and uses an emotion engine to identify the emotion.

[0647] Step 9:

[0648] The server adjusts the dialogue content based on the recognized emotional state and generates new questions and comments. The input is the emotional state, and the output is the content of the next dialogue. The server then uses the generative AI model again to design an appropriate dialogue.

[0649] Step 10:

[0650] The refined dialogue is sent to the terminal to support further interaction with the user. The input is the newly generated dialogue, and the output is what is presented to the user. The terminal provides the user with a continuous conversation, facilitating communication.

[0651] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0652] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0653] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0654] [Fourth Embodiment]

[0655] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0656] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0657] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0658] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0659] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0660] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0661] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0662] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0663] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0664] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0665] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0666] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0667] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] This invention is an interactive system aimed at preventing dementia in the elderly, and consists of a terminal, a server, and a storage device. The embodiments thereof are described in detail below.

[0669] First, the user accesses the system using a terminal. The user can then select past images stored in the device's interface. These selected images are used as triggers to evoke the user's past experiences and memories.

[0670] The server is responsible for analyzing image data. It obtains characteristic information about the image using metadata associated with the image and image analysis techniques. For example, if a specific landmark or person is depicted in the image, that information is identified.

[0671] Next, the server automatically generates questions to present to the user based on this analyzed information. Natural language processing technology is used for this question generation, and the questions are output in a format that is easy for the user to answer naturally.

[0672] The device presents the generated questions to the user in either audio or text format. The user answers the questions, initiating a dialogue. The user's responses are then analyzed by the server, generating additional relevant questions or comments. This allows the dialogue to continue naturally and provides the user with opportunities to activate their memory.

[0673] Finally, the server saves the content of the interaction with the user to a recording device. This record allows the user to review past interactions at a later date, which is expected to facilitate long-term learning and memory enhancement.

[0674] For example, when a user selects a travel photo, the server generates questions based on the location and time of the photo, such as "Who did you visit this place with?" or "What was the most enjoyable part of this trip?", deepening the conversation with the user. In this way, the present invention contributes to elderly people reliving past experiences and stimulating their brains through conversation.

[0675] The following describes the processing flow.

[0676] Step 1:

[0677] The user accesses the photo storage service using their device. Through the interface, the user views a list of past images stored on the storage device and selects a specific image they wish to reminisce about.

[0678] Step 2:

[0679] The selected image is sent from the terminal to the server. The server uses metadata associated with the image (e.g., date and time of capture, location) and image analysis techniques to analyze the image's characteristic information (e.g., specific landmarks or people).

[0680] Step 3:

[0681] The server generates questions related to the image based on the analysis results. The question generation engine uses natural language processing technology to prepare multiple questions in a format that is easy for the user to answer.

[0682] Step 4:

[0683] The terminal presents the user with a question sent from the server, either verbally or as text. The user responds to the question from the terminal using either verbal or text input.

[0684] Step 5:

[0685] The user's response is sent from the terminal to the server. The server uses speech recognition and natural language processing technologies to analyze the response and generate additional questions or comments related to its content.

[0686] Step 6:

[0687] The device presents the user with additional questions or comments, continuing the conversation. By answering the questions, the user can share more detailed memories.

[0688] Step 7:

[0689] The server saves the content of the conversation with the user as a log to a recording device. The conversation log is made available for the user to review at a later date.

[0690] (Example 1)

[0691] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0692] In preventing dementia in the elderly, it is necessary to smoothly carry out a series of processes that effectively recall past memories and activate the brain through dialogue. However, existing systems have difficulty naturally continuing dialogue with users, and the activation of memories through dialogue is not currently sufficient.

[0693] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0694] In this invention, the server includes means for a user to select past visual data stored in a storage medium on an information terminal, means for an information processing device that analyzes the characteristic information of the selected visual data, and means for generating questions using a generative artificial intelligence model. This makes it possible to effectively evoke memories using visual data as a trigger and to naturally continue the dialogue with the user.

[0695] An "information terminal" is an electronic device that allows users to perform operations such as selecting visual data or responding to questions via an interface.

[0696] A "storage medium" is a recording device used to store data such as visual data and dialogue content.

[0697] "Visual data" refers to past images and photographs used to evoke past experiences and memories.

[0698] An "information processing device" is a computer used to analyze visual data and extract feature information.

[0699] "Feature information" refers to identifiable information such as landmarks and people contained in visual data.

[0700] A "generative artificial intelligence model" is a machine learning-based model used to automatically generate questions for users.

[0701] "Natural language processing technology" is a computer language processing technology used to analyze user responses and generate natural-sounding dialogue.

[0702] Modes for carrying out the invention

[0703] This invention is an interactive system aimed at preventing dementia in the elderly, and consists of an information terminal, a server, and a storage medium. A specific example of this system is described below.

[0704] Users access the system using an information terminal. This terminal is a general-purpose computer such as a tablet or smartphone, and it provides an interface for selecting past visual data (images and photographs) stored on a storage medium. This selected visual data is used as a trigger to evoke the user's past experiences and memories.

[0705] The server is responsible for analyzing visual data. Using visual data analysis technologies such as OpenCV and visual data recognition APIs, the server extracts feature information from image data. This feature information includes identifiable elements such as landmarks and people.

[0706] Next, the server uses a generative AI model to automatically generate questions to present to the user based on the extracted feature information. This utilizes natural language processing technology to output the questions in a format that is easy for the user to answer. For example, possible generated prompts might be "Who did you visit this place with?" or "What was the most memorable event of this trip?"

[0707] The terminal presents the user with generated questions, and the user responds. The server analyzes the user's responses and generates additional questions or comments based on them, continuing the dialogue. This process helps to evoke the user's memory and enables a more natural and richer conversation.

[0708] For example, if a user selects a photo from a family trip, the server will generate questions such as, "Who are the people in this photo?" or "What was the most enjoyable part of this trip?" This stimulates the user's memory and contributes to brain health.

[0709] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0710] Step 1:

[0711] The user accesses the system using an information terminal and selects past visual data from a storage medium. The input is the visual data selected by the user, and the output is the selected image data. Specifically, the terminal displays a user interface and presents the user with a list of images. The user selects a specific photograph while browsing using touch controls.

[0712] Step 2:

[0713] The server analyzes the selected image data received from the user. The input for this step is the selected image data, and the output is feature information about that image. The server uses visual data analysis techniques (e.g., OpenCV) to extract features such as landmarks and people. Specifically, the server inputs the image data into a machine learning model to analyze geometric shapes, colors, and structural features.

[0714] Step 3:

[0715] The server automatically generates questions based on extracted feature information. The input is the feature information, and the output is the generated question. The server uses a generative AI model to create questions in a natural language format. Specifically, the server inputs the feature information as a prompt into the generative AI model, resulting in a question such as, "Who did you visit this place with?"

[0716] Step 4:

[0717] The terminal presents the user with a question received from the server. The input is the generated question, and the output is the user's response. Specifically, the terminal displays the question in audio or text format and collects the user's answer as audio or text input.

[0718] Step 5:

[0719] The server analyzes the user's response and generates additional questions. Here, the input is the user's response, and the output is the additional questions. The server uses natural language processing techniques to analyze the user's intent and information to derive relevant questions. Specifically, the server runs an algorithm that analyzes the user's answer and generates new questions related to that response.

[0720] Step 6:

[0721] The server saves the entire dialogue session to a storage medium. The input is the dialogue log data, and the output is the saved data. Specifically, the server records the dialogue content as structured data in a database and manages it so that the user can access it later.

[0722] (Application Example 1)

[0723] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0724] The problem that this invention aims to solve is to provide an effective conversational support system that contributes to maintaining and improving the cognitive function of the elderly. This problem stems from the limited opportunities to recall memories in daily life and the lack of a constant companion to activate memories through conversation. In particular, there is a need for technology that generates natural conversations based on visual information.

[0725] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0726] In this invention, the server includes means for analyzing past visual information stored in a memory device to obtain feature information, means for automatically generating queries in natural language form using a generative AI model based on the analyzed feature information, and means for analyzing the generated questions and the user's responses and automatically generating additional related queries using natural language processing technology. This enables the user to activate their memory and maintain cognitive function through interactive dialogue triggered by their past visual information.

[0727] A "terminal" is a portable information device used by a user to access storage and select past visual information.

[0728] A "memory device" is a data storage unit that stores past visual information that the user can select.

[0729] "Visual information" refers to image data used to evoke past experiences and memories of the user.

[0730] "Feature information" refers to identifying information such as landmarks and people extracted from analyzed visual information.

[0731] A "generative AI model" is an artificial intelligence technology that automatically generates queries in natural language format based on feature information.

[0732] An "inquiry" is a question in natural language format that is generated by a generative AI model based on visual information and presented to the user.

[0733] "Natural language processing technology" refers to language analysis techniques used to analyze user responses and generate related additional queries.

[0734] A "conversation" refers to the exchange between generated inquiries and user responses, and is the process of interaction that is stored in a recording device.

[0735] This invention is a system that activates cognitive functions in elderly individuals by enabling them to engage in dialogue based on their past visual information. This system consists of a portable information device, a server, and a storage device.

[0736] The user accesses the system using a terminal and selects past visual information stored in the storage device. The selected visual information is sent from the terminal to the server. The server receives the visual information and uses image analysis technology, such as the Google Cloud Vision API, to obtain feature information (landmarks, people, etc.).

[0737] Based on the analyzed feature information, the server automatically generates natural language queries using generative AI models such as OpenAI's GPT model. These queries are then presented to the user via the terminal. The user responds to the presented queries.

[0738] User responses are analyzed by a server using speech recognition and natural language processing technologies such as Amazon Transcribe. Based on the analysis results, the server generates relevant additional queries and presents them to the user again. This entire process enables continuous interaction between the user and the system, effectively retrieving the user's memory.

[0739] For example, if a user selects a past "family trip photo," the server generates questions such as "Where was this photo taken?" or "What was the most memorable event from this trip?" based on the feature information extracted from the photo.

[0740] An example of a prompt for a generative AI model is: "Image analysis result: Landmark 'Historical building', Person 'Relative'. Question generation: Generate a question that will help the user recall memories." Such prompts allow the system to provide a dialogue that makes it easy for the user to recall past experiences.

[0741] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0742] Step 1:

[0743] The user accesses the system using a portable information device and selects past visual information stored in its memory. The input is the user's selection, and the output is the selected visual information. Based on the user's selection, the terminal sends the corresponding visual information to the server.

[0744] Step 2:

[0745] The server uses image analysis technology to extract feature information from the received visual information. The input is selected visual information, and the output is feature information such as landmarks and people in the image. In this process, the server analyzes the feature information using the Google Cloud Vision API, among others.

[0746] Step 3:

[0747] The server automatically generates natural language queries using a generative AI model based on the acquired feature information. The input is the feature information, and the output is the generated query. The server generates the question using OpenAI's GPT model and sends it to the terminal.

[0748] Step 4:

[0749] The terminal presents a generated query to the user. The user responds to it. The input is the generated query, and the output is the user's response. The terminal presents the query in voice or text format and receives the user's response.

[0750] Step 5:

[0751] The server analyzes the user's response and generates additional relevant queries using natural language processing techniques. The input is the user's response, and the output is the additional queries. The server converts speech to text using tools like Amazon Transcribe and then applies natural language processing.

[0752] Step 6:

[0753] The terminal then presents the user with additional queries to facilitate continued interaction. The input is the additional queries, and the output is the continuation of the interaction. The terminal saves this interaction to a recording device, which the user can review later.

[0754] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0755] This invention is a system that combines an emotion engine with image analysis technology and natural language processing technology, with the aim of stimulating the cognitive functions of the elderly and enriching their conversations.

[0756] The user first operates the device to select images from their memory to recall past experiences. This image selection serves as a means of stimulating the user's memories and emotions.

[0757] The server receives the selected image and extracts its feature information using image analysis technology. This feature information includes the date and time the image was taken, the location, and information about objects and people in the image. Based on this, the server uses natural language processing technology to generate questions that are easy for the user to answer naturally and presents them to the user through the terminal.

[0758] When a user responds to a presented question via voice or text, the device sends the response to the server. The emotion engine on the server analyzes the tone and content of the user's voice to recognize the user's emotional state. This recognition of emotional state is used, for example, to determine whether the user is feeling nostalgic or amused after viewing an image.

[0759] The emotion data obtained by the emotion engine is used by the server to generate further questions and comments. In this process, the tone and content of the dialogue are adjusted according to the user's emotions. For example, if the user is feeling happy, more positive questions can be asked.

[0760] Furthermore, the user's responses and emotional changes are recorded by a recording device. This recording is used by the user to review their emotional changes and conversation history at a later date.

[0761] For example, if a user selects a photo of a fun family gathering, the server will generate questions such as "How did you feel about this gathering?" based on the facial recognition results obtained from the photo, providing a dialogue that elicits positive emotions from the user.

[0762] Thus, the present invention is a system that effectively promotes the activation of cognitive functions by retrieving the user's past memories through images and adjusting responses using an emotion engine.

[0763] The following describes the processing flow.

[0764] Step 1:

[0765] The user displays a list of images saved from their device's storage and selects a specific image. Through this image selection, the user attempts to reminisce about their past memories.

[0766] Step 2:

[0767] The device sends the selected image to the server. The server uses image analysis techniques to extract feature information from the image. This includes information such as face recognition, landmarks, and date and time within the image. Using this analysis result, the server prepares to generate appropriate questions relevant to the user.

[0768] Step 3:

[0769] The server generates appropriate questions using natural language processing techniques based on the extracted feature information. The generated questions are crafted in a format that makes it easy for the user to answer naturally. These questions are sent to the device, which then presents them to the user via voice or text.

[0770] Step 4:

[0771] The user responds to questions presented by the device using voice or text. The device sends this response to the server in real time. In the case of a voice response, it is transmitted as audio data.

[0772] Step 5:

[0773] The server analyzes the response received from the user using an emotion engine. The emotion engine recognizes the user's emotional state based on the tone of voice and the content of what is being said. For example, if the user's voice is cheerful, it detects a positive emotion.

[0774] Step 6:

[0775] The server considers the recognized emotional state and generates the next questions and comments. At this stage, adjustments are made according to the user's emotions. If the emotional state is positive, questions and comments are generated to continue the positive conversation. The generated content is then sent back to the device.

[0776] Step 7:

[0777] The device presents the user with newly generated questions and comments, facilitating the smooth continuation of the dialogue. The user responds, and the dialogue cycle continues.

[0778] Step 8:

[0779] The server saves data on the user's responses and emotional changes to a recording device. This allows the user to review the conversation history later and reflect on the changes in their emotions.

[0780] (Example 2)

[0781] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0782] To improve cognitive function in older adults, a system is needed that stimulates past memories and guides conversations according to emotions. However, conventional systems have shortcomings in recognizing emotions and generating appropriate dialogues, making it difficult to provide natural and effective cognitive stimulation for older adults.

[0783] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0784] In this invention, the server includes means for the user to select past images stored in information memory, means for analyzing the features of the selected images using image analysis technology, means for generating questions based on the analyzed features using natural language processing technology, and means for using emotion analysis technology to recognize the emotional state based on the generated questions. This enables the provision of interactive dialogue tailored to the user's emotions and memories, and activates the user's cognitive functions.

[0785] An "information processing device" refers to any electronic device used for inputting, processing, storing, and outputting data, which can be operated by the user through an interface.

[0786] "Information storage" refers to data storage that stores digital information and allows it to be retrieved as needed.

[0787] "Image analysis technology" refers to all technologies used to extract features and patterns from digital images and perform classification and recognition.

[0788] "Natural language processing technology" refers to the technology that enables computers to understand human language and process it automatically.

[0789] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate natural language sentences from input data.

[0790] "Emotional analysis technology" refers to technology that automatically analyzes and classifies a speaker's emotional state from audio and text information.

[0791] "Information recording" refers to the process of saving conversations and data so that they can be referenced later.

[0792] The following configuration and procedure are necessary to implement the invention. This invention is a system for stimulating the cognitive functions of the elderly and enriching their conversations, and its details are described below.

[0793] First, the user uses the terminal to select past images stored in the information memory. The terminal is equipped with input devices such as a touchscreen, keyboard, and mouse, and the user can operate it through the interface. The selected images are saved in standard formats such as JPEG and PNG.

[0794] Next, the terminal sends the selected image data to the server. The server uses image analysis techniques to extract features from this image. This analysis utilizes AI-based image processing libraries (e.g., OpenCV, TensorFlow). The feature information includes the date and time of capture, location, and information about objects and people in the image.

[0795] The server uses a generative AI model based on the extracted feature information and applies natural language processing techniques (e.g., the GPT model) to generate questions for the user. These questions are then presented to the user via the terminal.

[0796] For example, if a user selects a photo from a family trip, the server will generate a question such as, "Tell us about the most enjoyable moment from your family trip," based on the information obtained from that image. This prompt helps the user recall their memories.

[0797] The user responds to the generated questions using voice or text. The terminal sends the response to the server. The server uses sentiment analysis technology (e.g., IBM Watson Tone Analyzer) to analyze the user's tone of voice and text content to recognize the user's emotional state.

[0798] To provide symmetrical dialogue that reflects emotions, the server can adjust the dialogue based on the user's emotions and generate additional questions or comments. This allows the system to provide a valuable interactive experience for the user and support the activation of cognitive functions.

[0799] Furthermore, the history of the conversation and data on the user's emotional changes are stored in the information log, which the user can use to look back on it at a later date.

[0800] Thus, the present invention provides a system that supports the cognitive function of the elderly by combining image analysis technology, natural language processing technology, and sentiment analysis technology.

[0801] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0802] Step 1:

[0803] The user operates the device to select past images stored in the information memory. This selection is made via touch or mouse clicks, and the input image data is displayed on the device's screen. The images selected by the user are used as basic data for subsequent processing.

[0804] Step 2:

[0805] The terminal sends the selected image data to the server. The images are in formats such as JPEG or PNG, and this becomes the input data for the server's analysis system. After receiving the data, the server prepares to pass the images to the analysis tool.

[0806] Step 3:

[0807] The server extracts features from the received image data using image analysis techniques. Specifically, AI algorithms (e.g., OpenCV, TensorFlow) are applied to obtain information such as the date and time of capture, location, and objects or people within the image. This feature data forms the basis for the next questions generated.

[0808] Step 4:

[0809] The server uses natural language processing techniques with feature information obtained from image analysis as input. It utilizes a generative AI model (e.g., GPT model) to generate questions for the user. During this process, the output is adjusted to ensure it is easy for the user to answer. The generated questions are then sent to the terminal.

[0810] Step 5:

[0811] The terminal presents the user with a question sent from the server. The question is displayed as text on the screen and may be read aloud if necessary. At this stage, the user receives the question on the screen.

[0812] Step 6:

[0813] The user responds to the presented questions either verbally or in text. This response is recorded as input data on the device and, if verbal, is converted to text using speech recognition technology. This text data forms the basis for the next analysis step.

[0814] Step 7:

[0815] The terminal sends the user's response text to the server. Upon receiving this transmitted data, the server prepares to perform the next analysis.

[0816] Step 8:

[0817] The server applies sentiment analysis technology to analyze the emotional state of the user's responses. This analysis considers tone information obtained from the speech and the content of the text. The analysis results become important input data for generating the next dialogue.

[0818] Step 9:

[0819] The server uses the results of the emotional state analysis as input and generates adaptive additional questions and comments using a generative AI model. In this step, the generated content is adjusted according to the user's emotional state. The results are sent to the terminal, and the interactive dialogue with the user continues.

[0820] (Application Example 2)

[0821] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0822] Cognitive decline in older adults is often exacerbated by a lack of interpersonal interaction and stimulation of memories based on past experiences. Furthermore, eliciting emotions through conversation is difficult, and providing dialogue that responds to those emotions is even more challenging. Current technology does not adequately provide solutions that can appropriately understand a user's emotional state and provide dialogue accordingly. Therefore, there is a need for new methods that effectively stimulate cognitive function and richly evoke past memories and emotions.

[0823] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0824] In this invention, the server includes means for the user to select past images stored in a memory device, means for analyzing the characteristic information of the selected images, and means for adjusting the dialogue content based on the analyzed emotions. This makes it possible to stimulate the past memories of elderly people, understand their emotions, and provide more appropriate and enriching dialogue.

[0825] A "terminal" is a device used by a user to operate, and is equipped with storage and communication functions.

[0826] A "storage device" is a device for holding data, and is a medium for storing information such as images and text.

[0827] "Image analysis" is a technique for extracting feature information from selected images, and is the process of recognizing objects and photographic information within an image.

[0828] "Feature information" refers to data obtained through image analysis, including information about people, objects, time, and location within the image.

[0829] "Question generation" is the process of creating questions to present to users based on analyzed feature information.

[0830] "User response" refers to the user's reply to a presented question, which is information sent to the server as audio or text.

[0831] "Emotional analysis" is a technology that evaluates a user's emotional state based on their responses and tone of voice.

[0832] "Dialogue adjustment" is the process of dynamically changing the tone and content of a dialogue based on analyzed emotions.

[0833] "Providing interaction" is the process of offering additional dialogue and feedback tailored to the user's emotional state.

[0834] A "recording device" is a device for saving the content of conversations with users and is a medium for making past logs accessible.

[0835] The system that realizes this invention includes several main components. First, the terminal allows the user to select an image from its storage device to reminisce about past memories. The selected image is sent to a server, which uses image analysis technology to extract detailed feature information from the image. This feature information includes people and objects in the image, the date and time the image was taken, and the location where it was taken.

[0836] The server then uses natural language processing techniques based on the extracted feature information to generate questions that are easy for the user to answer. The generated questions are presented to the user through the terminal, and the user's voice or text response is sent back to the server.

[0837] Sentiment analysis is performed by an emotion engine based on the user's responses and tone of voice. This evaluates the user's emotional state, and the tone and content of the conversation are adjusted accordingly. For example, if the user is enjoying looking at an image, more positive questions and comments will be provided.

[0838] Furthermore, past conversations and emotional changes are saved in the recording device, allowing users to later review their emotional progression and conversation history. This supports the long-term activation of cognitive functions.

[0839] As a concrete example, when a user selects a family photo, the server asks a question such as, "What is the most memorable event from this photo?" The emotion engine analyzes the user's cheerful response and then asks follow-up questions such as, "Could you tell us more about that memory?"

[0840] An example of a prompt for a generative AI model is the question, "Output questions to guide the conversation about related events and emotions so that the model can describe the user's emotions associated with the image."

[0841] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0842] Step 1:

[0843] The user operates the device and selects a past image from its storage. The input is the user's instructions, and the output is the selected image. The user's action is to tap the image through the device's interface.

[0844] Step 2:

[0845] The terminal sends the selected image to the server. The input is the selected image, and the output is the completion of the transfer of the image data to the server. The terminal uploads the image data to the server using the network.

[0846] Step 3:

[0847] The server extracts feature information from received images using image analysis techniques. The input is image data, and the output is feature information (people, objects, date and time, location, etc.). The server applies image processing algorithms to identify important elements within the image.

[0848] Step 4:

[0849] The server analyzes the extracted feature information and uses natural language processing techniques to generate questions that are easy for the user to answer. The input is feature information, and the output is the generated questions. The server uses a generative AI model to construct highly relevant questions.

[0850] Step 5:

[0851] The generated question is sent to the device and presented to the user. The input is the generated question, and the output is the display of the question to the user. The device displays the question in the UI and prompts the user for a voice or text response.

[0852] Step 6:

[0853] The user responds to questions using voice or text. Input is the user's response, and output is the response data input to the terminal. The user inputs their answers using a microphone or keyboard.

[0854] Step 7:

[0855] The terminal sends the user's response to the server. The input is the response data from the user, and the output is the completion of data transmission to the server. The terminal processes the response data and forwards it to the server.

[0856] Step 8:

[0857] The server analyzes the user's response using sentiment analysis technology to recognize their emotional state. The input is the response data, and the output is the recognized emotional state. The server evaluates the text and tone of the response and uses an emotion engine to identify the emotion.

[0858] Step 9:

[0859] The server adjusts the dialogue content based on the recognized emotional state and generates new questions and comments. The input is the emotional state, and the output is the content of the next dialogue. The server then uses the generative AI model again to design an appropriate dialogue.

[0860] Step 10:

[0861] The refined dialogue is sent to the terminal to support further interaction with the user. The input is the newly generated dialogue, and the output is what is presented to the user. The terminal provides the user with a continuous conversation, facilitating communication.

[0862] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0863] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0864] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0865] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0866] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0867] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0868] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0869] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0870] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0871] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0872] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0873] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0874] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0875] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0876] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0877] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0878] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0879] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0880] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0881] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0882] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0883] The following is further disclosed regarding the embodiments described above.

[0884] (Claim 1)

[0885] A means for the user to select past images stored in the device's memory,

[0886] A means for analyzing the feature information of the selected image,

[0887] A means for generating questions based on analyzed feature information,

[0888] A means of presenting a generated question to the user and receiving the user's response,

[0889] A means of analyzing the user's response, generating additional questions based on the response, and presenting them to the user again,

[0890] A means for saving conversations with users to a recording device,

[0891] A system that includes this.

[0892] (Claim 2)

[0893] The system according to claim 1, comprising means for using image analysis techniques in analyzing the feature information of selected images.

[0894] (Claim 3)

[0895] The system according to claim 1, comprising means of using natural language processing technology when analyzing user responses.

[0896] "Example 1"

[0897] (Claim 1)

[0898] In an information terminal, a means for the user to select past visual data stored on a storage medium,

[0899] A means which is an information processing device that analyzes the feature information of selected visual data,

[0900] A means for generating questions using a generative artificial intelligence model based on analyzed feature information,

[0901] A means for presenting a generated question to the user via an information terminal and receiving the user's response,

[0902] A means of analyzing the user's response, generating additional questions based on the response, and presenting them to the user again,

[0903] A means of saving the content of conversations with users to a storage medium,

[0904] A system that includes this.

[0905] (Claim 2)

[0906] The system according to claim 1, which uses visual data analysis technology in analyzing the feature information of selected visual data.

[0907] (Claim 3)

[0908] The system according to claim 1, which uses natural language processing technology when analyzing user responses.

[0909] "Application Example 1"

[0910] (Claim 1)

[0911] On a terminal, a means for the user to select past visual information stored in the memory device,

[0912] A means for analyzing the feature information of selected visual information,

[0913] A means for generating queries based on analyzed feature information,

[0914] A means of presenting the generated query to the user and receiving the user's response,

[0915] A means of analyzing the user's response, generating additional inquiries based on the response, and presenting them to the user again,

[0916] A means for saving conversations with users to a recording device,

[0917] The means characterized in that the terminal is a portable information terminal,

[0918] A means for automatically generating queries in natural language form using a generation AI model based on the analyzed visual information,

[0919] A means for analyzing user responses using natural language processing technology and automatically generating related additional queries,

[0920] A system that includes this.

[0921] (Claim 2)

[0922] The system according to claim 1, which uses image analysis technology in the analysis of selected visual information.

[0923] (Claim 3)

[0924] The system according to claim 1, which uses natural language processing techniques when generating queries using a generative AI model.

[0925] "Example 2 of combining an emotion engine"

[0926] (Claim 1)

[0927] In an information processing device, a means for a user to select past images stored in information memory,

[0928] A means of using image analysis techniques to analyze the features of selected images,

[0929] A means for generating questions using natural language processing techniques based on analyzed features,

[0930] A means of presenting a generated question to the user and receiving the user's response,

[0931] A means of using emotion analysis technology to analyze the user's response based on the tone and content of their voice and recognize their emotional state,

[0932] A means of generating and re-presenting additional questions based on recognized emotions to the user,

[0933] A means of saving information in a record to remember interactions with users,

[0934] A system that includes this.

[0935] (Claim 2)

[0936] The system according to claim 1, comprising means for using an AI-based processing algorithm in analyzing the features of selected images.

[0937] (Claim 3)

[0938] The system according to claim 1, comprising means of using a generative AI model when analyzing user responses.

[0939] "Application example 2 when combining with an emotional engine"

[0940] (Claim 1)

[0941] A means for the user to select past images stored in the device's memory,

[0942] A means for analyzing the feature information of the selected image,

[0943] A means for generating questions based on analyzed feature information,

[0944] A means of presenting a generated question to the user and receiving the user's response,

[0945] A means of analyzing the user's response, generating additional questions based on the response, and presenting them to the user again,

[0946] A means for analyzing the user's emotional state and adjusting the content of the dialogue based on the analyzed emotions,

[0947] A means of providing further interaction based on the adjusted dialogue content,

[0948] A means for saving conversations with users to a recording device,

[0949] A system that includes this.

[0950] (Claim 2)

[0951] The system according to claim 1, comprising means for using image analysis techniques in analyzing the feature information of selected images.

[0952] (Claim 3)

[0953] The system according to claim 1, comprising means of using natural language processing technology and sentiment analysis technology when analyzing user responses. [Explanation of symbols]

[0954] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. On a terminal, a means for the user to select past visual information stored in the memory device, A means for analyzing the feature information of selected visual information, A means for generating queries based on analyzed feature information, A means of presenting the generated query to the user and receiving the user's response, A means of analyzing the user's response, generating additional inquiries based on the response, and presenting them to the user again, A means for saving conversations with users to a recording device, The means characterized in that the terminal is a portable information terminal, A means for automatically generating queries in natural language form using a generation AI model based on the analyzed visual information, A means for analyzing user responses using natural language processing technology and automatically generating related additional queries, A system that includes this.

2. The system according to claim 1, which uses image analysis technology in the analysis of selected visual information.

3. The system according to claim 1, which uses natural language processing technology when generating queries using a generative AI model.

Images