System

Abstract

To provide a system.SOLUTION: A system includes: a device for interacting with an aged person; first means for identifying a feeling state of the aged person from a voice of the aged person using a feeling recognition engine; second means for receiving the voice and text data which are input by the aged person; third means for generating a first prompt sentence for outputting a response suitable for the feeling state and input contents on the basis of the feeling state and the input contents of the aged person; fourth means for generating the response by inputting the generated first prompt sentence into a generative AI model; and fifth means for providing the aged person with the generated response.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In order for elderly people to obtain information about end-of-life planning, they often need the assistance of a person with specialized knowledge. However, human supporters are not always available, and because end-of-life planning is a sensitive topic, many elderly people are reluctant to interact with humans. Under these circumstances, how can we ensure that elderly people can obtain appropriate information about end-of-life planning? [Means for solving the problem]

[0005] We provide a system that combines a robot that can converse with the elderly with generative AI. This system provides information about end-of-life planning through dialogue with the elderly. The robot has the ability to recognize emotions and generate gestures, creating a sense of familiarity with the elderly. The generative AI generates natural conversations, empathizes with what the elderly are saying, and asks questions, thereby drawing their interest. It also provides all kinds of information about end-of-life planning, such as recording memories and wills, sorting out belongings and preparing for funerals, and consulting on inheritance and taxes. This allows the elderly to obtain appropriate information about end-of-life planning. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] As one embodiment of the present invention, a robot that can interact with elderly people is provided. This robot has an emotion recognition function and can recognize the emotional state of the elderly person. It also has a function of generating gestures, which allows it to create a friendly relationship with the elderly person. Specifically, when the elderly person expresses joy, the robot generates gestures that match the joy.

[0029] "Example 2"

[0030] In addition, a generative AI is provided. This generative AI has the ability to generate natural conversations, empathize with what the elderly person is saying, and ask questions. Specifically, when an elderly person talks about their past memories, the generative AI generates an empathetic response based on that content. Also, when an elderly person expresses a question about end-of-life planning, the generative AI generates an answer to that question.

[0031] "Example 3"

[0032] Furthermore, the system has the function of providing information on end-of-life planning, such as recording memories and wills, sorting out belongings, preparing for funerals, and consulting on inheritance and taxes. Specifically, if an elderly person requests to record a will, the system will record the contents of the will and play it back as needed. Also, if an elderly person requests information on sorting out belongings, the system will provide information on how to sort out belongings and important points to note.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The robot that interacts with the elderly is activated.

[0036] Step 2: The robot recognizes the emotional state of the elderly person using its emotion recognition function. For example, if the elderly person smiles, the robot recognizes the smile and determines that the elderly person is happy.

[0037] Step 3: The robot uses its gesture generation function to create a friendly atmosphere with the elderly. For example, if the elderly person expresses joy, the robot will generate a gesture that matches the joy.

[0038] "Example 2"

[0039] Step 1: The generative AI is launched.

[0040] Step 2: The generative AI listens to what the elderly person says and generates natural conversations based on that content. For example, if the elderly person talks about their past memories, the generative AI generates empathetic responses based on that content.

[0041] Step 3: When an elderly person expresses a question about end-of-life planning, the generative AI generates an answer to that question. For example, if an elderly person asks for information about sorting out their belongings, the generative AI will provide information on how to sort out their belongings and what to be careful about.

[0042] "Example 3"

[0043] Step 1: When an elderly person requests information about end-of-life planning, the system will provide the corresponding information. For example, if an elderly person requests to record a will, the system will record the contents of the will and play it back as needed.

[0044] Step 2: If an elderly person requests information about sorting out their belongings, the system will provide them with information on how to sort out their belongings and what to be careful about.

[0045] Step 3: If the elderly person requests information about other end-of-life matters, such as funeral arrangements, inheritance, or tax advice, the system will provide that information.

[0046] Example 1

[0047] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0048] In order for elderly people to live their daily lives without feeling lonely, it is important to provide appropriate information and dialogue that is sensitive to their emotions. However, with conventional robot systems, it was difficult to accurately recognize the emotional state of the elderly and generate natural dialogue and gestures accordingly. In addition, the provision of information and consultation functions regarding end-of-life planning was insufficient. As a result, it was not possible to alleviate the loneliness and anxiety felt by elderly people.

[0049] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0050] In this invention, the server includes a robot means for interacting with the elderly, an emotion recognition means, a means for generating gestures and movements, a generative AI means for generating natural conversations, a means for empathizing with what the elderly is saying and asking questions, a means for recording memories and wills, a means for providing information on end-of-life planning such as sorting out belongings, preparing for a funeral, and consultations on inheritance and taxes, and a means for recognizing the emotional state of the elderly, generating appropriate gestures and movements based on the recognized emotions, and executing the generated gestures and movements. This enables conversations and gestures that are sensitive to the emotions of the elderly, and also enhances the information and consultation functions on end-of-life planning.

[0051] A "robot that interacts with the elderly" is an automated device that can have natural conversations with the elderly.

[0052] "Emotion recognition" is a technology that analyzes facial expressions and voice data of elderly people to identify their emotional state (joy, sadness, anger, etc.).

[0053] "Gesture generation" is a technology that allows a robot to generate appropriate actions (such as waving or smiling) based on recognized emotions.

[0054] "Generative AI" is an artificial intelligence technology that generates natural conversations with the elderly.

[0055] "Empathizing and asking questions" is a technique for deepening dialogue by showing empathy for what the elderly person is saying and asking appropriate questions.

[0056] "Memories and Wills Recording" is a technology that stores the memories and wills of elderly people in digital format.

[0057] "Providing information on end-of-life planning" refers to technology that provides elderly people with information related to end-of-life planning, such as sorting out personal belongings, funeral preparations, inheritance, and tax consultations.

[0058] "Recognizing the emotional state, generating appropriate gestures based on the recognized emotions, and executing the generated gestures" refers to a series of processes in which the robot analyzes the emotions of the elderly person and performs actions in accordance with those emotions.

[0059] MODE FOR CARRYING OUT THE INVENTION

[0060] This invention relates to a robot system that interacts with the elderly, and includes functions such as emotion recognition, gesture generation, generative AI that generates natural conversations, the ability to empathize with and ask questions about what the elderly are saying, the ability to record memories and wills, and the ability to provide information about end-of-life planning.

[0061] 1. Program Generation

[0062] The server generates a program for the robot to interact with the elderly. This program includes an emotion recognition function and a gesture generation function. The emotion recognition function is implemented using, for example, "TENSORFLOW (registered trademark)." The gesture generation function is implemented using known technology.

[0063] 2. Program processing explanation

[0064] The server analyzes data acquired from the camera and microphone to realize the emotion recognition function. Specifically, it analyzes facial expressions in facial images acquired from the camera and analyzes audio data acquired from the microphone using TensorFlow. This makes it possible to recognize the emotional state of the elderly.

[0065] The terminal (robot) generates appropriate gestures based on the emotion information received from the server. The robot has movement patterns corresponding to the emotion, and selects and executes one of them.

[0066] 3. Examples and prompts

[0067] As a concrete example, consider the case where an elderly person says, "I'm very happy today." In this case, the server analyzes the voice input and recognizes that the elderly person is feeling happy. Next, the device (robot) generates gestures that match the emotion of joy (for example, waving, smiling, etc.), creating a friendly atmosphere toward the elderly person.

[0068] Example prompts to input to a generative AI model:

[0069] "What gestures should a robot make when an elderly person expresses joy?"

[0070] Using this prompt, the generative AI model can suggest appropriate gestures based on the elderly person's emotions.

[0071] In this way, the present invention can realize conversations and gestures that are sensitive to the feelings of the elderly, and can also enhance the provision of information and consultation functions regarding end-of-life planning.

[0072] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0073] Step 1:

[0074] The server acquires facial images and voice data of elderly people from the camera and microphone. The input is facial image data from the camera and voice data from the microphone, and the output is a state ready for analyzing these data. Specifically, the server initializes the camera and microphone devices and acquires data in real time.

[0075] Step 2:

[0076] The server performs facial expression analysis on the acquired facial image data. The input is facial image data acquired from the camera, and the output is the emotional state (happiness, sadness, anger, etc.) based on the elderly person's facial expression. Specifically, the server extracts feature points from the facial image and inputs them into an emotion recognition model to estimate the emotional state.

[0077] Step 3:

[0078] The server analyzes the acquired voice data using TensorFlow and estimates emotions from the tone of voice and the content of the words. The input is voice data acquired from a microphone, and the output is the emotional state based on the voice. Specifically, the server preprocesses the voice data and inputs it into a voice recognition model to estimate the emotional state.

[0079] Step 4:

[0080] The server integrates the results of facial expression analysis and voice analysis to recognize the overall emotional state of the elderly person. The input is the results of facial expression analysis and voice analysis, and the output is the overall emotional state of the elderly person. Specifically, the server weights and integrates the results of both analyses to determine the final emotional state.

[0081] Step 5:

[0082] The terminal (robot) generates appropriate gestures based on the emotional information received from the server. The input is the emotional information from the server, and the output is the generated gesture movement sequence. As a specific movement, the terminal selects a movement pattern according to the emotion and generates a motor control signal to execute this.

[0083] Step 6:

[0084] The terminal (robot) executes the generated gestures. The input is a motion sequence, and the output is the actual robot motion. Specific motions are controlled by the terminal, which controls motors and actuators to perform actions such as waving or smiling.

[0085] Step 7:

[0086] The user (elderly person) watches the robot's movements and continues the dialogue. The input is the robot's movements and dialogue content, and the output is the elderly person's new utterances and reactions. In terms of specific actions, the elderly person expresses their emotions in response to the robot's movements and continues the next dialogue.

[0087] (Application example 1)

[0088] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] When elderly people visit a physical store, it can be difficult for them to receive guidance within the store or product explanations. In particular, in situations where a response that corresponds to their emotional state is required, conventional guidance systems often do not respond adequately. Furthermore, in order for elderly people to use stores with peace of mind, it is important to respond in a friendly and empathetic manner. To solve these issues, a system is needed that can recognize the emotional state of the elderly and provide appropriate gestures and dialogue content.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0091] In this invention, the server includes a robot means for interacting with the elderly, an emotion recognition function means, a function means for generating gestures, a generative AI means for generating natural conversations, a function means for empathizing with what the elderly is saying and asking questions, a function means for recording memories and wills, a function means for sorting belongings and preparing for a funeral, a function means for providing information on end-of-life planning such as consultations on inheritance and taxes, and a function means that is installed in a physical store and provides store guidance and product explanations while using gestures according to the emotional state of the elderly. This enables elderly people to receive friendly service and appropriate guidance according to their emotional state when they visit a physical store.

[0092] A "robot that interacts with the elderly" is an automated mechanical device that can have natural conversations with the elderly.

[0093] "Emotion recognition function" is a technology that identifies the emotional state of an elderly person from their facial expressions, tone of voice, etc.

[0094] The "function to generate gestures and movements" is a technology that generates appropriate gestures and movements according to the recognized emotional state.

[0095] "Generative AI that generates natural conversations" is an artificial intelligence technology that generates natural conversations when interacting with the elderly.

[0096] The "function of empathizing with and asking questions about what the elderly are saying" refers to the technology of showing empathy for what the elderly are saying and asking appropriate questions.

[0097] "Recording memories and wills" is a function that records the memories and wills of elderly people.

[0098] "Sorting out belongings and preparing for funerals" is a function that assists elderly people in sorting out their belongings and preparing for funerals.

[0099] The "function to provide information on end-of-life planning, such as consultations on inheritance and taxes" is a function that provides information for elderly people to consult on inheritance and taxes related to end-of-life planning.

[0100] "A function that is installed in a physical store and provides store guidance and product explanations while using gestures that correspond to the emotional state of the elderly" refers to technology that is installed in a physical store and provides store guidance and product explanations while using gestures that correspond to the emotional state of the elderly.

[0101] The system for implementing this invention is centered around a robot that interacts with elderly people. The system includes the following main hardware and software:

[0102] 1. Hardware:

[0103] Camera: Used to capture the facial expressions of the elderly.

[0104] Robot body: A mechanical device for interacting and gesturing.

[0105] Microphone: Used to capture the voice of the elderly person.

[0106] Speaker: Used to output the generated audio.

[0107] 2. Software:

[0108] A library for capturing and processing camera footage.

[0109] A deep learning framework to load emotion recognition models and make predictions.

[0110] A library used as a speech synthesis engine to conduct voice-based dialogue with the elderly.

[0111] The system works as follows:

[0112] 1. Emotion recognition:

[0113] The camera captures the elderly person's facial expressions and converts them into grayscale.

[0114] The image is resized to 48x48 pixels and fed into the emotion recognition model.

[0115] Obtain emotion labels from the model's prediction results.

[0116] 2. Gesture generation:

[0117] Depending on the recognized emotional state, appropriate gestures are generated.

[0118] For example, if the emotion of joy is recognized, the robot will make gestures that express joy.

[0119] 3. Voice interaction:

[0120] Generative AI is used to generate dialogue content that corresponds to the emotional state of the elderly person.

[0121] The generated dialogue content is output as voice.

[0122] To illustrate, consider the following scenario:

[0123] When an elderly person enters the store, the robot will say aloud, "Hello! I'll make a gesture of joy. Is there anything I can help you with?"

[0124] If an elderly person looks at a product with joy, the robot will explain, using happy gestures, "That product is very popular!"

[0125] Examples of prompts to input to a generative AI model include:

[0126] "Generate gestures that an elderly person would make if they were happy."

[0127] "Generate gestures for when an elderly person is surprised."

[0128] This means that when elderly people visit a physical store, they will be able to receive friendly service and appropriate guidance that is tailored to their emotional state.

[0129] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0130] Step 1:

[0131] The camera captures the facial expressions of the elderly.

[0132] Input: Camera image

[0133] Output: Captured image frames

[0134] Specific operation: The camera captures the elderly person's face in real time and acquires video data.

[0135] Step 2:

[0136] The server converts the captured image frames to grayscale.

[0137] Input: Captured image frames

[0138] Output: Grayscale image

[0139] Specific operation: Converts a color image to a grayscale image.

[0140] Step 3:

[0141] The server resizes the grayscale image to 48x48 pixels.

[0142] Input: Grayscale image

[0143] Output: Resized image

[0144] Specific behavior: Resizes a grayscale image to 48x48 pixels.

[0145] Step 4:

[0146] The server inputs the resized image into the emotion recognition model.

[0147] Input: Resized image

[0148] Output: Emotion label

[0149] Specific operation: The resized image is input into the emotion recognition model to predict the emotion label.

[0150] Step 5:

[0151] The server generates appropriate gestures based on the emotion labels.

[0152] Input: Emotion label

[0153] Output: Gesture instructions

[0154] Specific actions: Generate gestures corresponding to the emotion labels and send instructions to the robot.

[0155] Step 6:

[0156] The robot executes the generated gesture.

[0157] Input: Gesture instructions

[0158] Output: Gesture movement

[0159] Specific behavior: The robot follows instructions and performs appropriate gestures.

[0160] Step 7:

[0161] The server generates dialogue content according to the emotional state of the elderly person.

[0162] Input: Emotion label

[0163] Output: Dialogue content

[0164] Specific operation: Using generative AI, natural dialogue content is generated according to the emotional state of the elderly person.

[0165] Step 8:

[0166] The server inputs the generated dialogue content into a speech synthesis engine.

[0167] Input: Dialogue

[0168] Output: Audio data

[0169] Specific operation: Convert the generated dialogue content into voice data.

[0170] Step 9:

[0171] The robot plays the audio data.

[0172] Input: Audio data

[0173] Output: Audio output

[0174] Specific operation: The generated voice data is played back through the robot's speaker, and the conversation content is conveyed to the elderly.

[0175] Example 2

[0176] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0177] When communicating with the elderly, it is necessary to effectively recognize emotions, respond with empathy, and provide information about end-of-life planning. However, conventional systems have difficulty providing these functions in an integrated manner, and there is a lack of dialogue systems that the elderly can use with confidence. Furthermore, technology for generating appropriate responses to the elderly's input has not been sufficiently developed. This makes it difficult to alleviate the loneliness and anxiety felt by the elderly.

[0178] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a device for interacting with the elderly, emotion recognition means, gesture generation means, generative AI means for generating natural conversation, means for empathizing with what the elderly is saying and asking questions, means for recording memories and wills, means for providing information on end-of-life planning such as sorting belongings, funeral preparations, and inheritance and tax consultations, means for receiving text data input by a user, generating and inputting prompt sentences to the generative AI model, and means for displaying the response generated by the generative AI model to the user. This makes it possible to realize natural conversation with the elderly and to provide emotion recognition, empathetic responses, and information on end-of-life planning in an integrated manner.

[0179] A "device for interacting with the elderly" is a device that includes hardware and software that allows the elderly to interact with the elderly through voice and text.

[0180] "Emotion recognition means" is a technology for analyzing and recognizing the emotional state of elderly people from their voices, facial expressions, and text.

[0181] "Means for generating gestures" refers to technology that enables robots and avatars to generate natural gestures when interacting with elderly people.

[0182] "Generative AI means for generating natural conversations" refers to artificial intelligence technology for generating natural and fluent conversations in dialogue with the elderly.

[0183] "Means for empathizing with and asking questions about what the elderly say" refers to technology for generating empathetic responses and appropriate questions in response to what the elderly say.

[0184] "Mechanisms for recording memories and wills" is a technology for digitally recording the memories and wills spoken by elderly people.

[0185] "Means of providing information on end-of-life planning, such as sorting out belongings, preparing for funerals, and consulting on inheritance and taxes" refers to technology that allows elderly people to obtain various information on end-of-life planning.

[0186] "Means for receiving text data entered by a user and generating and inputting a prompt sentence to a generative AI model" refers to a technology for receiving text data entered by a user and generating a prompt sentence to appropriately input the text data to a generative AI model.

[0187] "Means for displaying to the user the response generated by the generative AI model" refers to technology for visually or audibly displaying to the user the response generated by the generative AI model.

[0188] MODE FOR CARRYING OUT THE INVENTION

[0189] This invention is a system for realizing natural dialogue with elderly people, and integrates emotion recognition, generation of empathetic responses, provision of information on end-of-life planning, etc. Specific embodiments of this system are described below.

[0190] System configuration

[0191] The system consists of the following main components:

[0192] 1. Devices for interacting with the elderly: These are devices that include hardware and software that allow elderly people to interact with the elderly through voice and text. Examples include smartphones, tablets, and dedicated conversation robots.

[0193] 2. Emotion recognition: This technology analyzes and recognizes the emotional state of elderly people from their voices, facial expressions, and text. For example, it uses voice analysis software and facial expression recognition cameras.

[0194] 3. Means of generating gestures: This technology allows robots and avatars to generate natural gestures when interacting with the elderly. For example, motion capture technology is used.

[0195] 4. Generative AI means for generating natural conversations: This is an artificial intelligence technology for generating natural and fluent conversations with elderly people. For example, OpenAI's GPT-4 (registered trademark) is used as the generative AI model.

[0196] 5. Means of empathizing with and asking questions about what the elderly say: Techniques for generating empathetic responses and appropriate questions in response to what the elderly say.

[0197] 6. Recording of memories and wills: Technologies for digitally recording the memories and wills spoken by older people, for example, using cloud storage services.

[0198] 7. Means of providing information on end-of-life planning: This is technology for providing information on end-of-life planning, such as sorting out belongings, preparing for a funeral, and consulting on inheritance and taxes.

[0199] 8. A means for receiving text data entered by a user and generating and inputting prompt sentences to a generative AI model: A technology for receiving text data entered by a user and generating prompt sentences to appropriately input it into a generative AI model.

[0200] 9. Means for displaying responses generated by a generative AI model to the user: A technology for visually or audibly displaying responses generated by a generative AI model to the user.

[0201] System Operation

[0202] The user (elderly person) uses the dialogue device to input text data. For example, the user might input, "I miss the memories of traveling when I was young." The device then sends this text data to the server. The server analyzes the received text data and generates a prompt sentence to input into the generative AI model. For example, it generates a prompt sentence such as, "The user is talking about memories of traveling when they were young. What kind of response would you generate?"

[0203] A generative AI model (e.g., GPT-4) generates an appropriate response based on this prompt, such as "Which place is most memorable?" The server sends the generated response to the device, which then displays it to the user.

[0204] Specific examples

[0205] As a concrete example, consider an elderly person talking about past memories. If the user inputs, "I have fond memories of the trips I took when I was younger," the generative AI will generate an empathetic response such as, "Which place left the most lasting impression on you?" Similarly, if an elderly person expresses a question about end-of-life planning, for example, by inputting, "What should I do to start preparing for the end of my life?", the generative AI will generate an answer such as, "It would be a good idea to start by writing a will and organizing your assets."

[0206] Example prompt sentence:

[0207] "I have fond memories of traveling when I was young."

[0208] "How do I start planning my end?"

[0209] In this way, the system can realize natural conversations with the elderly, and can comprehensively recognize emotions, respond with empathy, and provide information on end-of-life planning.

[0210] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0211] Step 1:

[0212] The user inputs text data using the terminal. For example, the user inputs "I miss the memories of traveling when I was young." The input text data is stored in the memory of the terminal.

[0213] Step 2:

[0214] The terminal sends the input text data to the server. Specifically, the terminal converts the text data into JSON format and sends it to the server using an HTTP POST request. The input is the user's text data, and the output is the data to be sent to the server.

[0215] Step 3:

[0216] The server receives text data sent from the terminal. The received data is stored in the server's memory. The input is the text data from the terminal, and the output is the data stored in the server's memory.

[0217] Step 4:

[0218] The server analyzes the received text data and generates a prompt to be input to the generative AI model. For example, if the received data is "I miss the memories of traveling when I was young," the prompt would be "The user is talking about their memories of traveling when they were young. What response would you like to generate?" The input is the text data stored on the server, and the output is the generated prompt.

[0219] Step 5:

[0220] The server inputs a prompt to the generative AI model. The generative AI model (e.g., GPT-4) generates an appropriate response based on the prompt. For example, it generates the response, "Which place is most memorable?" The input is the generated prompt, and the output is the response generated by the generative AI model.

[0221] Step 6:

[0222] The server sends the generated response to the terminal. Specifically, it converts the generated response into JSON format and sends it to the terminal as an HTTP response. The input is the response generated by the generative AI model, and the output is the data sent to the terminal.

[0223] Step 7:

[0224] The terminal receives the response data sent from the server and displays it to the user. For example, the terminal displays the message "Which place is the most memorable?" on the screen. The input is the response data from the server, and the output is the message displayed to the user.

[0225] In this way, the system can realize natural conversations with the elderly, and can comprehensively recognize emotions, respond with empathy, and provide information on end-of-life planning.

[0226] (Application example 2)

[0227] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0228] When elderly people spend time alone at home, it is necessary to provide an environment where they can live with peace of mind, reducing feelings of loneliness and anxiety. Also, a system that can respond quickly in emergencies is necessary. Furthermore, it is important to make it easy for elderly people to obtain information about end-of-life planning.

[0229] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a device means for interacting with the elderly, an emotion recognition means, a means for generating gestures and movements, a generative AI means for generating natural conversations, a means for empathizing with the elderly's story and asking questions, a means for recording memories and wills, a means for sorting belongings and preparing for a funeral, a means for providing information on end-of-life planning such as inheritance and tax consultations, and a means for automatically sending notifications in the event of an emergency. This provides an environment where the elderly can live in peace and enables rapid response in the event of an emergency. It also makes it easy to obtain information on end-of-life planning.

[0230] A "device for interacting with the elderly" is a device that includes hardware and software for conducting natural conversations with the elderly.

[0231] "Emotion recognition means" is a technology for analyzing and recognizing the emotions of elderly people from their facial expressions and tone of voice.

[0232] The "means for generating gestures and movements" is a technology that generates natural gestures and movements during conversations to facilitate communication with the elderly.

[0233] "Generative AI means for generating natural conversations" is an artificial intelligence technology for generating natural responses in dialogue with the elderly.

[0234] "Means for empathizing with and asking questions about what the elderly are saying" is a technology for generating empathetic responses and appropriate questions to what the elderly are saying.

[0235] "Means for recording memories and wills" is a technology for recording the memories and wills of elderly people in digital format.

[0236] "Tools for sorting out belongings and preparing for funerals" refers to technology that provides information and support for elderly people regarding sorting out belongings and preparing for funerals.

[0237] "Means of providing information on end-of-life planning, such as consultations on inheritance and taxes" refers to technology that provides information to help elderly people resolve questions about inheritance and taxes.

[0238] "Means for automatically sending notifications in emergencies" refers to technology that automatically sends notifications to family members and emergency contacts when an elderly person experiences an emergency.

[0239] This invention is a system for providing an environment where elderly people can live safely at home. The system includes a device for interacting with elderly people, emotion recognition means, gesture generation means, generative AI means for generating natural conversations, means for empathizing with elderly people's stories and asking questions, means for recording memories and wills, means for sorting belongings and preparing funerals, means for providing information on end-of-life planning such as consultations on inheritance and taxes, and means for automatically sending notifications in the event of an emergency.

[0240] System Program

[0241] The system program is configured as follows:

[0242] 1. Devices for interacting with the elderly:

[0243] It receives text and voice input from the elderly person and sends it to a generative AI.

[0244] The response from the generative AI is returned to the elderly person.

[0245] 2. Emotion recognition means:

[0246] It analyzes the facial expressions and tone of voice of elderly people to recognize their emotional state.

[0247] The recognition results are fed back to the generative AI to adjust the content of the dialogue.

[0248] 3. Means of generating gestures:

[0249] It generates natural gestures and hand movements during conversations with elderly people, facilitating communication.

[0250] 4. Generative AI methods for generating natural conversations:

[0251] Generate natural responses based on the elderly person's input.

[0252] Empathize with what the elderly are saying and generate appropriate questions.

[0253] 5. Means of recording memories and wills:

[0254] Record the memories and wills of the elderly in digital form.

[0255] 6. Arranging belongings and funeral arrangements:

[0256] Provides information and support for elderly people sorting out their belongings and preparing for funerals.

[0257] 7. Means of providing information on end-of-life planning, such as inheritance and tax consultations:

[0258] Provides information to help seniors resolve questions about inheritance and taxes.

[0259] 8. Means of sending automatic notifications in case of emergency:

[0260] When an elderly person has an emergency, notifications are automatically sent to family members and emergency contacts.

[0261] Hardware and software used

[0262] Hardware: Devices such as smartphones, tablets, and computers.

[0263] Software: Python®, OpenAI API.

[0264] Processing flow

[0265] 1. The elderly person talks to the system using a smartphone or tablet.

[0266] 2. The system receives the elderly person's input and analyzes their emotional state using emotion recognition means.

[0267] 3. The analysis results are fed back to the generative AI to generate natural conversations.

[0268] 4. Return the generated response to the elderly person and generate gestures as needed.

[0269] 5. If an elderly person has questions about end-of-life planning, provide them with relevant information.

[0270] 6. Automatically notify family members and emergency contacts in case of an emergency.

[0271] Specific examples

[0272] Prompt Sentence Examples

[0273] Elderly person: I went on a trip with my family a long time ago, and I have fond memories of that time.

[0274] Example of the response generated

[0275] AI: That's a wonderful memory! Where did you travel to? What was the most memorable experience?

[0276] In this way, generative AI empathizes with what the elderly person is saying and asks questions to expand on the conversation, allowing the elderly person to continue the conversation with a sense of security.

[0277] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0278] Step 1:

[0279] A user speaks to the system using a smartphone or tablet. The input can be in the form of voice or text, and the device receives it. In the case of voice input, the device uses speech recognition software to convert it to text. The output is the user input in the form of text.

[0280] Step 2:

[0281] The terminal transmits the user's input text to the emotion recognition means. The emotion recognition means analyzes the user's input text and recognizes the user's emotional state. Specifically, it uses natural language processing techniques to classify the emotion of the text. The input is the user's text input, and the output is the recognized emotional state.

[0282] Step 3:

[0283] The device sends the recognized emotional state and the user's input text to the generative AI means. The generative AI means generates natural conversation based on this information. Specifically, it uses a generative AI model (e.g., OpenAI GPT-3 (registered trademark)) to generate empathetic responses and appropriate questions. The input is the user's text input and emotional state, and the output is the generated response text.

[0284] Step 4:

[0285] The device returns the generated response text to the user. If speech output is required, the device uses text-to-speech software to convert text to speech. The input is the generated response text, and the output is the response returned to the user.

[0286] Step 5:

[0287] When a user asks a question about end-of-life planning, the device uses a generative AI means to provide relevant information. The generative AI means generates appropriate end-of-life planning information based on the user's question. The input is the user's question text, and the output is the relevant end-of-life planning information.

[0288] Step 6:

[0289] In an emergency, if the user enters keywords such as "emergency" or "help," the device will automatically activate a means to send an emergency notification. Specifically, it uses SMTP to send a notification to family members and emergency contacts. The input is the user's emergency keyword, and the output is the sent emergency notification.

[0290] Example 3

[0291] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0292] There is a need for systems that allow elderly people to efficiently and appropriately obtain information about end-of-life planning, record memories and wills, organize belongings, prepare for funerals, and smoothly navigate complex procedures such as inheritance and tax consultations. However, existing systems have difficulty providing these functions in an integrated manner, forcing elderly people to use multiple systems to obtain the information they need. Furthermore, usability has been reduced due to the lack of dialogue that responds to the elderly's emotional state and the lack of natural conversation generation using speech recognition and generative AI models.

[0293] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0294] In this invention, the server includes a device means for interacting with the elderly person, an emotion recognition function means, a function means for generating gestures and movements, a generative AI means for generating natural conversations, a function means for empathizing with what the elderly person is saying and asking questions, a function means for recording memories and wills, a function means for providing information on end-of-life planning such as sorting out belongings, preparing for a funeral, and consulting about inheritance and taxes, a means for converting speech to text using speech recognition software, a means for storing the text data in a database, a means for generating answers to user questions using a generative AI model, and a means for converting text data to speech using speech synthesis software. This allows the elderly person to efficiently obtain all information related to end-of-life planning through a single system and, through natural conversations tailored to their emotions, smoothly carry out complex procedures such as recording memories and wills, sorting out belongings, preparing for a funeral, and consulting about inheritance and taxes.

[0295] The "device means for interacting with the elderly" is a device that provides an interface for the elderly to interact with the system through voice or text.

[0296] The "emotion recognition function means" is a function that analyzes and recognizes the emotional state of an elderly person from their voice, facial expressions, gestures, etc.

[0297] The "functional means for generating gestures" is a function that allows a virtual character or robot to generate gestures so that the system can achieve natural communication in dialogue with the elderly.

[0298] "Generative artificial intelligence means for generating natural conversation" refers to artificial intelligence technology for generating natural and fluent conversation in dialogue with elderly people.

[0299] "Functional means of empathizing with and asking questions about what the elderly are saying" refers to the function of deepening dialogue by showing empathy for what the elderly are saying and asking appropriate questions.

[0300] The "means for recording memories and wills" is a function that allows elderly people to record and save their memories and wills in audio or text format.

[0301] "Functional means for providing information on end-of-life planning, such as sorting out personal belongings, preparing for a funeral, and consulting on inheritance and taxes" is a function that allows elderly people to obtain various information related to end-of-life planning.

[0302] "Means for converting speech to text using speech recognition software" refers to speech recognition technology for converting the senior's speech input into text data.

[0303] "Means for storing text data in a database" refers to database technology for safely storing the converted text data.

[0304] "Means for generating answers to user questions using a generative AI model" refers to an artificial intelligence model for generating appropriate answers to user questions.

[0305] "Means for converting text data into speech using speech synthesis software" refers to speech synthesis technology for converting text data into speech and playing it back to the user.

[0306] This invention is a system that enables elderly people to efficiently obtain information about end-of-life planning and smoothly carry out complex procedures such as recording memories and wills, sorting out belongings, preparing for funerals, and receiving advice on inheritance and taxes. A specific embodiment of this system is described below.

[0307] System configuration

[0308] The system consists of the following main components:

[0309] 1. Devices and means for communicating with the elderly

[0310] 2. Emotion recognition function means

[0311] 3. Functional means for generating gestures

[0312] 4. Generative AI methods for generating natural conversations

[0313] 5. Functional means of empathizing with and asking questions about what the elderly are saying

[0314] 6. A means of recording memories and wills

[0315] 7. Functional means for providing information on end-of-life planning, such as sorting out belongings, preparing for a funeral, and consulting on inheritance and taxes.

[0316] 8. Using speech recognition software to convert speech to text

[0317] 9. How to store text data in a database

[0318] 10. How to use generative AI models to generate answers to user questions

[0319] 11. Means of converting text data into speech using speech synthesis software

[0320] Hardware and software used

[0321] Devices: smartphones, tablets, computers, etc.

[0322] Speech Recognition Software: Popular Speech Recognition APIs

[0323] Database: A general database system (e.g., MySQL (registered trademark))

[0324] Generative AI model: A general generative AI model (e.g., OpenAI's GPT-3)

[0325] Speech synthesis software: General speech synthesis APIs

[0326] Program processing

[0327] Recording and reproducing wills

[0328] 1. When a user wishes to record a will, they access the system using a smartphone or tablet. For example, the user may say, "I want to record a will."

[0329] 2. The device uses speech recognition software to convert the user's speech into text, and also provides a text input interface that allows the user to input text directly.

[0330] 3. The server saves the converted text data or the directly entered text data in the database, along with metadata such as the user ID and timestamp.

[0331] 4. If the user wishes to replay the contents of the will, they can access the system again using their terminal and send a request saying, "I would like to replay the will."

[0332] 5. The server searches the database for the user's will data and sends the found data to the terminal.

[0333] 6. The device uses speech synthesis software to convert the text data into speech and play it back to the user.

[0334] Providing information on sorting belongings

[0335] 1. When a user wants information about sorting out belongings, they access the system using a tablet or smartphone. For example, the user types in the text, "Please tell me how to sort out belongings."

[0336] 2. The terminal provides an interface for users to input their questions. When the user inputs a question and presses the send button, the question is sent to the server.

[0337] 3. The server uses the generative AI model to generate an appropriate answer to the user's question. The generative AI model analyzes the content of the question and creates an answer based on related information.

[0338] 4. The server sends the generated answer to the device, optionally including related links and additional information along with the text data of the answer.

[0339] 5. The terminal displays the received answer to the user, who can review the answer and ask for more detailed information if necessary.

[0340] Examples of specific examples and prompts

[0341] Recording of wills

[0342] The user uses a smartphone and says, "I want to record my will."

[0343] The device uses speech recognition software to convert the speech into text and send it to the server.

[0344] The server stores the text data in a database.

[0345] Providing information on sorting belongings

[0346] The user uses a tablet to input into the system, "Please tell me how to organize the belongings of the deceased."

[0347] The server uses a generative AI model to generate answers such as, "When sorting through belongings, it is recommended that you first identify important documents and valuables, and then sort and dispose of unnecessary items."

[0348] The terminal displays the generated answer to the user.

[0349] In this way, the system responds to the user's needs regarding end-of-life planning. The flow of the specification process in the third embodiment will be described with reference to FIG.

[0350] Recording and reproducing wills

[0351] Recording of wills

[0352] Step 1:

[0353] When a user wishes to record a will, they access the system using a smartphone or tablet, for example by saying "I want to record a will."

[0354] Input: User's voice commands

[0355] Output: Audio data

[0356] Step 2:

[0357] The terminal uses speech recognition software to convert the user's speech into text.

[0358] Input: Audio data

[0359] Output: Text data

[0360] Step 3:

[0361] The terminal also provides a text input interface, allowing the user to input text directly.

[0362] Input: User text input

[0363] Output: Text data

[0364] Step 4:

[0365] The server saves the converted text data or the directly entered text data in a database, along with metadata such as the user ID and timestamp.

[0366] Input: Text data, user ID, timestamp

[0367] Output: Data stored in the database

[0368] Regeneration of a will

[0369] Step 1:

[0370] If the user wishes to replay the contents of the will, he or she accesses the system again using the terminal and sends a request saying, "I would like to replay the will."

[0371] Input: User request

[0372] Output: Request data

[0373] Step 2:

[0374] The server searches the database for the corresponding user's will data and transmits the found data to the terminal.

[0375] Input: Request data, User ID

[0376] Output: Will data

[0377] Step 3:

[0378] The terminal uses speech synthesis software to convert the text data into speech and play it back to the user.

[0379] Input: Will data

[0380] Output: Audio data

[0381] Providing information on sorting belongings

[0382] Step 1:

[0383] When a user wants information about sorting out belongings, they access the system using a tablet or smartphone. For example, the user can enter the text "Tell me how to sort out belongings."

[0384] Input: User question

[0385] Output: Question data

[0386] Step 2:

[0387] The terminal provides an interface for the user to input a question, and when the user inputs a question and presses the send button, the question is sent to the server.

[0388] Input: Question data

[0389] Output: Request to server

[0390] Step 3:

[0391] The server uses a generative AI model to generate appropriate answers to user questions. The generative AI model analyzes the content of the question and creates an answer based on related information.

[0392] Input: Question data

[0393] Output: Response data

[0394] Step 4:

[0395] The server then sends the generated answer to the device, which may include related links and additional information along with the text data of the answer.

[0396] Input: Answer data

[0397] Output: Response data to the terminal

[0398] Step 5:

[0399] The terminal displays the received answer to the user, who can then review the displayed answer and request more detailed information if necessary.

[0400] Input: Answer data

[0401] Output: Data displayed to the user

[0402] In this way, the system responds to users' end-of-life needs.

[0403] (Application example 3)

[0404] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0405] When elderly people prepare for the end of their lives, they need to centrally manage and safely store a wide range of information, such as recording wills, sorting belongings, and consulting about inheritance and taxes. However, because this information is highly private and security is important, it is difficult to manage it properly. In addition, it is a great burden for elderly people to organize this information themselves, so efficient support is required.

[0406] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes an automatic device means for interacting with the elderly, an emotion recognition function means, a function means for generating gestures and movements, a generative AI means for generating natural conversations, a function means for empathizing with what the elderly is saying and asking questions, a function means for recording memories and wills, a function means for providing information on end-of-life planning such as sorting out belongings, funeral preparations, and consultations on inheritance and taxes, a function means for converting voice input into text and saving it, and a function means for saving the contents of wills in JSON format. This enables the elderly to centrally manage and safely save information on end-of-life planning.

[0407] "Automatic device means" refers to a mechanical device for interacting with the elderly, and has the function of communicating with the elderly through voice input, output, and movement.

[0408] The "emotion recognition function means" has the function of analyzing and recognizing the emotional state of the elderly person from their voice, facial expression, movements, etc.

[0409] The "functional means for generating gestures and hand movements" has a function for generating natural gestures and hand movements during conversation to facilitate communication with the elderly.

[0410] "Generative AI means" refers to a system that uses artificial intelligence technology to generate natural conversations and has the functionality to naturally engage in dialogue with the elderly.

[0411] The "empathy and questioning function means" has the function of deepening dialogue by showing empathy for what the elderly person is saying and asking appropriate questions.

[0412] A "means for recording memories and wills" is a device that has the function of recording the contents of memories and wills spoken by elderly people and playing them back as needed.

[0413] "End-of-life planning information provision function means" is a means that has the function of providing information regarding end-of-life planning, such as sorting out belongings, preparing for a funeral, and consulting about inheritance and taxes.

[0414] The "voice input text conversion means" has a function for converting the voice spoken by the elderly person into text data.

[0415] The "text storage means" has a function for safely storing the converted text data.

[0416] A "JSON format storage means" is a means that has the function of saving the contents of a will in JSON format.

[0417] The system for implementing this invention includes an automated device that interacts with the elderly, an emotion recognition function, a function to generate gestures and movements, a generative AI that generates natural conversations, a function to empathize with what the elderly is saying and ask questions, a function to record memories and wills, and a function to provide information on end-of-life planning such as sorting belongings, preparing for a funeral, and consulting about inheritance and taxes, a function to convert voice input into text and save it, and a function to save the contents of a will in JSON format.

[0418] Hardware and software used

[0419] Hardware: Smartphone (microphone, storage)

[0420] Software: Python

[0421] System Operation

[0422] 1. Voice input: The user speaks their will into the smartphone's microphone. The smartphone captures the voice and saves it as audio data.

[0423] 2. Speech recognition: Using a library on the smartphone, the captured voice data is converted into text, using a specified API to perform highly accurate speech recognition.

[0424] 3. Text storage: The converted text data is saved in JSON format in the smartphone's storage, ensuring the contents of the will are stored securely.

[0425] Specific examples

[0426] For example, if a user says, "My will is to leave all my assets to my son," the smartphone captures this speech and uses the library to convert it into text, which becomes "My will is to leave all my assets to my son," and stores it in JSON format.

[0427] Prompt Sentence Examples

[0428] "I want to record my will. I want to say, 'My will is to leave all my assets to my son.'"

[0429] In this way, elderly people can centrally manage and safely store information related to end-of-life planning.

[0430] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0431] Step 1:

[0432] The user speaks their will into the smartphone's microphone. The input is the user's voice, and the smartphone captures this voice. Specifically, the smartphone's microphone collects the voice data and temporarily stores it in its internal memory.

[0433] Step 2:

[0434] The device converts the captured voice data into text using the library. The input is voice data and the output is text data. Specifically, the device sends voice data and receives text data.

[0435] Step 3:

[0436] The device saves the converted text data in JSON format. The input is text data, and the output is a JSON format file. Specifically, the device converts the text data into JSON format and saves it in local storage.

[0437] Step 4:

[0438] The device checks the contents of the saved JSON file and notifies the user that the save is complete. The input is a JSON file and the output is a notification message. Specifically, the device loads the JSON file, checks its contents, and then displays a notification message to the user.

[0439] Step 5:

[0440] The user can play back the saved will as needed. The input is the user's playback request, and the output is the display of text data or audio playback. Specifically, the device reads the saved JSON file and displays the text data or plays it back using speech synthesis technology.

[0441] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0442] "Example 1"

[0443] In one embodiment of the present invention, a robot that interacts with an elderly person includes an emotion engine that recognizes the user's emotions. This emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc., and conveys this information to the robot. The robot adjusts the content of the interaction based on this information. For example, if the user expresses joy, the robot will engage in a dialogue that matches that joy. On the other hand, if the user expresses sadness, the robot will engage in an empathetic dialogue.

[0444] "Example 2"

[0445] In another embodiment of the present invention, the generative AI tailors the dialogue based on the user's emotional state. To do this, the generative AI uses information from the emotion engine. For example, if the user expresses joy, the generative AI generates dialogue that matches that joy. Alternatively, if the user expresses sadness, the generative AI generates empathetic dialogue.

[0446] "Example 3"

[0447] In yet another embodiment of the present invention, a robot and a generative AI work together to respond to a user's emotions. Specifically, the robot uses an emotion engine to recognize the user's emotions and communicates that information to the generative AI. The generative AI then adjusts the dialogue content based on that information. For example, if the user expresses joy, the robot will use gestures that match that joy, and the generative AI will generate dialogue that matches that joy.

[0448] The processing flow of each embodiment will be described below.

[0449] "Example 1"

[0450] Step 1: The robot starts a dialogue with the user.

[0451] Step 2: The emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0452] Step 3: The emotion engine communicates this information to the robot.

[0453] Step 4: The robot adjusts the dialogue based on this information.

[0454] "Example 2"

[0455] Step 1: The generative AI begins a dialogue with the user.

[0456] Step 2: The emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0457] Step 3: The emotion engine communicates this information to the generative AI.

[0458] Step 4: The generative AI adjusts the dialogue based on this information.

[0459] "Example 3"

[0460] Step 1: The robot starts a dialogue with the user.

[0461] Step 2: The emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0462] Step 3: The emotion engine communicates this information to the robot and generative AI.

[0463] Step 4: The robot adjusts its gestures based on this information, and the generative AI adjusts the dialogue based on this information.

[0464] Example 1

[0465] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0466] When interacting with the elderly, it is necessary to recognize their emotions appropriately and generate natural conversations and gestures accordingly. It is also important to provide information and record-keeping support for elderly people regarding end-of-life planning. However, conventional systems often lack sufficient functionality in terms of emotion recognition accuracy, natural conversation, and provision of end-of-life information. For this reason, there is a need for a conversational robot system that the elderly can use with confidence.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0468] In this invention, the server includes a means for capturing the user's facial expressions and voice, a means for recognizing the user's emotions using an emotion engine, and a means for transmitting the recognized emotion information to the robot. This makes it possible to accurately recognize the emotions of the elderly and generate natural conversations and gestures based on those emotions. Furthermore, the system can also provide information about end-of-life planning and support for recording, making it possible to provide an interactive robot system that the elderly can use with peace of mind.

[0469] A "robot that interacts with the elderly" is an autonomous mechanical device designed to engage in natural dialogue with the elderly.

[0470] "Emotion recognition means" is a technology for analyzing and recognizing emotions from a user's facial expressions, tone of voice, choice of words, etc.

[0471] The "means for generating gestures" is a technology that enables a robot to make appropriate gestures according to the user's emotions and the content of the conversation.

[0472] "Generative AI means for generating natural conversations" refers to artificial intelligence technology that generates appropriate responses in order to have natural conversations with users.

[0473] "Methods for empathizing with and asking questions about what the elderly are saying" are techniques for showing empathy for what the elderly are saying and asking appropriate questions.

[0474] "Means for recording memories and wills" refers to technology for recording the memories and wills of elderly people.

[0475] "Means of providing information on end-of-life planning" refers to technology for providing information on end-of-life planning, such as sorting out belongings, preparing for a funeral, and consulting on inheritance and taxes.

[0476] "Means for capturing the user's facial expressions and voice" refers to technology for acquiring the user's facial expressions and voice using a camera or microphone.

[0477] "Means for recognizing a user's emotions using an emotion engine" refers to a technology that analyzes acquired facial and voice data to recognize a user's emotions.

[0478] The "means for transmitting the recognized emotional information to the robot" is a technology for transmitting the emotional information recognized by the emotion engine to the robot.

[0479] The "means for generating dialogue content based on emotional information" is a technology for generating appropriate dialogue content based on the recognized emotional information.

[0480] "Means for generating gestures that match emotions" is a technology that enables a robot to make appropriate gestures based on the recognized emotional information.

[0481] "Means for generating appropriate dialogue content using a generative AI model" refers to technology for using a generative AI model to generate appropriate responses in dialogue with a user.

[0482] This invention relates to a robot system that interacts with the elderly, and includes functions such as emotion recognition, gesture generation, generative AI that generates natural conversations, the ability to empathize with and ask questions about what the elderly are saying, and the ability to record memories and wills and provide information about end-of-life planning.

[0483] Hardware and software used

[0484] This system uses the following hardware and software:

[0485] Robot body: An autonomous mechanical device for interacting with the elderly. Equipped with a camera, microphone, speaker, motor, etc.

[0486] Emotion engine: Software that analyzes and recognizes emotions from a user's facial expressions, tone of voice, choice of words, etc.

[0487] Generative AI: An artificial intelligence technology for generating natural conversations. It uses generative AI models to generate appropriate dialogue content.

[0488] Server: A computer system that runs the emotion engine and transmits the recognized emotion information to the robot.

[0489] Program processing

[0490] The server acquires data from the robot's camera and microphone using a means of capturing the user's facial expressions and voice. The acquired data is analyzed using an emotion engine to recognize the user's emotions. The recognized emotion information is then sent from the server to the robot.

[0491] The robot generates dialogue content based on the emotional information it receives. Using generative AI, it engages in natural dialogue that reflects the user's emotions. For example, if the user expresses joy, the robot will respond with, "That's wonderful!" The robot also generates gestures that match the user's emotions, creating a friendly atmosphere for the user.

[0492] Specific examples

[0493] As a concrete example, consider the case where the user says, "I'm very happy today." In this case, the following occurs:

[0494] 1. The user says to the robot, "I'm very happy today."

[0495] 2. The robot captures the user's smile with a camera and records their voice with a microphone.

[0496] 3. The server sends the captured data to the emotion engine, which then recognizes the emotion as "joy."

[0497] 4. The server sends the information "The user is happy" to the robot.

[0498] 5. The robot generates the dialogue "That's great!"

[0499] 6. The robot generates gestures that express joy, such as waving.

[0500] 7. The robot will say "That's great!" and make a waving motion.

[0501] Prompt Sentence Examples

[0502] Examples of prompts to input to a generative AI model might include:

[0503] If the user says, "I'm very happy today," how should the robot respond and what gestures should it make?

[0504] Using this prompt, the generative AI model can suggest appropriate dialogue content and gestures.

[0505] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0506] Step 1:

[0507] The user talks to the robot.

[0508] Input: User's voice and facial expressions.

[0509] Specific action: The user speaks to the robot, saying, "I'm very happy today."

[0510] Step 2:

[0511] The robot captures the user's facial expressions and voice.

[0512] Input: User's voice and facial expressions.

[0513] Output: Captured voice and facial expression data.

[0514] Specific operation: The robot's camera takes a picture of the user's face and the microphone records the user's voice.

[0515] Step 3:

[0516] The server uses an emotion engine to recognize the user's emotion.

[0517] Input: Captured voice and facial expression data.

[0518] Output: Recognized emotion information.

[0519] Specific operation: The server sends the captured data to the emotion engine, which then recognizes "happiness" from the user's tone of voice and facial expression.

[0520] Step 4:

[0521] The server transmits the recognized emotional information to the robot.

[0522] Input: Recognized emotion information.

[0523] Output: Emotion information sent to the robot.

[0524] Specific operation: The server sends the information "The user is happy" to the robot.

[0525] Step 5:

[0526] The robot generates dialogue content based on emotional information.

[0527] Input: Emotion information sent to the robot.

[0528] Output: The generated dialogue.

[0529] What it does: The robot uses a generative AI model to generate the dialogue "That's great!"

[0530] Step 6:

[0531] The robot generates gestures that match its emotions.

[0532] Input: Emotion information sent to the robot.

[0533] Output: The generated gesture.

[0534] Specific actions: The robot generates gestures that express joy, such as waving.

[0535] Step 7:

[0536] The robot will interact with the user and perform gestures.

[0537] Input: Generated dialogue and gestures.

[0538] Output: Dialogue and gesture execution for the user.

[0539] Specific action: The robot waves its hand while saying "That's great!"

[0540] (Application example 1)

[0541] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0542] For elderly people to live safely at home, it is necessary to properly recognize their emotional state and respond quickly when abnormal emotional states are detected. However, current systems lack the functionality to recognize the emotional state of elderly people in real time and to conduct appropriate dialogue or emergency contact, making it difficult to fully ensure the safety of elderly people.

[0543] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a robot means for interacting with the elderly, an emotion recognition function means, and a function means for generating gestures. This makes it possible to recognize the emotional state of the elderly in real time, to conduct an appropriate dialogue when an abnormal emotional state is detected, and to send a notification to an emergency contact if necessary.

[0544] A "robot that interacts with the elderly" is an automated device that can have natural conversations with the elderly.

[0545] The "emotion recognition function" is a function that recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0546] The "function of generating gestures and movements" is a function that allows the robot to generate gestures and movements according to the emotional state of the user.

[0547] "Generative AI that generates natural conversations" is an artificial intelligence technology that allows for natural dialogue with users.

[0548] The "function of empathizing with and asking questions about what the elderly are saying" refers to the function of showing empathy for what the elderly are saying and asking appropriate questions.

[0549] "Recording memories and wills" is a function that records the memories and wills of elderly people.

[0550] "Sorting out belongings and preparing for funerals" is a function that assists elderly people in sorting out their belongings and preparing for funerals.

[0551] The "function to provide information on end-of-life planning, such as consultations on inheritance and taxes" is a function that provides information on inheritance and taxes to support elderly people in their end-of-life planning.

[0552] "A function that recognizes the emotional state of the elderly, engages in appropriate dialogue when an abnormal emotional state is detected, and sends a notification to an emergency contact if necessary" is a function that recognizes the emotional state of the elderly in real time, engages in appropriate dialogue when an abnormal emotional state is detected, and sends a notification to an emergency contact if necessary.

[0553] To implement the present invention, the following system configuration and program are required.

[0554] System Configuration

[0555] 1. Hardware:

[0556] Camera: Used to capture the facial expressions and movements of the elderly in real time.

[0557] Robot body: A device for interacting with the elderly and making gestures.

[0558] Emergency Contact System: A system for sending notifications to emergency contacts when abnormal emotional states are detected.

[0559] 2. Software:

[0560] Library A: A library for video processing.

[0561] Library B: A library for emotion recognition.

[0562] Library C: A library for controlling the robot's movements.

[0563] System A: A system for sending notifications to emergency contacts.

[0564] Program processing

[0565] The server acquires video footage from the camera in real time and uses Library B to recognize the emotional state of the elderly person. Library C is used to generate the robot's behavior and dialogue content depending on the recognized emotion. For example, if the elderly person is feeling happy, the robot will use gestures that match the joy and say, "It's a nice day today!" On the other hand, if the elderly person is feeling scared, the robot will say, "Are you okay? Is there anything I can help you with?" and at the same time use System A to send a notification to the emergency contact that "The elderly person is feeling scared."

[0566] Specific examples

[0567] For example, if an elderly person suddenly feels frightened while alone at home, the camera will capture their facial expression and Library B will recognize the emotion of fear. The robot will then ask, "Are you OK? Is there anything I can help you with?", and the emergency contact system will simultaneously send a notification to the emergency contact. In this way, the safety of the elderly can be ensured.

[0568] Prompt Sentence Examples

[0569] Generate a program for a security robot that recognizes the emotional state of an elderly person in real time, and if they feel fear or anxiety, will engage in appropriate dialogue and notify emergency contacts.

[0570] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0571] Step 1:

[0572] The server acquires video from the camera in real time. The input is the video data from the camera and the output is video frames, which are used in subsequent processing steps.

[0573] Step 2:

[0574] The server passes the acquired video frames to Library B for emotion recognition. The input is the video frames, and the output is the recognized emotion data. Specifically, emotions are analyzed from facial expressions, tone of voice, choice of words, etc.

[0575] Step 3:

[0576] The server uses Library C to generate robot behavior and dialogue content based on the recognized emotion data. The input is emotion data, and the output is the robot's movement instructions and dialogue content. For example, if the emotion of joy is recognized, the robot will make gestures that match the joy and say, "Today is a nice day!"

[0577] Step 4:

[0578] If the server recognizes an abnormal emotional state (e.g., fear or anxiety), it uses System A to send a notification to the emergency contact. The input is the abnormal emotional data, and the output is a notification to the emergency contact. Specifically, it sends a message saying, "The elderly person is feeling fearful."

[0579] Step 5:

[0580] The server executes the robot's actions and dialogue. The input is the robot's action instructions and dialogue content, and the output is the robot's actual actions and dialogue. Specifically, the robot uses gestures and dialogue appropriately.

[0581] Example 2

[0582] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0583] In dialogue with the elderly, there are issues with insufficient emotional recognition and empathetic response generation. Furthermore, information about end-of-life planning may not be provided appropriately. Furthermore, there is also the problem of difficulty in adjusting the content of the dialogue according to the user's emotional state.

[0584] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a device means for interacting with the elderly, an emotion recognition means, a means for generating gestures and movements, a generative AI means for generating natural conversations, a means for empathizing with the elderly and asking questions, a means for recording memories and wills, a means for providing information on end-of-life planning such as sorting belongings, funeral preparations, and inheritance and tax consultations, a means for receiving user input, a means for preprocessing data, a means for analyzing the user's emotional state using an emotion recognition engine, a means for generating a response based on the user's input and emotional state using a generative AI model, and a means for providing the generated response to the user. This improves emotion recognition and empathetic response generation in dialogue with the elderly, enabling appropriate provision of information on end-of-life planning. It also facilitates adjusting the content of the dialogue according to the user's emotional state.

[0585] The "device means for interacting with the elderly" is a device including hardware and software for interacting with the elderly through voice and text.

[0586] "Emotion recognition means" is a technology for analyzing and identifying emotions from a user's voice or text.

[0587] The "means for generating gestures" is a technology for generating and displaying natural gestures and hand movements during a conversation.

[0588] "Generative AI means for generating natural conversations" refers to artificial intelligence technology for generating natural conversations based on user input.

[0589] "Means for empathizing with and asking questions about what the elderly are saying" is a technique for generating empathetic responses and questions to what the elderly are saying.

[0590] "Means for recording memories and wills" refers to technology for recording and preserving the memories and wills of the elderly.

[0591] "Means of providing information on end-of-life planning, such as sorting out personal belongings, preparing for a funeral, and consulting on inheritance and taxes" refers to technology for providing information related to end-of-life planning.

[0592] The "means for receiving user input" refers to technology for receiving voice or text data input by the user.

[0593] "Means for preprocessing data" refers to technology for converting received data into a format that is easy to analyze.

[0594] The "means for analyzing the emotional state of a user using an emotion recognition engine" is a technique for identifying the emotional state of a user using an emotion recognition engine.

[0595] "Means for generating responses based on a user's input and emotional state using a generative AI model" refers to technology for using a generative AI model to generate responses appropriate to a user's input and emotional state.

[0596] The "means for providing the generated response to the user" refers to a technique for providing the generated response to the user by display or audio.

[0597] This invention relates to a system for interacting with elderly people, and includes technologies for recognizing emotions, generating empathetic responses, and providing information about end-of-life planning. Specific embodiments of this system are described below.

[0598] Hardware and Software Configuration

[0599] server:

[0600] Hardware: A server with a powerful GPU (e.g., a powerful graphics processing unit)

[0601] Software: Natural language processing libraries (e.g., natural language processing libraries), emotion recognition engines (e.g., emotion recognition engines)

[0602] Device:

[0603] Hardware: Smartphone, tablet, or dedicated interactive device

[0604] Software: speech recognition software (e.g., voice recognition software), speech synthesis software (e.g., speech synthesis software)

[0605] System Operation

[0606] Receiving user input:

[0607] The user inputs voice or text data through the terminal, for example, by speaking into the microphone of a smartphone or by typing text on a keyboard.

[0608] Data preprocessing:

[0609] The server converts the received data into text format, denoising and tokenising it, and the audio data is converted to text using speech recognition software.

[0610] Emotion recognition:

[0611] The server uses an emotion recognition engine to analyze the user's emotional state. For example, the server inputs text data into the emotion recognition engine to identify the user's emotional state (e.g., joy, sadness, anger, etc.).

[0612] Conversation generation:

[0613] The server uses a generative AI model to generate a response based on the user's input and emotional state. The generative AI model receives a prompt and generates an appropriate response.

[0614] Sending a response:

[0615] The server sends the generated response to the terminal for delivery to the user, and the terminal displays the generated response to the user, or uses speech synthesis software to convert the text to speech if a voice response is required.

[0616] Specific examples

[0617] Example 1: Elderly people talking about past memories

[0618] User input: "I went on a trip with my family a long time ago and it was so much fun."

[0619] Generative AI response: "That's a wonderful memory. Where did you travel?"

[0620] Example 2: When an elderly person expresses doubts about end-of-life planning

[0621] User input: "I don't know where to start when it comes to end-of-life planning."

[0622] Generative AI response: "As a first step in preparing for your end, we recommend that you organize your assets and important documents."

[0623] Prompt Sentence Examples

[0624] "When a user talks about a past memory, generate an empathetic response."

[0625] "When a user expresses a question about end-of-life planning, generate an answer to that question."

[0626] In this way, the server, terminal, and user each play their respective roles, and the entire system operates smoothly, making conversation with the elderly more natural and empathetic.

[0627] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0628] Step 1: Receiving User Input

[0629] Description: The user inputs voice or text data through the terminal.

[0630] Input: User voice or text data

[0631] Output: Audio or text data sent to the device

[0632] What it does: A user speaks into the microphone of their smartphone or tablet or enters text on the keyboard, and the device sends this data to a server.

[0633] Step 2: Preprocessing the data

[0634] Description: Converts data received by the server into text format, denoising and tokenising it.

[0635] Input: Voice or text data sent from the device

[0636] Output: Preprocessed text data

[0637] What it does: The server uses speech recognition software to convert the audio data into text, then filters out unwanted noise from the text and breaks it into tokens for words and phrases.

[0638] Step 3: Emotion Recognition

[0639] Description: The server uses an emotion recognition engine to analyze the user's emotional state.

[0640] Input: Preprocessed text data

[0641] Output: User's emotional state (e.g., happy, sad, angry, etc.)

[0642] Specific operation: The server inputs the text data into an emotion recognition engine to identify the user's emotional state. For example, the emotion recognition engine analyzes the text data and identifies that the user is feeling "joy."

[0643] Step 4: Conversation generation

[0644] Description: The server uses a generative AI model to generate responses based on the user's input and emotional state.

[0645] Input: User input and emotional state

[0646] Output: The generated response text

[0647] Specific operation: The server inputs a prompt sentence into the generative AI model, which generates a response appropriate to the user's emotional state and input. For example, if a user inputs, "I once went on a trip with my family and it was so much fun," the generative AI model will generate a response such as, "That's a wonderful memory. Where did you go on that trip?"

[0648] Step 5: Sending a Response

[0649] Description: The server sends the generated response to the terminal and provides it to the user.

[0650] Input: Generated response text

[0651] Output: The response provided to the user

[0652] What happens: The server sends the generated text response to the device, which displays it to the user. If a voice response is required, the device uses speech synthesis software to convert the text to speech.

[0653] (Application example 2)

[0654] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0655] When interacting with elderly people, there is a need for systems that can generate natural conversations and respond to what the elderly say through empathy and questions. Furthermore, there is a need for systems that can adjust the content of the conversation based on the elderly person's emotional state. Furthermore, there is a lack of methods to make interactions with elderly customers more natural and empathetic in brick-and-mortar stores.

[0656] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0657] In this invention, the server includes an automatic device means for interacting with the elderly, an emotion recognition function means, a gesture generation function means, a generative AI means for generating natural conversations, a function means for empathizing with what the elderly is saying and asking questions, a function means for recording memories and wills, a function means for sorting belongings and preparing for a funeral, a function means for providing information on end-of-life planning such as inheritance and tax consultations, an application means installed in smart glasses worn by staff in a physical store, a means for acquiring elderly customer utterances, a means for recognizing the emotional state from the customer's utterances, a means for generating natural responses based on the user's utterances and emotional state, and a means for displaying the generated responses on the display of the smart glasses. This makes it possible to make interactions with elderly customers in a physical store more natural and empathetic.

[0658] An "automatic device" is a mechanical device designed to interact with an older adult.

[0659] The "emotion recognition function" is a function that identifies the emotional state of a user from their speech and facial expressions.

[0660] The "function of generating gestures" is a function that generates natural movements and gestures during a conversation.

[0661] "Generative AI" is an artificial intelligence that generates natural conversations, empathizes with what the elderly are saying, and asks questions.

[0662] The "empathic function" is the ability to generate empathic responses to what the elderly person is saying.

[0663] The "question function" is a function that generates appropriate questions in response to what the elderly person is saying.

[0664] "Means for recording memories and wills" are means for recording the memories and wills of elderly people.

[0665] "Means for sorting belongings" are means for providing information and support regarding sorting belongings.

[0666] "Funeral preparation tools" are tools that provide information and support regarding funeral preparations.

[0667] "Inheritance and tax consultation means" refers to means for providing consultation and information regarding inheritance and taxes.

[0668] A "brick and mortar store" is a store that exists in a physical location.

[0669] "Smart glasses" are glasses-type devices equipped with a display that displays information.

[0670] An "application means" is software for performing a specific function.

[0671] "Means for acquiring statements" refers to a means for acquiring statements from elderly customers.

[0672] The "emotional state recognition means" is a means for recognizing the emotional state of a customer from their speech.

[0673] The "response generation means" is a means for generating natural responses based on the user's utterances and emotional state.

[0674] The "response display means" is a means for displaying the generated response on the display of the smart glasses.

[0675] This invention is a system for making interactions with elderly customers in brick-and-mortar stores more natural and empathetic. The system is implemented using an application installed on smart glasses.

[0676] System configuration

[0677] The system includes the following hardware and software:

[0678] Hardware: Smart glasses (e.g., Google Glass®)

[0679] Software: OpenAI API, Emotion Recognition Library

[0680] Program processing explanation

[0681] The server implements the system using the following means:

[0682] 1. Speech acquisition method: Speech from elderly customers is acquired through the microphone in the smart glasses, and the user's speech is input into the system in real time.

[0683] 2. Emotional State Recognition: Recognizes the emotional state from customer utterances using the Emotion Recognition library, which identifies the user's emotional state and provides data to generate an appropriate response.

[0684] 3. Response generation: Using the OpenAI API, natural responses are generated based on the user's utterances and emotional state. The generative AI model uses the prompt sentence to generate an appropriate response.

[0685] 4. Response display means: The generated response is displayed on the smart glasses display, allowing staff to provide natural dialogue to elderly customers.

[0686] Specific examples

[0687] For example, if an elderly customer is talking about past travels, the following dialogue is possible:

[0688] User says: "I once went on a trip to Kyoto with my family."

[0689] Generative AI response: "That's a wonderful memory. What part of Kyoto is most memorable to you?"

[0690] Prompt Sentence Examples

[0691] Below are some example prompts that the generative AI uses to generate responses:

[0692] User's emotional state: joy

[0693] User says: I once went on a trip to Kyoto with my family.

[0694] Generative AI response:

[0695] In this way, interactions with elderly customers in brick-and-mortar stores can be made more natural and empathetic.

[0696] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0697] Step 1:

[0698] The server receives the elderly customer's speech through the microphone in the smart glasses. The input is the elderly customer's voice, and the output is text-format speech data. Specifically, it uses voice recognition software to convert the speech into text.

[0699] Step 2:

[0700] The server inputs the acquired utterance data into the Emotion Recognition library to recognize the emotional state. The input is text-format utterance data, and the output is the emotional state (e.g., joy, sadness, surprise, etc.). Specifically, emotions are identified using a text analysis algorithm.

[0701] Step 3:

[0702] The server inputs prompt sentences into the generative AI model based on the utterance data and the recognized emotional state, and generates natural responses. The input is the utterance data and the emotional state, and the output is the generated response text. Specifically, the following prompt sentences are input into the generative AI model:

[0703] User's emotional state: joy

[0704] User says: I once went on a trip to Kyoto with my family.

[0705] Generative AI response:

[0706] Step 4:

[0707] The server displays the generated response text on the display of the smart glasses. The input is the generated response text, and the output is the text displayed on the display of the smart glasses. Specifically, the text is displayed using display control software of the smart glasses.

[0708] Step 5:

[0709] The user (staff member) checks the response displayed on the smart glasses display and engages in a natural dialogue with the elderly customer. The input is the response text displayed on the display, and the output is the dialogue with the elderly customer. Specifically, the staff member reads out the displayed text or uses it as a reference to proceed with the dialogue.

[0710] In this way, interactions with elderly customers in brick-and-mortar stores can be made more natural and empathetic.

[0711] Example 3

[0712] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0713] When elderly people are preparing for the end of their lives, they need a system that can provide centralized support for recording and playing back wills, providing information on sorting belongings, and providing dialogue that responds to their emotions. However, in conventional systems, these functions are distributed, which means that users have to use multiple systems, which is inconvenient. In addition, the system lacks the functionality to provide dialogue that responds to the elderly's emotions, limiting the user experience.

[0714] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes a robot means for interacting with the elderly, an emotion recognition means, a means for generating gestures and movements, a generative AI means for generating natural conversations, a means for empathizing with the elderly and asking questions, a means for recording memories and wills, a means for providing information on end-of-life planning such as sorting out belongings, funeral preparations, and inheritance and tax consultations, a means for saving the contents of the will in a database, a means for playing back the saved will, a means for searching and providing information on sorting out belongings from the database, a means for generating dialogue content based on the user's emotions, and a means for performing gestures and movements in accordance with the generated dialogue content. This allows a single system to meet the diverse needs of the elderly regarding end-of-life planning and provides a richer user experience through dialogue tailored to emotions.

[0715] "Elderly" refers to people who are older and who especially need to prepare for the end of their lives.

[0716] "Interactive robot means" refers to a mechanical device for conducting natural conversations with elderly people.

[0717] "Emotion recognition means" refers to technology for detecting and recognizing the emotional state of an elderly person.

[0718] "Means for generating gestures" refers to the functionality that enables the robot to make appropriate gestures while interacting with the elderly.

[0719] "Generative AI means" refers to artificial intelligence technology for generating natural conversations with the elderly.

[0720] "Means of empathizing and asking questions" refers to the function of showing empathy for what the elderly person is saying and asking appropriate questions.

[0721] "Means for recording memories and wills" refers to technology that allows elderly people to record their memories and wills.

[0722] "Means of providing information about end-of-life planning" refers to the function of providing elderly people with information related to end-of-life planning, such as sorting out personal belongings, preparing for a funeral, and consulting about inheritance and taxes.

[0723] "Means for storing the contents of a will in a database" refers to technology for storing the contents of a will entered by an elderly person in a database.

[0724] "Means for reproducing a stored will" refers to the function for reproducing the contents of a will stored in a database.

[0725] "Means for searching and providing information on sorting out belongings from a database" refers to technology for searching information on sorting out belongings from a database and providing it to elderly people.

[0726] "Means for generating dialogue content based on the user's emotions" refers to a technology for generating appropriate dialogue content according to the user's emotional state.

[0727] "Means for making gestures that match the content of the generated dialogue" refers to a function that enables the robot to make appropriate gestures in accordance with the content of the generated dialogue.

[0728] MODE FOR CARRYING OUT THE INVENTION

[0729] This invention is a system that provides comprehensive support for elderly people in preparing for the end of life by providing them with the information and dialogue they need. The system is realized using hardware and software such as robots, emotion recognition functions, generative AI, and databases.

[0730] Hardware and software used

[0731] Robot: A mechanical device for interacting with the elderly, with the ability to recognize emotions and generate gestures.

[0732] Emotion Recognition Engine: Software for detecting and recognizing the emotional state of older adults.

[0733] Generative AI: Artificial intelligence technology for generating natural conversations.

[0734] Database: Data storage for storing information about the contents of wills and end-of-life planning.

[0735] Terminal: A device (e.g., tablet, smartphone) that provides an interface for the elderly to interact with the system.

[0736] Program processing

[0737] 1. If a user wishes to record a will, they enter the contents of the will into the system via their terminal.

[0738] 2. The server stores the entered will in a database.

[0739] 3. If the user wishes to regenerate the will, he / she sends a request to the system through his / her terminal.

[0740] 4. The server searches for the saved contents of the will and plays it on the device.

[0741] 5. If the user wishes to request information about estate clearance, he or she sends a request to the system via the terminal.

[0742] 6. The server searches the database for information regarding sorting of belongings and provides it to the terminal.

[0743] 7. The robot uses an emotion engine to recognize the user's emotions and conveys that information to the generative AI.

[0744] 8. The generative AI generates dialogue content based on that information, and the robot uses gestures that match the generated dialogue content.

[0745] Specific examples

[0746] The user requests the system to record a will.

[0747] The device will prompt you to "Enter your will."

[0748] The user types, "Please divide all my assets among my children."

[0749] The server stores this information in a database.

[0750] The user requests "play the will."

[0751] The terminal sends a playback request to the server.

[0752] The server searches for the stored will and sends it to the device.

[0753] The device will play back the contents of the will as audio or text.

[0754] Prompt Sentence Examples

[0755] "I would like to record my will. Please enter the details."

[0756] "Please tell me how to sort out my belongings."

[0757] "Generate dialogue for when the user is happy."

[0758] This system can satisfy the diverse needs of elderly people regarding end-of-life planning in one system and provide a richer user experience through dialogue that responds to their emotions. The flow of the specific processing in Example 3 will be described with reference to FIG.

[0759] Program processing flow

[0760] Recording and reproducing wills

[0761] Step 1:

[0762] When a user wishes to record a will, he or she sends a request to the system through a terminal. The input is a request saying "I want to record a will." The terminal receives this request and displays a prompt saying "Please enter the contents of your will."

[0763] Step 2:

[0764] The user inputs the contents of the will. The input is text data such as "Please divide all my assets among my children." The device receives this input and sends it to the server.

[0765] Step 3:

[0766] The server saves the received will in a database. The input is the text data of the will, and the output is the record of the will saved in the database. The server notifies the terminal that saving is complete.

[0767] Step 4:

[0768] When a user wishes to replay a will, he or she sends a request to the system through the terminal. The input is a request to "replay the will." The terminal receives this request and sends it to the server.

[0769] Step 5:

[0770] The server retrieves the saved will from the database. The input is the playback request, and the output is the retrieved text data of the will. The server sends this data to the terminal.

[0771] Step 6:

[0772] The terminal plays back the received will as audio or text. The input is the text data of the will, and the output is the playback of audio or text.

[0773] Providing information on sorting belongings

[0774] Step 1:

[0775] When a user requests information about sorting out belongings, they send a request to the system through their terminal. The input is a request such as "Tell me how to sort out belongings." The terminal receives this request and sends it to the server.

[0776] Step 2:

[0777] The server retrieves information about estate clearance from a database. The input is an information request, and the output is the retrieved estate clearance information. The server sends this information to the terminal.

[0778] Step 3:

[0779] The terminal displays the received estate clearance information to the user. The input is estate clearance information data, and the output is the display of the information.

[0780] Collaboration between robots and generative AI

[0781] Step 1:

[0782] The user expresses their emotions. The input is "I'm very happy today." The robot analyzes this statement with its emotion recognition engine and recognizes the user's emotions.

[0783] Step 2:

[0784] The robot communicates the recognized emotional information to the generative AI. The input is the recognized emotional data, and the output is the transmission of emotional information to the generative AI.

[0785] Step 3:

[0786] The generative AI generates dialogue content based on emotional information. The input is emotional data, and the output is the generated dialogue content. The generative AI sends this dialogue content to the robot.

[0787] Step 4:

[0788] The robot makes gestures that match the generated dialogue content. The input is the generated dialogue content, and the output is the gestures.

[0789] Step 5:

[0790] The robot communicates the generated dialogue content to the user. The input is the generated dialogue content, and the output is the execution of the dialogue.

[0791] (Application example 3)

[0792] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0793] A lack of emotional support and concrete information is a challenge for elderly people when they are preparing for their end of life. In particular, there is a need for a system that provides a wide range of information related to end-of-life planning, such as recording wills, sorting belongings, funeral preparations, and consultations on inheritance and taxes. There is also a growing need for a system that can engage in appropriate dialogue while being sensitive to the emotions of elderly people.

[0794] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0795] In this invention, the server includes a robot means for interacting with the elderly, an emotion recognition means, a means for generating gestures and movements, a generative AI means for generating natural conversations, a means for empathizing with what the elderly is saying and asking questions, a means for recording memories and wills, a means for providing information on end-of-life planning such as sorting out belongings, funeral preparations, and consultations on inheritance and taxes, a means for generating dialogue content based on the elderly's emotions, a means for recording the contents of the will and playing it back as needed, a means for providing methods and points to note for sorting out belongings, a means for providing information on funeral preparations, and a means for providing consultations on inheritance and taxes. This makes it possible for elderly people to receive emotional support while centrally obtaining a wide range of information on end-of-life planning.

[0796] An "elderly conversation robot" is an autonomous mechanical device designed to have natural conversations with the elderly.

[0797] "Emotion recognition function" is a technology for analyzing and recognizing emotions from a user's facial expressions and voice.

[0798] The "function of generating gestures" is a function that enables the robot to perform natural movements while interacting with the user.

[0799] "Generative AI that generates natural conversations" is an artificial intelligence technology that generates natural dialogue based on user input.

[0800] The "function to empathize with and ask questions about what the elderly person is saying" is a function to show empathy for what the user is saying and to ask appropriate questions.

[0801] "Recording memories and wills" is a function that saves the contents of a user's memories and wills in digital format.

[0802] The "function to provide information on end-of-life planning, such as sorting out belongings, preparing for a funeral, and consulting on inheritance and taxes" is a function that allows users to obtain various information on end-of-life planning.

[0803] The "function of generating dialogue content based on the emotions of the elderly" is a function of generating appropriate dialogue content according to the emotional state of the user.

[0804] The "function to record the contents of a will and play it back as needed" is a function to record the user's will and play back the contents when needed.

[0805] The "function of providing methods and points to note for sorting out belongings of the deceased" is a function that provides the user with specific methods and points to note for sorting out belongings of the deceased.

[0806] The "function of providing information on funeral preparations" is a function that provides the user with information on funeral preparations.

[0807] The "function for providing advice on inheritance and taxes" is a function that allows users to receive advice on inheritance and taxes.

[0808] As an embodiment of the present invention, a system using a robot to support end-of-life planning will be described. This system includes a robot that can converse with elderly people, an emotion recognition function, a function to generate gestures, a generative AI that generates natural conversations, a function to empathize with what the elderly are saying and ask questions, and a function to record memories and wills, and provide information on end-of-life planning such as sorting out belongings, preparing for a funeral, and consulting on inheritance and taxes.

[0809] Hardware used

[0810] Robot Body: An autonomous mechanical device for interacting with the elderly

[0811] Camera and microphone: Devices for capturing the user's facial expressions and voice.

[0812] Software used

[0813] Emotion Recognition Library: A library for analyzing user emotions

[0814] AI Conversation Model: An artificial intelligence model for generating natural conversations (e.g., GPT-3, ChatGPT (registered trademark))

[0815] End of Life Services Module: A proprietary module for providing information on end-of-life planning

[0816] Data processing and calculation

[0817] 1. Emotion Recognition: The camera and microphone are used to capture the user's facial expressions and voice, and the Emotion Recognition Library is used to analyze their emotions, allowing the system to grasp the user's emotional state in real time.

[0818] 2. Dialogue generation: Based on the analyzed emotional data, the AI ​​Conversation Model generates appropriate dialogue content, enabling natural conversations that are sensitive to the user's emotions.

[0819] 3. Service provision: In response to user requests, the End of Life Services Module provides information on recording wills, sorting belongings, funeral preparations, and inheritance and tax consultations.

[0820] Specific examples

[0821] For example, if an elderly person says, "I want to record my will," the robot will recognize that emotion and use generative AI to respond, "Please tell me the contents of your will." The user can then speak the contents of their will, and the robot will record it and play it back as needed.

[0822] Prompt Sentence Examples

[0823] User: I want to record my will.

[0824] Robot: Please tell me what the will says.

[0825] User: I want my son to inherit my assets.

[0826] Robot: Understood. I have recorded the will. Is there anything else I can help you with?

[0827] In this way, end-of-life support robots can provide comprehensive support for elderly people's end-of-life preparations and provide emotionally sensitive dialogue.

[0828] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0829] Step 1:

[0830] emotion recognition

[0831] The server captures the user's facial expressions and voice using a camera and microphone. As input, it obtains the user's real-time video and audio data. It uses the Emotion Recognition Library to analyze these data and recognize the user's emotional state. As output, it obtains the user's emotional state (e.g., joy, sadness, anger, etc.).

[0832] Step 2:

[0833] Dialogue Generation

[0834] The server receives the user's emotional state obtained in step 1 as input. It uses an AI Conversation Model (e.g., GPT-3, ChatGPT) to generate appropriate dialogue content based on the user's emotions. The server uses the user's emotional state and the user's utterances as input. The server obtains the generated dialogue content as output.

[0835] Step 3:

[0836] Executing the dialogue

[0837] The terminal (robot) outputs the dialogue content generated in step 2 to the user as voice. Specifically, the dialogue content is played back as voice through the robot's speaker. This allows for natural dialogue with the user.

[0838] Step 4:

[0839] Service Request

[0840] The user requests a specific service from the robot (e.g., recording a will, advice on sorting out belongings, etc.). The robot receives the user's voice request as input. The robot then sends this request to the server.

[0841] Step 5:

[0842] Service provision

[0843] The server receives the user's request and provides the appropriate information using the End of Life Services Module. The server uses the user's request as input. For example, if a request is made to record a will, the server records the will's contents and executes the function of replaying it as needed. The server obtains the information and services to be provided to the user as output.

[0844] Step 6:

[0845] Output of information

[0846] The terminal (robot) outputs the information and services obtained in step 5 to the user by voice or on a display. Specifically, the information is provided by voice through the robot's speaker or displayed on a display, allowing the user to obtain the information they need.

[0847] Step 7:

[0848] Get feedback

[0849] The server receives feedback from the user. As input, it recaptures the user's voice and facial expressions and analyzes them using emotion recognition. As output, it obtains the user's satisfaction level and any additional requests. This allows the system to continuously provide services that meet the user's needs.

[0850] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0851] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0852] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0853] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0854] [Second embodiment]

[0855] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0856] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0857] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0858] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0859] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0860] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0861] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0862] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0863] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0864] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0865] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0866] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0867] "Example 1"

[0868] As one embodiment of the present invention, a robot that can interact with elderly people is provided. This robot has an emotion recognition function and can recognize the emotional state of the elderly person. It also has a function of generating gestures, which allows it to create a friendly relationship with the elderly person. Specifically, when the elderly person expresses joy, the robot generates gestures that match the joy.

[0869] "Example 2"

[0870] In addition, a generative AI is provided. This generative AI has the ability to generate natural conversations, empathize with what the elderly person is saying, and ask questions. Specifically, when an elderly person talks about their past memories, the generative AI generates an empathetic response based on that content. Also, when an elderly person expresses a question about end-of-life planning, the generative AI generates an answer to that question.

[0871] "Example 3"

[0872] Furthermore, the system has the function of providing information on end-of-life planning, such as recording memories and wills, sorting out belongings, preparing for funerals, and consulting on inheritance and taxes. Specifically, if an elderly person requests to record a will, the system will record the contents of the will and play it back as needed. Also, if an elderly person requests information on sorting out belongings, the system will provide information on how to sort out belongings and important points to note.

[0873] The processing flow of each embodiment will be described below.

[0874] "Example 1"

[0875] Step 1: The robot that interacts with the elderly is activated.

[0876] Step 2: The robot recognizes the emotional state of the elderly person using its emotion recognition function. For example, if the elderly person smiles, the robot recognizes the smile and determines that the elderly person is happy.

[0877] Step 3: The robot uses its gesture generation function to create a friendly atmosphere with the elderly. For example, if the elderly person expresses joy, the robot will generate a gesture that matches the joy.

[0878] "Example 2"

[0879] Step 1: The generative AI is launched.

[0880] Step 2: The generative AI listens to what the elderly person says and generates natural conversations based on that content. For example, if the elderly person talks about their past memories, the generative AI generates empathetic responses based on that content.

[0881] Step 3: When an elderly person expresses a question about end-of-life planning, the generative AI generates an answer to that question. For example, if an elderly person asks for information about sorting out their belongings, the generative AI will provide information on how to sort out their belongings and what to be careful about.

[0882] "Example 3"

[0883] Step 1: When an elderly person requests information about end-of-life planning, the system will provide the corresponding information. For example, if an elderly person requests to record a will, the system will record the contents of the will and play it back as needed.

[0884] Step 2: If an elderly person requests information about sorting out their belongings, the system will provide them with information on how to sort out their belongings and what to be careful about.

[0885] Step 3: If the elderly person requests information about other end-of-life matters, such as funeral arrangements, inheritance, or tax advice, the system will provide that information.

[0886] Example 1

[0887] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0888] In order for elderly people to live their daily lives without feeling lonely, it is important to provide appropriate information and dialogue that is sensitive to their emotions. However, with conventional robot systems, it was difficult to accurately recognize the emotional state of the elderly and generate natural dialogue and gestures accordingly. In addition, the provision of information and consultation functions regarding end-of-life planning was insufficient. As a result, it was not possible to alleviate the loneliness and anxiety felt by elderly people.

[0889] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0890] In this invention, the server includes a robot means for interacting with the elderly, an emotion recognition means, a means for generating gestures and movements, a generative AI means for generating natural conversations, a means for empathizing with what the elderly is saying and asking questions, a means for recording memories and wills, a means for providing information on end-of-life planning such as sorting out belongings, preparing for a funeral, and consultations on inheritance and taxes, and a means for recognizing the emotional state of the elderly, generating appropriate gestures and movements based on the recognized emotions, and executing the generated gestures and movements. This enables conversations and gestures that are sensitive to the emotions of the elderly, and also enhances the information and consultation functions on end-of-life planning.

[0891] A "robot that interacts with the elderly" is an automated device that can have natural conversations with the elderly.

[0892] "Emotion recognition" is a technology that analyzes facial expressions and voice data of elderly people to identify their emotional state (joy, sadness, anger, etc.).

[0893] "Gesture generation" is a technology that allows a robot to generate appropriate actions (such as waving or smiling) based on recognized emotions.

[0894] "Generative AI" is an artificial intelligence technology that generates natural conversations with the elderly.

[0895] "Empathizing and asking questions" is a technique for deepening dialogue by showing empathy for what the elderly person is saying and asking appropriate questions.

[0896] "Memories and Wills Recording" is a technology that stores the memories and wills of elderly people in digital format.

[0897] "Providing information on end-of-life planning" refers to technology that provides elderly people with information related to end-of-life planning, such as sorting out personal belongings, funeral preparations, inheritance, and tax consultations.

[0898] "Recognizing the emotional state, generating appropriate gestures based on the recognized emotions, and executing the generated gestures" refers to a series of processes in which the robot analyzes the emotions of the elderly person and performs actions in accordance with those emotions.

[0899] MODE FOR CARRYING OUT THE INVENTION

[0900] This invention relates to a robot system that interacts with the elderly, and includes functions such as emotion recognition, gesture generation, generative AI that generates natural conversations, the ability to empathize with and ask questions about what the elderly are saying, the ability to record memories and wills, and the ability to provide information about end-of-life planning.

[0901] 1. Program Generation

[0902] The server generates a program for the robot to interact with the elderly. This program includes an emotion recognition function and a gesture generation function. The emotion recognition function is implemented using, for example, TensorFlow. The gesture generation function is implemented using known technology.

[0903] 2. Program processing explanation

[0904] The server analyzes data acquired from the camera and microphone to realize the emotion recognition function. Specifically, it analyzes facial expressions in facial images acquired from the camera and analyzes audio data acquired from the microphone using TensorFlow. This makes it possible to recognize the emotional state of the elderly.

[0905] The terminal (robot) generates appropriate gestures based on the emotion information received from the server. The robot has movement patterns corresponding to the emotion, and selects and executes one of them.

[0906] 3. Examples and prompts

[0907] As a concrete example, consider the case where an elderly person says, "I'm very happy today." In this case, the server analyzes the voice input and recognizes that the elderly person is feeling happy. Next, the device (robot) generates gestures that match the emotion of joy (for example, waving, smiling, etc.), creating a friendly atmosphere toward the elderly person.

[0908] Example prompts to input to a generative AI model:

[0909] "What gestures should a robot make when an elderly person expresses joy?"

[0910] Using this prompt, the generative AI model can suggest appropriate gestures based on the elderly person's emotions.

[0911] In this way, the present invention can realize conversations and gestures that are sensitive to the feelings of the elderly, and can also enhance the provision of information and consultation functions regarding end-of-life planning.

[0912] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0913] Step 1:

[0914] The server acquires facial images and voice data of elderly people from the camera and microphone. The input is facial image data from the camera and voice data from the microphone, and the output is a state ready for analyzing these data. Specifically, the server initializes the camera and microphone devices and acquires data in real time.

[0915] Step 2:

[0916] The server performs facial expression analysis on the acquired facial image data. The input is facial image data acquired from the camera, and the output is the emotional state (happiness, sadness, anger, etc.) based on the elderly person's facial expression. Specifically, the server extracts feature points from the facial image and inputs them into an emotion recognition model to estimate the emotional state.

[0917] Step 3:

[0918] The server analyzes the acquired voice data using TensorFlow and estimates emotions from the tone of voice and the content of the words. The input is voice data acquired from a microphone, and the output is the emotional state based on the voice. Specifically, the server preprocesses the voice data and inputs it into a voice recognition model to estimate the emotional state.

[0919] Step 4:

[0920] The server integrates the results of facial expression analysis and voice analysis to recognize the overall emotional state of the elderly person. The input is the results of facial expression analysis and voice analysis, and the output is the overall emotional state of the elderly person. Specifically, the server weights and integrates the results of both analyses to determine the final emotional state.

[0921] Step 5:

[0922] The terminal (robot) generates appropriate gestures based on the emotional information received from the server. The input is the emotional information from the server, and the output is the generated gesture movement sequence. As a specific movement, the terminal selects a movement pattern according to the emotion and generates a motor control signal to execute this.

[0923] Step 6:

[0924] The terminal (robot) executes the generated gestures. The input is a motion sequence, and the output is the actual robot motion. Specific motions are controlled by the terminal, which controls motors and actuators to perform actions such as waving or smiling.

[0925] Step 7:

[0926] The user (elderly person) watches the robot's movements and continues the dialogue. The input is the robot's movements and dialogue content, and the output is the elderly person's new utterances and reactions. In terms of specific actions, the elderly person expresses their emotions in response to the robot's movements and continues the next dialogue.

[0927] (Application example 1)

[0928] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0929] When elderly people visit a physical store, it can be difficult for them to receive guidance within the store or product explanations. In particular, in situations where a response that corresponds to their emotional state is required, conventional guidance systems often do not respond adequately. Furthermore, in order for elderly people to use stores with peace of mind, it is important to respond in a friendly and empathetic manner. To solve these issues, a system is needed that can recognize the emotional state of the elderly and provide appropriate gestures and dialogue content.

[0930] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0931] In this invention, the server includes a robot means for interacting with the elderly, an emotion recognition function means, a function means for generating gestures, a generative AI means for generating natural conversations, a function means for empathizing with what the elderly is saying and asking questions, a function means for recording memories and wills, a function means for sorting belongings and preparing for a funeral, a function means for providing information on end-of-life planning such as consultations on inheritance and taxes, and a function means that is installed in a physical store and provides store guidance and product explanations while using gestures according to the emotional state of the elderly. This enables elderly people to receive friendly service and appropriate guidance according to their emotional state when they visit a physical store.

[0932] A "robot that interacts with the elderly" is an automated mechanical device that can have natural conversations with the elderly.

[0933] "Emotion recognition function" is a technology that identifies the emotional state of an elderly person from their facial expressions, tone of voice, etc.

[0934] The "function to generate gestures and movements" is a technology that generates appropriate gestures and movements according to the recognized emotional state.

[0935] "Generative AI that generates natural conversations" is an artificial intelligence technology that generates natural conversations when interacting with the elderly.

[0936] The "function of empathizing with and asking questions about what the elderly are saying" refers to the technology of showing empathy for what the elderly are saying and asking appropriate questions.

[0937] "Recording memories and wills" is a function that records the memories and wills of elderly people.

[0938] "Sorting out belongings and preparing for funerals" is a function that assists elderly people in sorting out their belongings and preparing for funerals.

[0939] The "function to provide information on end-of-life planning, such as consultations on inheritance and taxes" is a function that provides information for elderly people to consult on inheritance and taxes related to end-of-life planning.

[0940] "A function that is installed in a physical store and provides store guidance and product explanations while using gestures that correspond to the emotional state of the elderly" refers to technology that is installed in a physical store and provides store guidance and product explanations while using gestures that correspond to the emotional state of the elderly.

[0941] The system for implementing this invention is centered around a robot that interacts with elderly people. The system includes the following main hardware and software:

[0942] 1. Hardware:

[0943] Camera: Used to capture the facial expressions of the elderly.

[0944] Robot body: A mechanical device for interacting and gesturing.

[0945] Microphone: Used to capture the voice of the elderly person.

[0946] Speaker: Used to output the generated audio.

[0947] 2. Software:

[0948] A library for capturing and processing camera footage.

[0949] A deep learning framework to load emotion recognition models and make predictions.

[0950] A library used as a speech synthesis engine to conduct voice-based dialogue with the elderly.

[0951] The system works as follows:

[0952] 1. Emotion recognition:

[0953] The camera captures the elderly person's facial expressions and converts them into grayscale.

[0954] The image is resized to 48x48 pixels and fed into the emotion recognition model.

[0955] Obtain emotion labels from the model's prediction results.

[0956] 2. Gesture generation:

[0957] Depending on the recognized emotional state, appropriate gestures are generated.

[0958] For example, if the emotion of joy is recognized, the robot will make gestures that express joy.

[0959] 3. Voice interaction:

[0960] Generative AI is used to generate dialogue content that corresponds to the emotional state of the elderly person.

[0961] The generated dialogue content is output as voice.

[0962] To illustrate, consider the following scenario:

[0963] When an elderly person enters the store, the robot will say aloud, "Hello! I'll make a gesture of joy. Is there anything I can help you with?"

[0964] If an elderly person looks at a product with joy, the robot will explain, using happy gestures, "That product is very popular!"

[0965] Examples of prompts to input to a generative AI model include:

[0966] "Generate gestures that an elderly person would make if they were happy."

[0967] "Generate gestures for when an elderly person is surprised."

[0968] This means that when elderly people visit a physical store, they will be able to receive friendly service and appropriate guidance that is tailored to their emotional state.

[0969] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0970] Step 1:

[0971] The camera captures the facial expressions of the elderly.

[0972] Input: Camera image

[0973] Output: Captured image frames

[0974] Specific operation: The camera captures the elderly person's face in real time and acquires video data.

[0975] Step 2:

[0976] The server converts the captured image frames to grayscale.

[0977] Input: Captured image frames

[0978] Output: Grayscale image

[0979] Specific operation: Converts a color image to a grayscale image.

[0980] Step 3:

[0981] The server resizes the grayscale image to 48x48 pixels.

[0982] Input: Grayscale image

[0983] Output: Resized image

[0984] Specific behavior: Resizes a grayscale image to 48x48 pixels.

[0985] Step 4:

[0986] The server inputs the resized image into the emotion recognition model.

[0987] Input: Resized image

[0988] Output: Emotion label

[0989] Specific operation: The resized image is input into the emotion recognition model to predict the emotion label.

[0990] Step 5:

[0991] The server generates appropriate gestures based on the emotion labels.

[0992] Input: Emotion label

[0993] Output: Gesture instructions

[0994] Specific actions: Generate gestures corresponding to the emotion labels and send instructions to the robot.

[0995] Step 6:

[0996] The robot executes the generated gesture.

[0997] Input: Gesture instructions

[0998] Output: Gesture movement

[0999] Specific behavior: The robot follows instructions and performs appropriate gestures.

[1000] Step 7:

[1001] The server generates dialogue content according to the emotional state of the elderly person.

[1002] Input: Emotion label

[1003] Output: Dialogue content

[1004] Specific operation: Using generative AI, natural dialogue content is generated according to the emotional state of the elderly person.

[1005] Step 8:

[1006] The server inputs the generated dialogue content into a speech synthesis engine.

[1007] Input: Dialogue

[1008] Output: Audio data

[1009] Specific operation: Convert the generated dialogue content into voice data.

[1010] Step 9:

[1011] The robot plays the audio data.

[1012] Input: Audio data

[1013] Output: Audio output

[1014] Specific operation: The generated voice data is played back through the robot's speaker, and the conversation content is conveyed to the elderly.

[1015] Example 2

[1016] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1017] When communicating with the elderly, it is necessary to effectively recognize emotions, respond with empathy, and provide information about end-of-life planning. However, conventional systems have difficulty providing these functions in an integrated manner, and there is a lack of dialogue systems that the elderly can use with confidence. Furthermore, technology for generating appropriate responses to the elderly's input has not been sufficiently developed. This makes it difficult to alleviate the loneliness and anxiety felt by the elderly.

[1018] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a device for interacting with the elderly, emotion recognition means, gesture generation means, generative AI means for generating natural conversation, means for empathizing with what the elderly is saying and asking questions, means for recording memories and wills, means for providing information on end-of-life planning such as sorting belongings, funeral preparations, and inheritance and tax consultations, means for receiving text data input by a user, generating and inputting prompt sentences to the generative AI model, and means for displaying the response generated by the generative AI model to the user. This makes it possible to realize natural conversation with the elderly and to provide emotion recognition, empathetic responses, and information on end-of-life planning in an integrated manner.

[1019] A "device for interacting with the elderly" is a device that includes hardware and software that allows the elderly to interact with the elderly through voice and text.

[1020] "Emotion recognition means" is a technology for analyzing and recognizing the emotional state of elderly people from their voices, facial expressions, and text.

[1021] "Means for generating gestures" refers to technology that enables robots and avatars to generate natural gestures when interacting with elderly people.

[1022] "Generative AI means for generating natural conversations" refers to artificial intelligence technology for generating natural and fluent conversations in dialogue with the elderly.

[1023] "Means for empathizing with and asking questions about what the elderly say" refers to technology for generating empathetic responses and appropriate questions in response to what the elderly say.

[1024] "Mechanisms for recording memories and wills" is a technology for digitally recording the memories and wills spoken by elderly people.

[1025] "Means of providing information on end-of-life planning, such as sorting out belongings, preparing for funerals, and consulting on inheritance and taxes" refers to technology that allows elderly people to obtain various information on end-of-life planning.

[1026] "Means for receiving text data entered by a user and generating and inputting a prompt sentence to a generative AI model" refers to a technology for receiving text data entered by a user and generating a prompt sentence to appropriately input the text data to a generative AI model.

[1027] "Means for displaying to the user the response generated by the generative AI model" refers to technology for visually or audibly displaying to the user the response generated by the generative AI model.

[1028] MODE FOR CARRYING OUT THE INVENTION

[1029] This invention is a system for realizing natural dialogue with elderly people, and integrates emotion recognition, generation of empathetic responses, provision of information on end-of-life planning, etc. Specific embodiments of this system are described below.

[1030] System configuration

[1031] The system consists of the following main components:

[1032] 1. Devices for interacting with the elderly: These are devices that include hardware and software that allow elderly people to interact with the elderly through voice and text. Examples include smartphones, tablets, and dedicated conversation robots.

[1033] 2. Emotion recognition: This technology analyzes and recognizes the emotional state of elderly people from their voices, facial expressions, and text. For example, it uses voice analysis software and facial expression recognition cameras.

[1034] 3. Means of generating gestures: This technology allows robots and avatars to generate natural gestures when interacting with the elderly. For example, motion capture technology is used.

[1035] 4. Generative AI means for generating natural conversations: This is an artificial intelligence technology for generating natural and fluent conversations with the elderly. For example, OpenAI's GPT-4 is used as a generative AI model.

[1036] 5. Means of empathizing with and asking questions about what the elderly say: Techniques for generating empathetic responses and appropriate questions in response to what the elderly say.

[1037] 6. Recording of memories and wills: Technologies for digitally recording the memories and wills spoken by older people, for example, using cloud storage services.

[1038] 7. Means of providing information on end-of-life planning: This is technology for providing information on end-of-life planning, such as sorting out belongings, preparing for a funeral, and consulting on inheritance and taxes.

[1039] 8. A means for receiving text data entered by a user and generating and inputting prompt sentences to a generative AI model: A technology for receiving text data entered by a user and generating prompt sentences to appropriately input it into a generative AI model.

[1040] 9. Means for displaying responses generated by a generative AI model to the user: A technology for visually or audibly displaying responses generated by a generative AI model to the user.

[1041] System Operation

[1042] The user (elderly person) uses the dialogue device to input text data. For example, the user might input, "I miss the memories of traveling when I was young." The device then sends this text data to the server. The server analyzes the received text data and generates a prompt sentence to input into the generative AI model. For example, it generates a prompt sentence such as, "The user is talking about memories of traveling when they were young. What kind of response would you generate?"

[1043] A generative AI model (e.g., GPT-4) generates an appropriate response based on this prompt, such as "Which place is most memorable?" The server sends the generated response to the device, which then displays it to the user.

[1044] Specific examples

[1045] As a concrete example, consider an elderly person talking about past memories. If the user inputs, "I have fond memories of the trips I took when I was younger," the generative AI will generate an empathetic response such as, "Which place left the most lasting impression on you?" Similarly, if an elderly person expresses a question about end-of-life planning, for example, by inputting, "What should I do to start preparing for the end of my life?", the generative AI will generate an answer such as, "It would be a good idea to start by writing a will and organizing your assets."

[1046] Example prompt sentence:

[1047] "I have fond memories of traveling when I was young."

[1048] "How do I start planning my end?"

[1049] In this way, the system can realize natural conversations with the elderly, and can comprehensively recognize emotions, respond with empathy, and provide information on end-of-life planning.

[1050] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1051] Step 1:

[1052] The user inputs text data using the terminal. For example, the user inputs "I miss the memories of traveling when I was young." The input text data is stored in the memory of the terminal.

[1053] Step 2:

[1054] The terminal sends the input text data to the server. Specifically, the terminal converts the text data into JSON format and sends it to the server using an HTTP POST request. The input is the user's text data, and the output is the data to be sent to the server.

[1055] Step 3:

[1056] The server receives text data sent from the terminal. The received data is stored in the server's memory. The input is the text data from the terminal, and the output is the data stored in the server's memory.

[1057] Step 4:

[1058] The server analyzes the received text data and generates a prompt to be input to the generative AI model. For example, if the received data is "I miss the memories of traveling when I was young," the prompt would be "The user is talking about their memories of traveling when they were young. What response would you like to generate?" The input is the text data stored on the server, and the output is the generated prompt.

[1059] Step 5:

[1060] The server inputs a prompt to the generative AI model. The generative AI model (e.g., GPT-4) generates an appropriate response based on the prompt. For example, it generates the response, "Which place is most memorable?" The input is the generated prompt, and the output is the response generated by the generative AI model.

[1061] Step 6:

[1062] The server sends the generated response to the terminal. Specifically, it converts the generated response into JSON format and sends it to the terminal as an HTTP response. The input is the response generated by the generative AI model, and the output is the data sent to the terminal.

[1063] Step 7:

[1064] The terminal receives the response data sent from the server and displays it to the user. For example, the terminal displays the message "Which place is the most memorable?" on the screen. The input is the response data from the server, and the output is the message displayed to the user.

[1065] In this way, the system can realize natural conversations with the elderly, and can comprehensively recognize emotions, respond with empathy, and provide information on end-of-life planning.

[1066] (Application example 2)

[1067] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1068] When elderly people spend time alone at home, it is necessary to provide an environment where they can live with peace of mind, reducing feelings of loneliness and anxiety. Also, a system that can respond quickly in emergencies is necessary. Furthermore, it is important to make it easy for elderly people to obtain information about end-of-life planning.

[1069] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a device means for interacting with the elderly, an emotion recognition means, a means for generating gestures and movements, a generative AI means for generating natural conversations, a means for empathizing with the elderly's story and asking questions, a means for recording memories and wills, a means for sorting belongings and preparing for a funeral, a means for providing information on end-of-life planning such as inheritance and tax consultations, and a means for automatically sending notifications in the event of an emergency. This provides an environment where the elderly can live in peace and enables rapid response in the event of an emergency. It also makes it easy to obtain information on end-of-life planning.

[1070] A "device for interacting with the elderly" is a device that includes hardware and software for conducting natural conversations with the elderly.

[1071] "Emotion recognition means" is a technology for analyzing and recognizing the emotions of elderly people from their facial expressions and tone of voice.

[1072] The "means for generating gestures and movements" is a technology that generates natural gestures and movements during conversations to facilitate communication with the elderly.

[1073] "Generative AI means for generating natural conversations" is an artificial intelligence technology for generating natural responses in dialogue with elderly people.

[1074] "Means for empathizing with and asking questions about what the elderly are saying" is a technology for generating empathetic responses and appropriate questions to what the elderly are saying.

[1075] "Means for recording memories and wills" is a technology for recording the memories and wills of elderly people in digital format.

[1076] "Tools for sorting out belongings and preparing for funerals" refers to technology that provides information and support for elderly people regarding sorting out belongings and preparing for funerals.

[1077] "Means of providing information on end-of-life planning, such as consultations on inheritance and taxes" refers to technology that provides information to help elderly people resolve questions about inheritance and taxes.

[1078] "Means for automatically sending notifications in emergencies" refers to technology that automatically sends notifications to family members and emergency contacts when an elderly person experiences an emergency.

[1079] This invention is a system for providing an environment where elderly people can live safely at home. The system includes a device for interacting with elderly people, emotion recognition means, gesture generation means, generative AI means for generating natural conversations, means for empathizing with elderly people's stories and asking questions, means for recording memories and wills, means for sorting belongings and preparing funerals, means for providing information on end-of-life planning such as consultations on inheritance and taxes, and means for automatically sending notifications in the event of an emergency.

[1080] System Program

[1081] The system program is configured as follows:

[1082] 1. Devices for interacting with the elderly:

[1083] It receives text and voice input from the elderly person and sends it to a generative AI.

[1084] The response from the generative AI is returned to the elderly person.

[1085] 2. Emotion recognition means:

[1086] It analyzes the facial expressions and tone of voice of elderly people to recognize their emotional state.

[1087] The recognition results are fed back to the generative AI to adjust the content of the dialogue.

[1088] 3. Means of generating gestures:

[1089] It generates natural gestures and hand movements during conversations with elderly people, facilitating communication.

[1090] 4. Generative AI methods for generating natural conversations:

[1091] Generate natural responses based on the elderly person's input.

[1092] Empathize with what the elderly are saying and generate appropriate questions.

[1093] 5. Means of recording memories and wills:

[1094] Record the memories and wills of the elderly in digital form.

[1095] 6. Arranging belongings and funeral arrangements:

[1096] Provides information and support for elderly people sorting out their belongings and preparing for funerals.

[1097] 7. Means of providing information on end-of-life planning, such as inheritance and tax consultations:

[1098] Provides information to help seniors resolve questions about inheritance and taxes.

[1099] 8. Means of sending automatic notifications in case of emergency:

[1100] When an elderly person has an emergency, notifications are automatically sent to family members and emergency contacts.

[1101] Hardware and software used

[1102] Hardware: Devices such as smartphones, tablets, and computers.

[1103] Software: Python, OpenAI API.

[1104] Processing flow

[1105] 1. The elderly person talks to the system using a smartphone or tablet.

[1106] 2. The system receives the elderly person's input and analyzes their emotional state using emotion recognition means.

[1107] 3. The analysis results are fed back to the generative AI to generate natural conversations.

[1108] 4. Return the generated response to the elderly person and generate gestures as needed.

[1109] 5. If an elderly person has questions about end-of-life planning, provide them with relevant information.

[1110] 6. Automatically notify family members and emergency contacts in case of an emergency.

[1111] Specific examples

[1112] Prompt Sentence Examples

[1113] Elderly person: I went on a trip with my family a long time ago, and I have fond memories of that time.

[1114] Example of the response generated

[1115] AI: That's a wonderful memory! Where did you travel to? What was the most memorable experience?

[1116] In this way, generative AI empathizes with what the elderly person is saying and asks questions to expand on the conversation, allowing the elderly person to continue the conversation with a sense of security.

[1117] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1118] Step 1:

[1119] A user speaks to the system using a smartphone or tablet. The input can be in the form of voice or text, and the device receives it. In the case of voice input, the device uses speech recognition software to convert it to text. The output is the user input in the form of text.

[1120] Step 2:

[1121] The terminal transmits the user's input text to the emotion recognition means. The emotion recognition means analyzes the user's input text and recognizes the user's emotional state. Specifically, it uses natural language processing techniques to classify the emotion of the text. The input is the user's text input, and the output is the recognized emotional state.

[1122] Step 3:

[1123] The device sends the recognized emotional state and the user's input text to the generative AI means, which generates natural conversations based on this information. Specifically, it uses a generative AI model (e.g., OpenAI GPT-3) to generate empathetic responses and appropriate questions. The input is the user's text input and emotional state, and the output is the generated response text.

[1124] Step 4:

[1125] The device returns the generated response text to the user. If speech output is required, the device uses text-to-speech software to convert text to speech. The input is the generated response text, and the output is the response returned to the user.

[1126] Step 5:

[1127] When a user asks a question about end-of-life planning, the device uses a generative AI means to provide relevant information. The generative AI means generates appropriate end-of-life planning information based on the user's question. The input is the user's question text, and the output is the relevant end-of-life planning information.

[1128] Step 6:

[1129] In an emergency, if the user enters keywords such as "emergency" or "help," the device will automatically activate a means to send an emergency notification. Specifically, it uses SMTP to send a notification to family members and emergency contacts. The input is the user's emergency keyword, and the output is the sent emergency notification.

[1130] Example 3

[1131] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1132] There is a need for systems that allow elderly people to efficiently and appropriately obtain information about end-of-life planning, record memories and wills, organize belongings, prepare for funerals, and smoothly navigate complex procedures such as inheritance and tax consultations. However, existing systems have difficulty providing these functions in an integrated manner, forcing elderly people to use multiple systems to obtain the information they need. Furthermore, usability has been reduced due to the lack of dialogue that responds to the elderly's emotional state and the lack of natural conversation generation using speech recognition and generative AI models.

[1133] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1134] In this invention, the server includes a device means for interacting with the elderly person, an emotion recognition function means, a function means for generating gestures and movements, a generative AI means for generating natural conversations, a function means for empathizing with what the elderly person is saying and asking questions, a function means for recording memories and wills, a function means for providing information on end-of-life planning such as sorting out belongings, preparing for a funeral, and consulting about inheritance and taxes, a means for converting speech to text using speech recognition software, a means for storing the text data in a database, a means for generating answers to user questions using a generative AI model, and a means for converting text data to speech using speech synthesis software. This allows the elderly person to efficiently obtain all information related to end-of-life planning through a single system and, through natural conversations tailored to their emotions, smoothly carry out complex procedures such as recording memories and wills, sorting out belongings, preparing for a funeral, and consulting about inheritance and taxes.

[1135] The "device means for interacting with the elderly" is a device that provides an interface for the elderly to interact with the system through voice or text.

[1136] The "emotion recognition function means" is a function that analyzes and recognizes the emotional state of an elderly person from their voice, facial expressions, gestures, etc.

[1137] The "functional means for generating gestures" is a function that allows a virtual character or robot to generate gestures so that the system can achieve natural communication in dialogue with the elderly.

[1138] "Generative artificial intelligence means for generating natural conversation" refers to artificial intelligence technology for generating natural and fluent conversation in dialogue with elderly people.

[1139] "Functional means of empathizing with and asking questions about what the elderly are saying" refers to the function of deepening dialogue by showing empathy for what the elderly are saying and asking appropriate questions.

[1140] The "means for recording memories and wills" is a function that allows elderly people to record and save their memories and wills in audio or text format.

[1141] "Functional means for providing information on end-of-life planning, such as sorting out personal belongings, preparing for a funeral, and consulting on inheritance and taxes" is a function that allows elderly people to obtain various information related to end-of-life planning.

[1142] "Means for converting speech to text using speech recognition software" refers to speech recognition technology for converting the senior's speech input into text data.

[1143] "Means for storing text data in a database" refers to database technology for safely storing the converted text data.

[1144] "Means for generating answers to user questions using a generative AI model" refers to an artificial intelligence model for generating appropriate answers to user questions.

[1145] "Means for converting text data into speech using speech synthesis software" refers to speech synthesis technology for converting text data into speech and playing it back to the user.

[1146] This invention is a system that enables elderly people to efficiently obtain information about end-of-life planning and smoothly carry out complex procedures such as recording memories and wills, sorting out belongings, preparing for funerals, and receiving advice on inheritance and taxes. A specific embodiment of this system is described below.

[1147] System configuration

[1148] The system consists of the following main components:

[1149] 1. Devices and means for communicating with the elderly

[1150] 2. Emotion recognition function means

[1151] 3. Functional means for generating gestures

[1152] 4. Generative AI methods for generating natural conversations

[1153] 5. Functional means of empathizing with and asking questions about what the elderly are saying

[1154] 6. A means of recording memories and wills

[1155] 7. Functional means for providing information on end-of-life planning, such as sorting out belongings, preparing for a funeral, and consulting on inheritance and taxes.

[1156] 8. Using speech recognition software to convert speech to text

[1157] 9. How to store text data in a database

[1158] 10. How to use generative AI models to generate answers to user questions

[1159] 11. Means of converting text data into speech using speech synthesis software

[1160] Hardware and software used

[1161] Devices: smartphones, tablets, computers, etc.

[1162] Speech Recognition Software: Popular Speech Recognition APIs

[1163] Database: A common database system (e.g., MySQL)

[1164] Generative AI model: A general generative AI model (e.g., OpenAI's GPT-3)

[1165] Speech synthesis software: General speech synthesis APIs

[1166] Program processing

[1167] Recording and reproducing wills

[1168] 1. When a user wishes to record a will, they access the system using a smartphone or tablet. For example, the user may say, "I want to record a will."

[1169] 2. The device uses speech recognition software to convert the user's speech into text, and also provides a text input interface that allows the user to input text directly.

[1170] 3. The server saves the converted text data or the directly entered text data in the database, along with metadata such as the user ID and timestamp.

[1171] 4. If the user wishes to replay the contents of the will, they can access the system again using their terminal and send a request saying, "I would like to replay the will."

[1172] 5. The server searches the database for the user's will data and sends the found data to the terminal.

[1173] 6. The device uses speech synthesis software to convert the text data into speech and play it back to the user.

[1174] Providing information on sorting belongings

[1175] 1. When a user wants information about sorting out belongings, they access the system using a tablet or smartphone. For example, the user types in the text, "Please tell me how to sort out belongings."

[1176] 2. The terminal provides an interface for users to input their questions. When the user inputs a question and presses the send button, the question is sent to the server.

[1177] 3. The server uses the generative AI model to generate an appropriate answer to the user's question. The generative AI model analyzes the content of the question and creates an answer based on related information.

[1178] 4. The server sends the generated answer to the device, optionally including related links and additional information along with the text data of the answer.

[1179] 5. The terminal displays the received answer to the user, who can review the answer and ask for more detailed information if necessary.

[1180] Examples of specific examples and prompts

[1181] Recording of wills

[1182] The user uses a smartphone and says, "I want to record my will."

[1183] The device uses speech recognition software to convert the speech into text and send it to the server.

[1184] The server stores the text data in a database.

[1185] Providing information on sorting belongings

[1186] The user uses a tablet to input into the system, "Please tell me how to organize the belongings of the deceased."

[1187] The server uses a generative AI model to generate answers such as, "When sorting through belongings, it is recommended that you first identify important documents and valuables, and then sort and dispose of unnecessary items."

[1188] The terminal displays the generated answer to the user.

[1189] In this way, the system responds to the user's needs regarding end-of-life planning. The flow of the specification process in the third embodiment will be described with reference to FIG.

[1190] Recording and reproducing wills

[1191] Recording of wills

[1192] Step 1:

[1193] When a user wishes to record a will, they access the system using a smartphone or tablet, for example by saying "I want to record a will."

[1194] Input: User's voice commands

[1195] Output: Audio data

[1196] Step 2:

[1197] The terminal uses speech recognition software to convert the user's speech into text.

[1198] Input: Audio data

[1199] Output: Text data

[1200] Step 3:

[1201] The terminal also provides a text input interface, allowing the user to input text directly.

[1202] Input: User text input

[1203] Output: Text data

[1204] Step 4:

[1205] The server saves the converted text data or the directly entered text data in a database, along with metadata such as the user ID and timestamp.

[1206] Input: Text data, user ID, timestamp

[1207] Output: Data stored in the database

[1208] Regeneration of a will

[1209] Step 1:

[1210] If the user wishes to replay the contents of the will, he or she accesses the system again using the terminal and sends a request saying, "I would like to replay the will."

[1211] Input: User request

[1212] Output: Request data

[1213] Step 2:

[1214] The server searches the database for the corresponding user's will data and transmits the found data to the terminal.

[1215] Input: Request data, User ID

[1216] Output: Will data

[1217] Step 3:

[1218] The terminal uses speech synthesis software to convert the text data into speech and play it back to the user.

[1219] Input: Will data

[1220] Output: Audio data

[1221] Providing information on sorting belongings

[1222] Step 1:

[1223] When a user wants information about sorting out belongings, they access the system using a tablet or smartphone. For example, the user can enter the text "Tell me how to sort out belongings."

[1224] Input: User question

[1225] Output: Question data

[1226] Step 2:

[1227] The terminal provides an interface for the user to input a question, and when the user inputs a question and presses the send button, the question is sent to the server.

[1228] Input: Question data

[1229] Output: Request to server

[1230] Step 3:

[1231] The server uses a generative AI model to generate appropriate answers to user questions. The generative AI model analyzes the content of the question and creates an answer based on related information.

[1232] Input: Question data

[1233] Output: Response data

[1234] Step 4:

[1235] The server then sends the generated answer to the device, which may include related links and additional information along with the text data of the answer.

[1236] Input: Answer data

[1237] Output: Response data to the terminal

[1238] Step 5:

[1239] The terminal displays the received answer to the user, who can then review the displayed answer and request more detailed information if necessary.

[1240] Input: Answer data

[1241] Output: Data displayed to the user

[1242] In this way, the system responds to users' end-of-life needs.

[1243] (Application example 3)

[1244] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1245] When elderly people prepare for the end of their lives, they need to centrally manage and safely store a wide range of information, such as recording wills, sorting belongings, and consulting about inheritance and taxes. However, because this information is highly private and security is important, it is difficult to manage it properly. In addition, it is a great burden for elderly people to organize this information themselves, so efficient support is required.

[1246] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes an automatic device means for interacting with the elderly, an emotion recognition function means, a function means for generating gestures and movements, a generative AI means for generating natural conversations, a function means for empathizing with what the elderly is saying and asking questions, a function means for recording memories and wills, a function means for providing information on end-of-life planning such as sorting out belongings, funeral preparations, and consultations on inheritance and taxes, a function means for converting voice input into text and saving it, and a function means for saving the contents of wills in JSON format. This enables the elderly to centrally manage and safely save information on end-of-life planning.

[1247] "Automatic device means" refers to a mechanical device for interacting with the elderly, and has the function of communicating with the elderly through voice input, output, and movement.

[1248] The "emotion recognition function means" has the function of analyzing and recognizing the emotional state of the elderly person from their voice, facial expression, movements, etc.

[1249] The "functional means for generating gestures and hand movements" has a function for generating natural gestures and hand movements during conversation to facilitate communication with the elderly.

[1250] "Generative AI means" refers to a system that uses artificial intelligence technology to generate natural conversations and has the functionality to naturally engage in dialogue with the elderly.

[1251] The "empathy and questioning function means" has the function of deepening dialogue by showing empathy for what the elderly person is saying and asking appropriate questions.

[1252] A "means for recording memories and wills" is a device that has the function of recording the contents of memories and wills spoken by elderly people and playing them back as needed.

[1253] "End-of-life planning information provision function means" is a means that has the function of providing information regarding end-of-life planning, such as sorting out belongings, preparing for a funeral, and consulting about inheritance and taxes.

[1254] The "voice input text conversion means" has a function for converting the voice spoken by the elderly person into text data.

[1255] The "text storage means" has a function for safely storing the converted text data.

[1256] A "JSON format storage means" is a means that has the function of saving the contents of a will in JSON format.

[1257] The system for implementing this invention includes an automated device that interacts with the elderly, an emotion recognition function, a function to generate gestures and movements, a generative AI that generates natural conversations, a function to empathize with what the elderly is saying and ask questions, a function to record memories and wills, and a function to provide information on end-of-life planning such as sorting belongings, preparing for a funeral, and consulting about inheritance and taxes, a function to convert voice input into text and save it, and a function to save the contents of a will in JSON format.

[1258] Hardware and software used

[1259] Hardware: Smartphone (microphone, storage)

[1260] Software: Python

[1261] System Operation

[1262] 1. Voice input: The user speaks their will into the smartphone's microphone. The smartphone captures the voice and saves it as audio data.

[1263] 2. Speech recognition: Using a library on the smartphone, the captured voice data is converted into text, using a specified API to perform highly accurate speech recognition.

[1264] 3. Text storage: The converted text data is saved in JSON format in the smartphone's storage, ensuring the contents of the will are stored securely.

[1265] Specific examples

[1266] For example, if a user says, "My will is to leave all my assets to my son," the smartphone captures this speech and uses the library to convert it into text, which becomes "My will is to leave all my assets to my son," and stores it in JSON format.

[1267] Prompt Sentence Examples

[1268] "I want to record my will. I want to say, 'My will is to leave all my assets to my son.'"

[1269] In this way, elderly people can centrally manage and safely store information related to end-of-life planning.

[1270] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1271] Step 1:

[1272] The user speaks their will into the smartphone's microphone. The input is the user's voice, and the smartphone captures this voice. Specifically, the smartphone's microphone collects the voice data and temporarily stores it in its internal memory.

[1273] Step 2:

[1274] The device converts the captured voice data into text using the library. The input is voice data and the output is text data. Specifically, the device sends voice data and receives text data.

[1275] Step 3:

[1276] The device saves the converted text data in JSON format. The input is text data, and the output is a JSON format file. Specifically, the device converts the text data into JSON format and saves it in local storage.

[1277] Step 4:

[1278] The device checks the contents of the saved JSON file and notifies the user that the save is complete. The input is a JSON file and the output is a notification message. Specifically, the device loads the JSON file, checks its contents, and then displays a notification message to the user.

[1279] Step 5:

[1280] The user can play back the saved will as needed. The input is the user's playback request, and the output is the display of text data or audio playback. Specifically, the device reads the saved JSON file and displays the text data or plays it back using speech synthesis technology.

[1281] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1282] "Example 1"

[1283] In one embodiment of the present invention, a robot that interacts with an elderly person includes an emotion engine that recognizes the user's emotions. This emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc., and conveys this information to the robot. The robot adjusts the content of the interaction based on this information. For example, if the user expresses joy, the robot will engage in a dialogue that matches that joy. On the other hand, if the user expresses sadness, the robot will engage in an empathetic dialogue.

[1284] "Example 2"

[1285] In another embodiment of the present invention, the generative AI tailors the dialogue based on the user's emotional state. To do this, the generative AI uses information from the emotion engine. For example, if the user expresses joy, the generative AI generates dialogue that matches that joy. Alternatively, if the user expresses sadness, the generative AI generates empathetic dialogue.

[1286] "Example 3"

[1287] In yet another embodiment of the present invention, a robot and a generative AI work together to respond to a user's emotions. Specifically, the robot uses an emotion engine to recognize the user's emotions and communicates that information to the generative AI. The generative AI then adjusts the dialogue content based on that information. For example, if the user expresses joy, the robot will use gestures that match that joy, and the generative AI will generate dialogue that matches that joy.

[1288] The processing flow of each embodiment will be described below.

[1289] "Example 1"

[1290] Step 1: The robot starts a dialogue with the user.

[1291] Step 2: The emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1292] Step 3: The emotion engine communicates this information to the robot.

[1293] Step 4: The robot adjusts the dialogue based on this information.

[1294] "Example 2"

[1295] Step 1: The generative AI begins a dialogue with the user.

[1296] Step 2: The emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1297] Step 3: The emotion engine communicates this information to the generative AI.

[1298] Step 4: The generative AI adjusts the dialogue based on this information.

[1299] "Example 3"

[1300] Step 1: The robot starts a dialogue with the user.

[1301] Step 2: The emotion engine recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1302] Step 3: The emotion engine communicates this information to the robot and generative AI.

[1303] Step 4: The robot adjusts its gestures based on this information, and the generative AI adjusts the dialogue based on this information.

[1304] Example 1

[1305] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1306] When interacting with the elderly, it is necessary to recognize their emotions appropriately and generate natural conversations and gestures accordingly. It is also important to provide information and record-keeping support for elderly people regarding end-of-life planning. However, conventional systems often lack sufficient functionality in terms of emotion recognition accuracy, natural conversation, and provision of end-of-life information. For this reason, there is a need for a conversational robot system that the elderly can use with confidence.

[1307] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1308] In this invention, the server includes a means for capturing the user's facial expressions and voice, a means for recognizing the user's emotions using an emotion engine, and a means for transmitting the recognized emotion information to the robot. This makes it possible to accurately recognize the emotions of the elderly and generate natural conversations and gestures based on those emotions. Furthermore, the system can also provide information about end-of-life planning and support for recording, making it possible to provide an interactive robot system that the elderly can use with peace of mind.

[1309] A "robot that interacts with the elderly" is an autonomous mechanical device designed to engage in natural dialogue with the elderly.

[1310] "Emotion recognition means" is a technology for analyzing and recognizing emotions from a user's facial expressions, tone of voice, choice of words, etc.

[1311] The "means for generating gestures" is a technology that enables a robot to make appropriate gestures according to the user's emotions and the content of the conversation.

[1312] "Generative AI means for generating natural conversations" refers to artificial intelligence technology that generates appropriate responses in order to have natural conversations with users.

[1313] "Methods for empathizing with and asking questions about what the elderly are saying" are techniques for showing empathy for what the elderly are saying and asking appropriate questions.

[1314] "Means for recording memories and wills" refers to technology for recording the memories and wills of elderly people.

[1315] "Means of providing information on end-of-life planning" refers to technology for providing information on end-of-life planning, such as sorting out belongings, preparing for a funeral, and consulting on inheritance and taxes.

[1316] "Means for capturing the user's facial expressions and voice" refers to technology for acquiring the user's facial expressions and voice using a camera or microphone.

[1317] "Means for recognizing a user's emotions using an emotion engine" refers to a technology for analyzing acquired facial and voice data to recognize a user's emotions.

[1318] The "means for transmitting the recognized emotional information to the robot" is a technology for transmitting the emotional information recognized by the emotion engine to the robot.

[1319] The "means for generating dialogue content based on emotional information" is a technology for generating appropriate dialogue content based on the recognized emotional information.

[1320] "Means for generating gestures that match emotions" is a technology that enables a robot to make appropriate gestures based on the recognized emotional information.

[1321] "Means for generating appropriate dialogue content using a generative AI model" refers to technology for using a generative AI model to generate appropriate responses in dialogue with a user.

[1322] This invention relates to a robot system that interacts with the elderly, and includes functions such as emotion recognition, gesture generation, generative AI that generates natural conversations, the ability to empathize with and ask questions about what the elderly are saying, and the ability to record memories and wills and provide information about end-of-life planning.

[1323] Hardware and software used

[1324] This system uses the following hardware and software:

[1325] Robot body: An autonomous mechanical device for interacting with the elderly. Equipped with a camera, microphone, speaker, motor, etc.

[1326] Emotion engine: Software that analyzes and recognizes emotions from a user's facial expressions, tone of voice, choice of words, etc.

[1327] Generative AI: An artificial intelligence technology for generating natural conversations. It uses generative AI models to generate appropriate dialogue content.

[1328] Server: A computer system that runs the emotion engine and transmits the recognized emotion information to the robot.

[1329] Program processing

[1330] The server acquires data from the robot's camera and microphone using a means of capturing the user's facial expressions and voice. The acquired data is analyzed using an emotion engine to recognize the user's emotions. The recognized emotion information is then sent from the server to the robot.

[1331] The robot generates dialogue content based on the emotional information it receives. Using generative AI, it engages in natural dialogue that reflects the user's emotions. For example, if the user expresses joy, the robot will respond with, "That's wonderful!" The robot also generates gestures that match the user's emotions, creating a friendly atmosphere for the user.

[1332] Specific examples

[1333] As a concrete example, consider the case where the user says, "I'm very happy today." In this case, the following occurs:

[1334] 1. The user says to the robot, "I'm very happy today."

[1335] 2. The robot captures the user's smile with a camera and records their voice with a microphone.

[1336] 3. The server sends the captured data to the emotion engine, which then recognizes the emotion as "joy."

[1337] 4. The server sends the information "The user is happy" to the robot.

[1338] 5. The robot generates the dialogue "That's great!"

[1339] 6. The robot generates gestures that express joy, such as waving.

[1340] 7. The robot will say "That's great!" and make a waving motion.

[1341] Prompt Sentence Examples

[1342] Examples of prompts to input to a generative AI model might include:

[1343] If the user says, "I'm very happy today," how should the robot respond and what gestures should it make?

[1344] Using this prompt, the generative AI model can suggest appropriate dialogue content and gestures.

[1345] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1346] Step 1:

[1347] The user talks to the robot.

[1348] Input: User's voice and facial expressions.

[1349] Specific action: The user speaks to the robot, saying, "I'm very happy today."

[1350] Step 2:

[1351] The robot captures the user's facial expressions and voice.

[1352] Input: User's voice and facial expressions.

[1353] Output: Captured voice and facial expression data.

[1354] Specific operation: The robot's camera takes a picture of the user's face and the microphone records the user's voice.

[1355] Step 3:

[1356] The server uses an emotion engine to recognize the user's emotion.

[1357] Input: Captured voice and facial expression data.

[1358] Output: Recognized emotion information.

[1359] Specific operation: The server sends the captured data to the emotion engine, which then recognizes "happiness" from the user's tone of voice and facial expression.

[1360] Step 4:

[1361] The server transmits the recognized emotional information to the robot.

[1362] Input: Recognized emotion information.

[1363] Output: Emotion information sent to the robot.

[1364] Specific operation: The server sends the information "The user is happy" to the robot.

[1365] Step 5:

[1366] The robot generates dialogue content based on emotional information.

[1367] Input: Emotion information sent to the robot.

[1368] Output: The generated dialogue.

[1369] What it does: The robot uses a generative AI model to generate the dialogue "That's great!"

[1370] Step 6:

[1371] The robot generates gestures that match its emotions.

[1372] Input: Emotion information sent to the robot.

[1373] Output: The generated gesture.

[1374] Specific actions: The robot generates gestures that express joy, such as waving.

[1375] Step 7:

[1376] The robot will interact with the user and perform gestures.

[1377] Input: Generated dialogue and gestures.

[1378] Output: Dialogue and gesture execution for the user.

[1379] Specific action: The robot waves its hand while saying "That's great!"

[1380] (Application example 1)

[1381] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1382] For elderly people to live safely at home, it is necessary to properly recognize their emotional state and respond quickly when abnormal emotional states are detected. However, current systems lack the functionality to recognize the emotional state of elderly people in real time and to conduct appropriate dialogue or emergency contact, making it difficult to fully ensure the safety of elderly people.

[1383] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a robot means for interacting with the elderly, an emotion recognition function means, and a function means for generating gestures. This makes it possible to recognize the emotional state of the elderly in real time, to conduct an appropriate dialogue when an abnormal emotional state is detected, and to send a notification to an emergency contact if necessary.

[1384] A "robot that interacts with the elderly" is an automated device that can have natural conversations with the elderly.

[1385] The "emotion recognition function" is a function that recognizes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1386] The "function of generating gestures and movements" is a function that allows the robot to generate gestures and movements according to the emotional state of the user.

[1387] "Generative AI that generates natural conversations" is an artificial intelligence technology that allows for natural dialogue with users.

[1388] The "function of empathizing with and asking questions about what the elderly are saying" refers to the function of showing empathy for what the elderly are saying and asking appropriate questions.

[1389] "Recording memories and wills" is a function that records the memories and wills of elderly people.

[1390] "Sorting out belongings and preparing for funerals" is a function that assists elderly people in sorting out their belongings and preparing for funerals.

[1391] The "function to provide information on end-of-life planning, such as consultations on inheritance and taxes" is a function that provides information on inheritance and taxes to support elderly people in their end-of-life planning.

[1392] "A function that recognizes the emotional state of the elderly, engages in appropriate dialogue when an abnormal emotional state is detected, and sends a notification to an emergency contact if necessary" is a function that recognizes the emotional state of the elderly in real time, engages in appropriate dialogue when an abnormal emotional state is detected, and sends a notification to an emergency contact if necessary.

[1393] To implement the present invention, the following system configuration and program are required.

[1394] System Configuration

[1395] 1. Hardware:

[1396] Camera: Used to capture the facial expressions and movements of the elderly in real time.

[1397] Robot body: A device for interacting with the elderly and making gestures.

[1398] Emergency Contact System: A system for sending notifications to emergency contacts when abnormal emotional states are detected.

[1399] 2. Software:

[1400] Library A: A library for video processing.

[1401] Library B: A library for emotion recognition.

[1402] Library C: A library for controlling the robot's movements.

[1403] System A: A system for sending notifications to emergency contacts.

[1404] Program processing

[1405] The server acquires video footage from the camera in real time and uses Library B to recognize the emotional state of the elderly person. Library C is used to generate the robot's behavior and dialogue content depending on the recognized emotion. For example, if the elderly person is feeling happy, the robot will use gestures that match the joy and say, "It's a nice day today!" On the other hand, if the elderly person is feeling scared, the robot will say, "Are you okay? Is there anything I can help you with?" and at the same time use System A to send a notification to the emergency contact that "The elderly person is feeling scared."

[1406] Specific examples

[1407] For example, if an elderly person suddenly feels frightened while alone at home, the camera will capture their facial expression and Library B will recognize the emotion of fear. The robot will then ask, "Are you OK? Is there anything I can help you with?", and the emergency contact system will simultaneously send a notification to the emergency contact. In this way, the safety of the elderly can be ensured.

[1408] Prompt Sentence Examples

[1409] Generate a program for a security robot that recognizes the emotional state of an elderly person in real time, and if they feel fear or anxiety, will engage in appropriate dialogue and notify emergency contacts.

[1410] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1411] Step 1:

[1412] The server acquires video from the camera in real time. The input is the video data from the camera and the output is video frames, which are used in subsequent processing steps.

[1413] Step 2:

[1414] The server passes the acquired video frames to Library B for emotion recognition. The input is the video frames, and the output is the recognized emotion data. Specifically, emotions are analyzed from facial expressions, tone of voice, choice of words, etc.

[1415] Step 3:

[1416] The server uses Library C to generate robot behavior and dialogue content based on the recognized emotion data. The input is emotion data, and the output is the robot's movement instructions and dialogue content. For example, if the emotion of joy is recognized, the robot will make gestures that match the joy and say, "Today is a nice day!"

[1417] Step 4:

[1418] If the server recognizes an abnormal emotional state (e.g., fear or anxiety), it uses System A to send a notification to the emergency contact. The input is the abnormal emotional data, and the output is a notification to the emergency contact. Specifically, it sends a message saying, "The elderly person is feeling fearful."

[1419] Step 5:

[1420] The server executes the robot's actions and dialogue. The input is the robot's action instructions and dialogue content, and the output is the robot's actual actions and dialogue. Specifically, the robot uses gestures and dialogue appropriately.

[1421] Example 2

[1422] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1423] In dialogue with the elderly, there are issues with insufficient emotional recognition and empathetic response generation. Furthermore, information about end-of-life planning may not be provided appropriately. Furthermore, there is also the problem of difficulty in adjusting the content of the dialogue according to the user's emotional state.

[1424] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a device means for interacting with the elderly, an emotion recognition means, a means for generating gestures and movements, a generative AI means for generating natural conversations, a means for empathizing with the elderly and asking questions, a means for recording memories and wills, a means for providing information on end-of-life planning such as sorting belongings, funeral preparations, and inheritance and tax consultations, a means for receiving user input, a means for preprocessing data, a means for analyzing the user's emotional state using an emotion recognition engine, a means for generating a response based on the user's input and emotional state using a generative AI model, and a means for providing the generated response to the user. This improves emotion recognition and empathetic response generation in dialogue with the elderly, enabling appropriate provision of information on end-of-life planning. It also facilitates adjusting the content of the dialogue according to the user's emotional state.

[1425] The "device means for interacting with the elderly" is a device including hardware and software for interacting with the elderly through voice and text.

[1426] "Emotion recognition means" is a technology for analyzing and identifying emotions from a user's voice or text.

[1427] The "means for generating gestures" is a technology for generating and displaying natural gestures and hand movements during a conversation.

[1428] "Generative AI means for generating natural conversations" refers to artificial intelligence technology for generating natural conversations based on user input.

[1429] "Means for empathizing with and asking questions about what the elderly are saying" is a technique for generating empathetic responses and questions to what the elderly are saying.

[1430] "Means for recording memories and wills" refers to technology for recording and preserving the memories and wills of the elderly.

[1431] "Means of providing information on end-of-life planning, such as sorting out personal belongings, preparing for a funeral, and consulting on inheritance and taxes" refers to technology for providing information related to end-of-life planning.

[1432] The "means for receiving user input" refers to technology for receiving voice or text data input by the user.

[1433] "Means for preprocessing data" refers to technology for converting received data into a format that is easy to analyze.

[1434] The "means for analyzing the emotional state of a user using an emotion recognition engine" is a technique for identifying the emotional state of a user using an emotion recognition engine.

[1435] "Means for generating responses based on a user's input and emotional state using a generative AI model" refers to technology for using a generative AI model to generate responses appropriate to a user's input and emotional state.

[1436] The "means for providing the generated response to the user" refers to a technique for providing the generated response to the user by display or audio.

[1437] This invention relates to a system for interacting with elderly people, and includes technologies for recognizing emotions, generating empathetic responses, and providing information about end-of-life planning. Specific embodiments of this system are described below.

[1438] Hardware and Software Configuration

[1439] server:

[1440] Hardware: A server with a powerful GPU (e.g., a powerful graphics processing unit)

[1441] Software: Natural language processing libraries (e.g., natural language processing libraries), emotion recognition engines (e.g., emotion recognition engines)

[1442] Device:

[1443] Hardware: Smartphone, tablet, or dedicated interactive device

[1444] Software: speech recognition software (e.g., voice recognition software), speech synthesis software (e.g., speech synthesis software)

[1445] System Operation

[1446] Receiving user input:

[1447] The user inputs voice or text data through the terminal, for example, by speaking into the microphone of the smartphone or by typing text on the keyboard.

[1448] Data preprocessing:

[1449] The server converts the received data into text format, denoising and tokenising it, and the audio data is converted to text using speech recognition software.

[1450] Emotion recognition:

[1451] The server uses an emotion recognition engine to analyze the user's emotional state. For example, the server inputs text data into the emotion recognition engine to identify the user's emotional state (e.g., joy, sadness, anger, etc.).

[1452] Conversation generation:

[1453] The server uses a generative AI model to generate a response based on the user's input and emotional state. The generative AI model receives a prompt and generates an appropriate response.

[1454] Sending a response:

[1455] The server sends the generated response to the terminal for delivery to the user, and the terminal displays the generated response to the user, or uses speech synthesis software to convert the text to speech if a voice response is required.

[1456] Specific examples

[1457] Example 1: Elderly people talking about past memories

[1458] User input: "I went on a trip with my family a long time ago and it was so much fun."

[1459] Generative AI response: "That's a wonderful memory. Where did you travel?"

[1460] Example 2: When an elderly person expresses doubts about end-of-life planning

[1461] User input: "I don't know where to start when it comes to end-of-life planning."

[1462] Generative AI response: "As a first step in preparing for your end, we recommend that you organize your assets and important documents."

[1463] Prompt Sentence Examples

[1464] "Generate an empathetic response when the user talks about a past memory."

[1465] "When a user expresses a question about end-of-life planning, generate an answer to that question."

[1466] In this way, the server, terminal, and user each play their respective roles, and the entire system operates smoothly, making conversation with the elderly more natural and empathetic.

[1467] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1468] Step 1: Receiving User Input

[1469] Description: The user inputs voice or text data through the terminal.

[1470] Input: User voice or text data

[1471] Output: Audio or text data sent to the device

[1472] What it does: A user speaks into the microphone of their smartphone or tablet or enters text on the keyboard, and the device sends this data to a server.

[1473] Step 2: Preprocessing the data

[1474] Description: Converts data received by the server into text format, denoising and tokenising it.

[1475] Input: Voice or text data sent from the device

[1476] Output: Preprocessed text data

[1477] What it does: The server uses speech recognition software to convert the audio data into text, then filters out unwanted noise from the text and breaks it into tokens for words and phrases.

[1478] Step 3: Emotion Recognition

[1479] Description: The server uses an emotion recognition engine to analyze the user's emotional state.

[1480] Input: Preprocessed text data

[1481] Output: User's emotional state (e.g., happy, sad, angry, etc.)

[1482] Specific operation: The server inputs the text data into an emotion recognition engine to identify the user's emotional state. For example, the emotion recognition engine analyzes the text data and identifies that the user is feeling "joy."

[1483] Step 4: Conversation generation

[1484] Description: The server uses a generative AI model to generate responses based on the user's input and emotional state.

[1485] Input: User input and emotional state

[1486] Output: The generated response text

[1487] Specific operation: The server inputs a prompt sentence into the generative AI model, which generates a response appropriate to the user's emotional state and input. For example, if a user inputs, "I once went on a trip with my family and it was so much fun," the generative AI model will generate a response such as, "That's a wonderful memory. Where did you go on that trip?"

[1488] Step 5: Sending a Response

[1489] Description: The server sends the generated response to the terminal and provides it to the user.

[1490] Input: Generated response text

[1491] Output: The response provided to the user

[1492] What happens: The server sends the generated text response to the device, which displays it to the user. If a voice response is required, the device uses speech synthesis software to convert the text to speech.

[1493] (Application example 2)

[1494] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1495] When interacting with elderly people, there is a need for systems that can generate natural conversations and respond to what the elderly say through empathy and questions. Furthermore, there is a need for systems that can adjust the content of the conversation based on the elderly person's emotional state. Furthermore, there is a lack of methods to make interactions with elderly customers more natural and empathetic in brick-and-mortar stores.

[1496] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1497] In this invention, the server includes an automatic device means for interacting with the elderly, an emotion recognition function means, a gesture generation function means, a generative AI means for generating natural conversations, a function means for empathizing with what the elderly is saying and asking questions, a function means for recording memories and wills, a function means for sorting belongings and preparing for a funeral, a function means for providing information on end-of-life planning such as inheritance and tax consultations, an application means installed in smart glasses worn by staff in a physical store, a means for acquiring elderly customer utterances, a means for recognizing the emotional state from the customer's utterances, a means for generating natural responses based on the user's utterances and emotional state, and a means for displaying the generated responses on the display of the smart glasses. This makes it possible to make interactions with elderly customers in a physical store more natural and empathetic.

[1498] An "automatic device" is a mechanical device designed to interact with an older adult.

[1499] The "emotion recognition function" is a function that identifies the emotional state of a user from their speech and facial expressions.

[1500] The "function of generating gestures" is a function that generates natural movements and gestures during a conversation.

[1501] "Generative AI" is an artificial intelligence that generates natural conversations, empathizes with what the elderly are saying, and asks questions.

[1502] The "empathic function" is the ability to generate empathic responses to what the elderly person is saying.

[1503] The "question function" is a function that generates appropriate questions in response to what the elderly person is saying.

[1504] "Means for recording memories and wills" are means for recording the memories and wills of elderly people.

[1505] "Means for sorting belongings" are means for providing information and support regarding sorting belongings.

[1506] "Funeral preparation tools" are tools that provide information and support regarding funeral preparations.

[1507] "Inheritance and tax consultation means" refers to means for providing consultation and information regarding inheritance and taxes.

[1508] A "brick and mortar store" is a store that exists in a physical location.

[1509] "Smart glasses" are glasses-type devices equipped with a display that displays information.

[1510] An "application means" is software for performing a specific function.

[1511] "Means for acquiring statements" refers to a means for acquiring statements from elderly customers.

[1512] The "emotional state recognition means" is a means for recognizing the emotional state of a customer from their speech.

[1513] The "response generation means" is a means for generating natural responses based on the user's utterances and emotional state.

[1514] The "response display means" is a means for displaying the generated response on the display of the smart glasses.

[1515] This invention is a system for making interactions with elderly customers in brick-and-mortar stores more natural and empathetic. The system is implemented using an application installed on smart glasses.

[1516] System configuration

[1517] The system includes the following hardware and software:

[1518] Hardware: Smart glasses (e.g., Google Glass)

[1519] Software: OpenAI API, Emotion Recognition Library

[1520] Program processing explanation

[1521] The server implements the system using the following means:

[1522] 1. Speech acquisition method: Speech from elderly customers is acquired through the microphone in the smart glasses, and the user's speech is input into the system in real time.

[1523] 2. Emotional State Recognition: Recognizes the emotional state from customer utterances using the Emotion Recognition library, which identifies the user's emotional state and provides data to generate an appropriate response.

[1524] 3. Response generation: Using the OpenAI API, natural responses are generated based on the user's utterances and emotional state. The generative AI model uses the prompt sentence to generate an appropriate response.

[1525] 4. Response display means: The generated response is displayed on the smart glasses display, allowing staff to provide natural conversations with elderly customers.

[1526] Specific examples

[1527] For example, if an elderly customer is talking about past travels, the following dialogue is possible:

[1528] User says: "I once went on a trip to Kyoto with my family."

[1529] Generative AI response: "That's a wonderful memory. What part of Kyoto is most memorable to you?"

[1530] Prompt Sentence Examples

[1531] Below are some example prompts that the generative AI uses to generate responses:

[1532] User's emotional state: joy

[1533] User says: I once went on a trip to Kyoto with my family.

[1534] Generative AI response:

[1535] In this way, interactions with elderly customers in brick-and-mortar stores can be made more natural and empathetic.

[1536] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1537] Step 1:

[1538] The server receives the elderly customer's speech through the microphone in the smart glasses. The input is the elderly customer's voice, and the output is text-format speech data. Specifically, it uses voice recognition software to convert the speech into text.

[1539] Step 2:

[1540] The server inputs the acquired utterance data into the Emotion Recognition library to recognize the emotional state. The input is text-format utterance data, and the output is the emotional state (e.g., joy, sadness, surprise, etc.). Specifically, emotions are identified using a text analysis algorithm.

[1541] Step 3:

[1542] The server inputs prompt sentences into the generative AI model based on the utterance data and the recognized emotional state, and generates natural responses. The input is the utterance data and the emotional state, and the output is the generated response text. Specifically, the following prompt sentences are input into the generative AI model:

[1543] User's emotional state: joy

[1544] User says: I once went on a trip to Kyoto with my family.

[1545] Generative AI response:

[1546] Step 4:

[1547] The server displays the generated response text on the display of the smart glasses. The input is the generated response text, and the output is the text displayed on the display of the smart glasses. Specifically, the text is displayed using display control software of the smart glasses.

[1548] Step 5:

[1549] The user (staff member) checks the response displayed on the smart glasses display and engages in a natural dialogue with the elderly customer. The input is the response text displayed on the display, and the output is the dialogue with the elderly customer. Specifically, the staff member reads out the displayed text or uses it as a reference to proceed with the dialogue.

[1550] In this way, interactions with elderly customers in brick-and-mortar stores can be made more natural and empathetic.

[1551] Example 3

[1552] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1553] When elderly people are preparing for the end of their lives, they need a system that can provide centralized support for recording and playing back wills, providing information on sorting belongings, and providing dialogue that responds to their emotions. However, in conventional systems, these functions are distributed, which means that users have to use multiple systems, which is inconvenient. In addition, the system lacks the functionality to provide dialogue that responds to the elderly's emotions, limiting the user experience.

[1554] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes a robot means for interacting with the elderly, an emotion recognition means, a means for generating gestures and movements, a generative AI means for generating natural conversations, a means for empathizing with the elderly and asking questions, a means for recording memories and wills, a means for providing information on end-of-life planning such as sorting out belongings, funeral preparations, and inheritance and tax consultations, a means for saving the contents of the will in a database, a means for playing back the saved will, a means for searching and providing information on sorting out belongings from the database, a means for generating dialogue content based on the user's emotions, and a means for performing gestures and movements in accordance with the generated dialogue content. This allows a single system to meet the diverse needs of the elderly regarding end-of-life planning and provides a richer user experience through dialogue tailored to emotions.

[1555] "Elderly" refers to people who are older and who especially need to prepare for the end of their lives.

[1556] "Interactive robot means" refers to a mechanical device for conducting natural conversations with elderly people.

[1557] "Emotion recognition means" refers to technology for detecting and recognizing the emotional state of an elderly person.

[1558] "Means for generating gestures" refers to the functionality that enables the robot to make appropriate gestures while interacting with the elderly.

[1559] "Generative AI means" refers to artificial intelligence technology for generating natural conversations with the elderly.

[1560] "Means of empathizing and asking questions" refers to the function of showing empathy for what the elderly person is saying and asking appropriate questions.

[1561] "Means for recording memories and wills" refers to technology that allows elderly people to record their memories and wills.

[1562] "Means of providing information about end-of-life planning" refers to the function of providing elderly people with information related to end-of-life planning, such as sorting out personal belongings, preparing for a funeral, and consulting about inheritance and taxes.

[1563] "Means for storing the contents of a will in a database" refers to technology for storing the contents of a will entered by an elderly person in a database.

[1564] "Means for reproducing a stored will" refers to the function for reproducing the contents of a will stored in a database.

[1565] "Means for searching and providing information on sorting out belongings from a database" refers to technology for searching information on sorting out belongings from a database and providing it to elderly people.

[1566] "Means for generating dialogue content based on the user's emotions" refers to a technology for generating appropriate dialogue content according to the user's emotional state.

[1567] "Means for making gestures that match the content of the generated dialogue" refers to a function that enables the robot to make appropriate gestures in accordance with the content of the generated dialogue.

[1568] MODE FOR CARRYING OUT THE INVENTION

[1569] This invention is a system that provides comprehensive support for elderly people in preparing for the end of life by providing them with the information and dialogue they need. The system is realized using hardware and software such as robots, emotion recognition functions, generative AI, and databases.

[1570] Hardware and software used

[1571] Robot: A mechanical device for interacting with the elderly, with the ability to recognize emotions and generate gestures.

[1572] Emotion Recognition Engine: Software for detecting and recognizing the emotional state of older adults.

[1573] Generative AI: Artificial intelligence technology for generating natural conversations.

[1574] Database: Data storage for storing information about the contents of wills and end-of-life planning.

[1575] Terminal: A device (e.g., tablet, smartphone) that provides an interface for the elderly to interact with the system.

[1576] Program processing

[1577] 1. If a user wishes to record a will, they enter the contents of the will into the system via their terminal.

[1578] 2. The server stores the entered will in a database.

[1579] 3. If the user wishes to regenerate the will, he / she sends a request to the system through his / her terminal.

[1580] 4. The server searches for the saved contents of the will and plays it on the device.

[1581] 5. If the user wishes to request information about estate clearance, he or she sends a request to the system via the terminal.

[1582] 6. The server searches the database for information regarding sorting of belongings and provides it to the terminal.

[1583] 7. The robot uses an emotion engine to recognize the user's emotions and conveys that information to the generative AI.

[1584] 8. The generative AI generates dialogue content based on that information, and the robot uses gestures that match the generated dialogue content.

[1585] Specific examples

[1586] The user requests the system to record a will.

[1587] The device will prompt you to "Enter your will."

[1588] The user types, "Please divide all my assets among my children."

[1589] The server stores this information in a database.

[1590] The user requests "play the will."

[1591] The terminal sends a playback request to the server.

[1592] The server searches for the stored will and sends it to the device.

[1593] The device will play back the contents of the will as audio or text.

[1594] Prompt Sentence Examples

[1595] "I would like to record my will. Please enter the details."

[1596] "Please tell me how to sort out my belongings."

[1597] "Generate dialogue for when the user is happy."

[1598] This system can satisfy the diverse needs of elderly people regarding end-of-life planning in one system and provide a richer user experience through dialogue that responds to their emotions. The flow of the specific processing in Example 3 will be described with reference to FIG.

[1599] Program processing flow

[1600] Recording and reproducing wills

[1601] Step 1:

[1602] When a user wishes to record a will, he or she sends a request to the system through a terminal. The input is a request saying "I want to record a will." The terminal receives this request and displays a prompt saying "Please enter the contents of your will."

[1603] Step 2:

[1604] The user inputs the contents of the will. The input is text data such as "Please divide all my assets among my children." The device receives this input and sends it to the server.

[1605] Step 3:

[1606] The server saves the received will in a database. The input is the text data of the will, and the output is the record of the will saved in the database. The server notifies the terminal that saving is complete.

[1607] Step 4:

[1608] When a user wishes to replay a will, he or she sends a request to the system through the terminal. The input is a request to "replay the will." The terminal receives this request and sends it to the server.

[1609] Step 5:

[1610] The server retrieves the saved will from the database. The input is the playback request, and the output is the retrieved text data of the will. The server sends this data to the terminal.

[1611] Step 6:

[1612] The terminal plays back the received will as audio or text. The input is the text data of the will, and the output is the playback of audio or text.

[1613] Providing information on sorting belongings

[1614] Step 1:

[1615] When a user requests information about sorting out belongings, they send a request to the system through their terminal. The input is a request such as "Tell me how to sort out belongings." The terminal receives this request and sends it to the server.

[1616] Step 2:

[1617] The server retrieves information about estate clearance from a database. The input is an information request, and the output is the retrieved estate clearance information. The server sends this information to the terminal.

[1618] Step 3:

[1619] The terminal displays the received estate clearance information to the user. The input is estate clearance information data, and the output is the display of the information.

[1620] Collaboration between robots and generative AI

[1621] Step 1:

[1622] The user expresses their emotions. The input is "I'm very happy today." The robot analyzes this statement with its emotion recognition engine and recognizes the user's emotions.

[1623] Step 2:

[1624] The robot communicates the recognized emotional information to the generative AI. The input is the recognized emotional data, and the output is the transmission of emotional information to the generative AI.

[1625] Step 3:

[1626] The generative AI generates dialogue content based on emotional information. The input is emotional data, and the output is the generated dialogue content. The generative AI sends this dialogue content to the robot.

[1627] Step 4:

[1628] The robot makes gestures that match the generated dialogue content. The input is the generated dialogue content, and the output is the gestures.

[1629] Step 5:

[1630] The robot communicates the generated dialogue content to the user. The input is the generated dialogue content, and the output is the execution of the dialogue.

[1631] (Application example 3)

[1632] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1633] A lack of emotional support and concrete information is a challenge for elderly people when they are preparing for their end of life. In particular, there is a need for a system that provides a wide range of information related to end-of-life planning, such as recording wills, sorting belongings, funeral preparations, and consultations on inheritance and taxes. There is also a growing need for a system that can engage in appropriate dialogue while being sensitive to the emotions of elderly people.

[1634] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1635] In this invention, the server includes a robot means for interacting with the elderly, an emotion recognition means, a means for generating gestures and movements, a generative AI means for generating natural conversations, a means for empathizing with what the elderly is saying and asking questions, a means for recording memories and wills, a means for providing information on end-of-life planning such as sorting out belongings, funeral preparations, and consultations on inheritance and taxes, a means for generating dialogue content based on the elderly's emotions, a means for recording the contents of the will and playing it back as needed, a means for providing methods and points to note for sorting out belongings, a means for providing information on funeral preparations, and a means for providing consultations on inheritance and taxes. This makes it possible for elderly people to receive emotional support while centrally obtaining a wide range of information on end-of-life planning.

[1636] An "elderly conversation robot" is an autonomous mechanical device designed to have natural conversations with the elderly.

[1637] "Emotion recognition function" is a technology for analyzing and recognizing emotions from a user's facial expressions and voice.

[1638] The "function of generating gestures" is a function that enables the robot to perform natural movements while interacting with the user.

[1639] "Generative AI that generates natural conversations" is an artificial intelligence technology that generates natural dialogue based on user input.

[1640] The "function to empathize with and ask questions about what the elderly person is saying" is a function to show empathy for what the user is saying and to ask appropriate questions.

[1641] "Recording memories and wills" is a function that saves the contents of a user's memories and wills in digital format.

[1642] The "function to provide information on end-of-life planning, such as sorting out belongings, preparing for a funeral, and consulting on inheritance and taxes" is a function that allows users to obtain various information on end-of-life planning.

[1643] The "function of generating dialogue content based on the emotions of the elderly" is a function of generating appropriate dialogue content according to the emotional state of the user.

[1644] The "function to record the contents of a will and play it back as needed" is a function to record the user's will and play back the contents when needed.

[1645] The "function of providing methods and points to note for sorting out belongings of the deceased" is a function that provides the user with specific methods and points to note for sorting out belongings of the deceased.

[1646] The "function of providing information on funeral preparations" is a function that provides the user with information on funeral preparations.

[1647] The "function for providing advice on inheritance and taxes" is a function that allows users to receive advice on inheritance and taxes.

[1648] As an embodiment of the present invention, a system using a robot to support end-of-life planning will be described. This system includes a robot that can converse with elderly people, an emotion recognition function, a function to generate gestures, a generative AI that generates natural conversations, a function to empathize with what the elderly are saying and ask questions, and a function to record memories and wills, and provide information on end-of-life planning such as sorting out belongings, preparing for a funeral, and consulting on inheritance and taxes.

[1649] Hardware used

[1650] Robot Body: An autonomous mechanical device for interacting with the elderly

[1651] Camera and microphone: Devices for capturing the user's facial expressions and voice.

[1652] Software used

[1653] Emotion Recognition Library: A library for analyzing user emotions

[1654] AI Conversation Model: An artificial intelligence model for generating natural conversations (e.g., GPT-3, ChatGPT)

[1655] End of Life Services Module: A proprietary module for providing information on end-of-life planning

[1656] Data processing and calculation

[1657] 1. Emotion Recognition: The camera and microphone are used to capture the user's facial expressions and voice, and the Emotion Recognition Library is used to analyze their emotions, allowing the system to grasp the user's emotional state in real time.

[1658] 2. Dialogue generation: Based on the analyzed emotional data, the AI ​​Conversation Model generates appropriate dialogue content, enabling natural conversations that are sensitive to the user's emotions.

[1659] 3. Service provision: In response to user requests, the End of Life Services Module provides information on recording wills, sorting belongings, funeral preparations, and inheritance and tax consultations.

[1660] Specific examples

[1661] For example, if an elderly person says, "I want to record my will," the robot will recognize that emotion and use generative AI to respond, "Please tell me the contents of your will." The user can then speak the contents of their will, and the robot will record it and play it back as needed.

[1662] Prompt Sentence Examples

[1663] User: I want to record my will.

[1664] Robot: Please tell me what the will says.

[1665] User: I want my son to inherit my assets.

[1666] Robot: Understood. I have recorded the will. Is there anything else I can help you with?

[1667] In this way, end-of-life support robots can provide comprehensive support for elderly people's end-of-life preparations and provide emotionally sensitive dialogue.

[1668] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1669] Step 1:

[1670] emotion recognition

[1671] The server captures the user's facial expressions and voice using a camera and microphone. As input, it obtains the user's real-time video and audio data. It uses the Emotion Recognition Library to analyze these data and recognize the user's emotional state. As output, it obtains the user's emotional state (e.g., joy, sadness, anger, etc.).

[1672] Step 2:

[1673] Dialogue Generation

[1674] The server receives the user's emotional state obtained in step 1 as input. It uses an AI Conversation Model (e.g., GPT-3, ChatGPT) to generate appropriate dialogue content based on the user's emotions. The server uses the user's emotional state and the user's utterances as input. The server obtains the generated dialogue content as output.

[1675] Step 3:

[1676] Executing the dialogue

[1677] The terminal (robot) outputs the dialogue content generated in step 2 to the user as voice. Specifically, the dialogue content is played back as voice through the robot's speaker. This allows for natural dialogue with the user.

[1678] Step 4:

[1679] Service Request

[1680] The user requests a specific service from the robot (e.g., recording a will, advice on sorting out belongings, etc.). The robot receives the user's voice request as input. The robot then sends this request to the server.

[1681] Step 5:

[1682] Service provision

[1683] The server receives the user's request and provides the appropriate information using the End of Life Services Module. The server uses the user's request as input. For example, if a request is made to record a will, the server records the will's contents and executes the function of replaying it as needed. The server obtains the information and services to be provided to the user as output.

[1684] Step 6:

[1685] Output of information

[1686] The terminal (robot) outputs the information and services obtained in step 5 to the user by voice or on a display. Specifically, the information is provided by voice through the robot's speaker or displayed on a display, allowing the user to obtain the information they need.

[1687] Step 7:

[1688] Get feedback

[1689] The server receives feedback from the user. As input, it recaptures the user's voice and facial expressions and analyzes them using emotion recognition. As output, it obtains the user's satisfaction level and any additional requests. This allows the system to continuously provide services that meet the user's needs.

[1690] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1691] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1692] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1693] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1694] [Third embodiment]

[1695] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1696] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1697] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1698] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1699] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1700] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1701] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1702] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1703] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1704] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1705] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1706] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1707] "Example 1"

[1708] As one embodiment of the present invention, a robot that can interact with elderly people is provided. This robot has an emotion recognition function and can recognize the emotional state of the elderly person. It also has a function of generating gestures, which allows it to create a friendly relationship with the elderly person. Specifically, when the elderly person expresses joy, the robot generates gestures that match the joy.

[1709] "Example 2"

[1710] In addition, a generative AI is provided. This generative AI has the ability to generate natural conversations, empathize with what the elderly person is saying, and ask questions. Specifically, when an elderly person talks about their past memories, the generative AI generates an empathetic response based on that content. Also, when an elderly person expresses a question about end-of-life planning, the generative AI generates an answer to that question.

[1711] "Example 3"

[1712] Furthermore, the system has the function of providing information on end-of-life planning, such as recording memories and wills, sorting out belongings, preparing for funerals, and consulting on inheritance and taxes. Specifically, if an elderly person requests to record a will, the system will record the contents of the will and play it back as needed. Also, if an elderly person requests information on sorting out belongings, the system will provide information on how to sort out belongings and important points to note.

[1713] The processing flow of each embodiment will be described below.

[1714] "Example 1"

[1715] Step 1: The robot that interacts with the elderly is activated.

[1716] Step 2: The robot recognizes the emotional state of the elderly person using its emotion recognition function. For example, if the elderly person smiles, the robot recognizes the smile and determines that the elderly person is happy.

[1717] Step 3: The robot uses its gesture generation function to create a friendly atmosphere with the elderly. For example, if the elderly person expresses joy, the robot will generate a gesture that matches the joy.

[1718] "Example 2"

[1719] Step 1: The generative AI is launched.

[1720] Step 2: The generative AI listens to what the elderly person says and generates natural conversations based on that content. For example, if the elderly person talks about their past memories, the generative AI generates empathetic responses based on that content.

[1721] Step 3: When an elderly person expresses a question about end-of-life planning, the generative AI generates an answer to that question. For example, if an elderly person asks for information about sorting out their belongings, the generative AI will provide information on how to sort out their belongings and what to be careful about.

[1722] "Example 3"

[1723] Step 1: When an elderly person requests information about end-of-life planning, the system will provide the corresponding information. For example, if an elderly person requests to record a will, the system will record the contents of the will and play it back as needed.

[1724] Step 2: If an elderly person requests information about sorting out their belongings, the system will provide them with information on how to sort out their belongings and what to be careful about.

[1725] Step 3: If the elderly person requests information about other end-of-life matters, such as funeral arrangements, inheritance, or tax advice, the system will provide that information.

[1726] Example 1

[1727] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1728] In order for elderly people to live their daily lives without feeling lonely, it is important to provide appropriate information and dialogue that is sensitive to their emotions. However, with conventional robot systems, it was difficult to accurately recognize the emotional state of the elderly and generate natural dialogue and gestures accordingly. In addition, the provision of information and consultation functions regarding end-of-life planning was insufficient. As a result, it was not possible to alleviate the loneliness and anxiety felt by elderly people.

[1729] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1730] In this invention, the server includes a robot means for interacting with the elderly, an emotion recognition means, a means for generating gestures and movements, a generative AI means for generating natural conversations, a means for empathizing with what the elderly is saying and asking questions, a means for recording memories and wills, a means for providing information on end-of-life planning such as sorting out belongings, preparing for a funeral, and consultations on inheritance and taxes, and a means for recognizing the emotional state of the elderly, generating appropriate gestures and movements based on the recognized emotions, and executing the generated gestures and movements. This enables conversations and gestures that are sensitive to the emotions of the elderly, and also enhances the information and consultation functions on end-of-life planning.

[1731] A "robot that interacts with the elderly" is an automated device that can have natural conversations with the elderly.

[1732] "Emotion recognition" is a technology that analyzes facial expressions and voice data of elderly people to identify their emotional state (joy, sadness, anger, etc.).

[1733] "Gesture generation" is a technology that allows a robot to generate appropriate actions (such as waving or smiling) based on recognized emotions.

[1734] "Generative AI" is an artificial intelligence technology that generates natural conversations with the elderly.

[1735] "Empathizing and asking questions" is a technique for deepening dialogue by showing empathy for what the elderly person is saying and asking appropriate questions.

[1736] "Memories and Wills Recording" is a technology that stores the memories and wills of elderly people in digital format.

[1737] "Providing information on end-of-life planning" refers to technology that provides elderly people with information related to end-of-life planning, such as sorting out personal belongings, funeral preparations, inheritance, and tax consultations.

[1738] "Recognizing the emotional state, generating appropriate gestures based on the recognized emotions, and executing the generated gestures" refers to a series of processes in which the robot analyzes the emotions of the elderly person and performs actions in accordance with those emotions.

[1739] MODE FOR CARRYING OUT THE INVENTION

[1740] This invention relates to a robot system that interacts with the elderly, and includes functions such as emotion recognition, gesture generation, generative AI that generates natural conversations, the ability to empathize with and ask questions about what the elderly are saying, the ability to record memories and wills, and the ability to provide information about end-of-life planning.

[1741] 1. Program Generation

[1742] The server generates a program for the robot to interact with the elderly. This program includes an emotion recognition function and a gesture generation function. The emotion recognition function is implemented using, for example, TensorFlow. The gesture generation function is implemented using known technology.

[1743] 2. Program processing explanation

[1744] The server analyzes data acquired from the camera and microphone to realize the emotion recognition function. Specifically, it analyzes facial expressions in facial images acquired from the camera and analyzes audio data acquired from the microphone using TensorFlow. This makes it possible to recognize the emotional state of the elderly.

[1745] The terminal (robot) generates appropriate gestures based on the emotion information received from the server. The robot has movement patterns corresponding to the emotion, and selects and executes one of them.

[1746] 3. Examples and prompts

[1747] As a concrete example, consider the case where an elderly person says, "I'm very happy today." In this case, the server analyzes the voice input and recognizes that the elderly person is feeling happy. Next, the device (robot) generates gestures that match the emotion of joy (for example, waving, smiling, etc.), creating a friendly atmosphere toward the elderly person.

[1748] Example prompts to input to a generative AI model:

[1749] "What gestures should a robot make when an elderly person expresses joy?"

[1750] Using this prompt, the generative AI model can suggest appropriate gestures based on the elderly person's emotions.

[1751] In this way, the present invention can realize conversations and gestures that are sensitive to the feelings of the elderly, and can also enhance the provision of information and consultation functions regarding end-of-life planning.

[1752] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1753] Step 1:

[1754] The server acquires facial images and voice data of elderly people from the camera and microphone. The input is facial image data from the camera and voice data from the microphone, and the output is a state ready for analyzing these data. Specifically, the server initializes the camera and microphone devices and acquires data in real time.

[1755] Step 2:

[1756] The server performs facial expression analysis on the acquired facial image data. The input is facial image data acquired from the camera, and the output is the emotional state (happiness, sadness, anger, etc.) based on the elderly person's facial expression. Specifically, the server extracts feature points from the facial image and inputs them into an emotion recognition model to estimate the emotional state.

[1757] Step 3:

[1758] The server analyzes the acquired voice data using TensorFlow and estimates emotions from the tone of voice and the content of the words. The input is voice data acquired from a microphone, and the output is the emotional state based on the voice. Specifically, the server preprocesses the voice data and inputs it into a voice recognition model to estimate the emotional state.

[1759] Step 4:

[1760] The server integrates the results of facial expression analysis and voice analysis to recognize the overall emotional state of the elderly person. The input is the results of facial expression analysis and voice analysis, and the output is the overall emotional state of the elderly person. Specifically, the server weights and integrates the results of both analyses to determine the final emotional state.

[1761] Step 5:

[1762] The terminal (robot) generates appropriate gestures based on the emotional information received from the server. The input is the emotional information from the server, and the output is the generated gesture movement sequence. As a specific movement, the terminal selects a movement pattern according to the emotion and generates a motor control signal to execute this.

[1763] Step 6:

[1764] The terminal (robot) executes the generated gestures. The input is a motion sequence, and the output is the actual robot motion. Specific motions are controlled by the terminal, which controls motors and actuators to perform actions such as waving or smiling.

[1765] Step 7:

[1766] The user (elderly person) watches the robot's movements and continues the dialogue. The input is the robot's movements and dialogue content, and the output is the elderly person's new utterances and reactions. In terms of specific actions, the elderly person expresses their emotions in response to the robot's movements and continues the next dialogue.

[1767] (Application example 1)

[1768] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1769] When elderly people visit a physical store, it can be difficult for them to receive guidance within the store or product explanations. In particular, in situations where a response that corresponds to their emotional state is required, conventional guidance systems often do not respond adequately. Furthermore, in order for elderly people to use stores with peace of mind, it is important to respond in a friendly and empathetic manner. To solve these issues, a system is needed that can recognize the emotional state of the elderly and provide appropriate gestures and dialogue content.

[1770] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1771] In this invention, the server includes a robot means for interacting with the elderly, an emotion recognition function means, a function means for generating gestures, a generative AI means for generating natural conversations, a function means for empathizing with what the elderly is saying and asking questions, a function means for recording memories and wills, a function means for sorting belongings and preparing for a funeral, a function means for providing information on end-of-life planning such as consultations on inheritance and taxes, and a function means that is installed in a physical store and provides store guidance and product explanations while using gestures according to the emotional state of the elderly. This enables elderly people to receive friendly service and appropriate guidance according to their emotional state when they visit a physical store.

[1772] A "robot that interacts with the elderly" is an automated mechanical device that can have natural conversations with the elderly.

[1773] "Emotion recognition function" is a technology that identifies the emotional state of an elderly person from their facial expressions, tone of voice, etc.

[1774] The "function to generate gestures and movements" is a technology that generates appropriate gestures and movements according to the recognized emotional state.

[1775] "Generative AI that generates natural conversations" is an artificial intelligence technology that generates natural conversations when interacting with the elderly.

[1776] The "function of empathizing with and asking questions about what the elderly are saying" refers to the technology of showing empathy for what the elderly are saying and asking appropriate questions.

[1777] "Recording memories and wills" is a function that records the memories and wills of elderly people.

[1778] "Sorting out belongings and preparing for funerals" is a function that assists elderly people in sorting out their belongings and preparing for funerals.

[1779] The "function to provide information on end-of-life planning, such as consultations on inheritance and taxes" is a function that provides information for elderly people to consult on inheritance and taxes related to end-of-life planning.

[1780] "A function that is installed in a physical store and provides store guidance and product explanations while using gestures that correspond to the emotional state of the elderly" refers to technology that is installed in a physical store and provides store guidance and product explanations while using gestures that correspond to the emotional state of the elderly.

[1781] The system for implementing this invention is centered around a robot that interacts with elderly people. The system includes the following main hardware and software:

[1782] 1. Hardware:

[1783] Camera: Used to capture the facial expressions of the elderly.

[1784] Robot body: A mechanical device for interacting and gesturing.

[1785] Microphone: Used to capture the voice of the elderly person.

[1786] Speaker: Used to output the generated audio.

[1787] 2. Software:

[1788] A library for capturing and processing camera footage.

[1789] A deep learning framework to load emotion recognition models and make predictions.

[1790] A library used as a speech synthesis engine to conduct voice-based dialogue with the elderly.

[1791] The system works as follows:

[1792] 1. Emotion recognition:

[1793] The camera captures the elderly person's facial expressions and converts them into grayscale.

[1794] The image is resized to 48x48 pixels and fed into the emotion recognition model.

[1795] Obtain emotion labels from the model's prediction results.

[1796] 2. Gesture generation:

[1797] Depending on the recognized emotional state, appropriate gestures are generated.

[1798] For example, if the emotion of joy is recognized, the robot will make gestures that express joy.

[1799] 3. Voice interaction:

[1800] Generative AI is used to generate dialogue content that corresponds to the emotional state of the elderly person.

[1801] The generated dialogue content is output as voice.

[1802] To illustrate, consider the following scenario:

[1803] When an elderly person enters the store, the robot will say aloud, "Hello! I'll make a gesture of joy. Is there anything I can help you with?"

[1804] If an elderly person looks at a product with joy, the robot will explain, using happy gestures, "That product is very popular!"

[1805] Examples of prompts to input to a generative AI model include:

[1806] "Generate gestures that an elderly person would make if they were happy."

[1807] "Generate gestures for when an elderly person is surprised."

[1808] This means that when elderly people visit a physical store, they will be able to receive friendly service and appropriate guidance that is tailored to their emotional state.

[1809] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1810] Step 1:

[1811] The camera captures the facial expressions of the elderly.

[1812] Input: Camera image

[1813] Output: Captured image frames

[1814] Specific operation: The camera captures the elderly person's face in real time and acquires video data.

[1815] Step 2:

[1816] The server converts the captured image frames to grayscale.

[1817] Input: Captured image frames

[1818] Output: Grayscale image

[1819] Specific operation: Converts a color image to a grayscale image.

[1820] Step 3:

[1821] The server resizes the grayscale image to 48x48 pixels.

[1822] Input: Grayscale image

[1823] Output: Resized image

[1824] Specific behavior: Resizes a grayscale image to 48x48 pixels.

[1825] Step 4:

[1826] The server inputs the resized image into the emotion recognition model.

[1827] Input: Resized image

[1828] Output: Emotion label

[1829] Specific operation: The resized image is input into the emotion recognition model to predict the emotion label.

[1830] Step 5:

[1831] The server generates appropriate gestures based on the emotion labels.

[1832] Input: Emotion label

[1833] Output: Gesture instructions

[1834] Specific actions: Generate gestures corresponding to the emotion labels and send instructions to the robot.

[1835] Step 6:

[1836] The robot executes the generated gesture.

[1837] Input: Gesture instructions

[1838] Output: Gesture movement

[1839] Specific behavior: The robot follows instructions and performs appropriate gestures.

[1840] Step 7:

[1841] The server generates dialogue content according to the emotional state of the elderly person.

[1842] Input: Emotion label

[1843] Output: Dialogue content

[1844] Specific operation: Using generative AI, natural dialogue content is generated according to the emotional state of the elderly person.

[1845] Step 8:

[1846] The server inputs the generated dialogue content into a speech synthesis engine.

[1847] Input: Dialogue

[1848] Output: Audio data

[1849] Specific operation: Convert the generated dialogue content into voice data.

[1850] Step 9:

[1851] The robot plays the audio data.

[1852] Input: Audio data

[1853] Output: Audio output

[1854] Specific operation: The generated voice data is played back through the robot's speaker, and the conversation content is conveyed to the elderly.

[1855] Example 2

[1856] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1857] When communicating with the elderly, it is necessary to effectively recognize emotions, respond with empathy, and provide information about end-of-life planning. However, conventional systems have difficulty providing these functions in an integrated manner, and there is a lack of dialogue systems that the elderly can use with confidence. Furthermore, technology for generating appropriate responses to the elderly's input has not been sufficiently developed. This makes it difficult to alleviate the loneliness and anxiety felt by the elderly.

[1858] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a device for interacting with the elderly, emotion recognition means, gesture generation means, generative AI means for generating natural conversation, means for empathizing with what the elderly is saying and asking questions, means for recording memories and wills, means for providing information on end-of-life planning such as sorting belongings, funeral preparations, and inheritance and tax consultations, means for receiving text data input by a user, generating and inputting prompt sentences to the generative AI model, and means for displaying the response generated by the generative AI model to the user. This makes it possible to realize natural conversation with the elderly and to provide emotion recognition, empathetic responses, and information on end-of-life planning in an integrated manner.

[1859] A "device for interacting with the elderly" is a device that includes hardware and software that allows the elderly to interact with the elderly through voice and text.

[1860] "Emotion recognition means" is a technology for analyzing and recognizing the emotional state of elderly people from their voices, facial expressions, and text.

[1861] "Means for generating gestures" refers to technology that enables robots and avatars to generate natural gestures when interacting with elderly people.

[1862] "Generative AI means for generating natural conversations" refers to artificial intelligence technology for generating natural and fluent conversations in dialogue with the elderly.

[1863] "Means for empathizing with and asking questions about what the elderly say" refers to technology for generating empathetic responses and appropriate questions in response to what the elderly say.

[1864] "Mechanisms for recording memories and wills" is a technology for digitally recording the memories and wills spoken by elderly people.

[1865] "Means of providing information on end-of-life planning, such as sorting out belongings, preparing for funerals, and consulting on inheritance and taxes" refers to technology that allows elderly people to obtain various information on end-of-life planning.

[1866] "Means for receiving text data entered by a user and generating and inputting a prompt sentence to a generative AI model" refers to a technology for receiving text data entered by a user and generating a prompt sentence to appropriately input the text data to a generative AI model.

[1867] "Means for displaying to the user the response generated by the generative AI model" refers to technology for visually or audibly displaying to the user the response generated by the generative AI model.

[1868] MODE FOR CARRYING OUT THE INVENTION

[1869] This invention is a system for realizing natural dialogue with elderly people, and integrates emotion recognition, generation of empathetic responses, provision of information on end-of-life planning, etc. Specific embodiments of this system are described below.

[1870] System configuration

[1871] The system consists of the following main components:

[1872] 1. Devices for interacting with the elderly: These are devices that include hardware and software that allow elderly people to interact with the elderly through voice and text. Examples include smartphones, tablets, and dedicated conversation robots.

[1873] 2. Emotion recognition: This technology analyzes and recognizes the emotional state of elderly people from their voices, facial expressions, and text. For example, it uses voice analysis software and facial expression recognition cameras.

[1874] 3. Means of generating gestures: This technology allows robots and avatars to generate natural gestures when interacting with the elderly. For example, motion capture technology is used.

[1875] 4. Generative AI means for generating natural conversations: This is an artificial intelligence technology for generating natural and fluent conversations with the elderly. For example, OpenAI's GPT-4 is used as a generative AI model.

[1876] 5. Means of empathizing with and asking questions about what the elderly say: Techniques for generating empathetic responses and appropriate questions in response to what the elderly say.

[1877] 6. Recording of memories and wills: Technologies for digitally recording the memories and wills spoken by older people, for example, using cloud storage services.

[1878] 7. Means of providing information on end-of-life planning: This is technology for providing information on end-of-life planning, such as sorting out belongings, preparing for a funeral, and consulting on inheritance and taxes.

[1879] 8. A means for receiving text data entered by a user and generating and inputting prompt sentences to a generative AI model: A technology for receiving text data entered by a user and generating prompt sentences to appropriately input it into a generative AI model.

[1880] 9. Means for displaying responses generated by a generative AI model to the user: A technology for visually or audibly displaying responses generated by a generative AI model to the user.

[1881] System Operation

[1882] The user (elderly person) uses the dialogue device to input text data. For example, the user might input, "I miss the memories of traveling when I was young." The device then sends this text data to the server. The server analyzes the received text data and generates a prompt sentence to input into the generative AI model. For example, it generates a prompt sentence such as, "The user is talking about memories of traveling when they were young. What kind of response would you generate?"

[1883] A generative AI model (e.g., GPT-4) generates an appropriate response based on this prompt, such as "Which place is most memorable?" The server sends the generated response to the device, which then displays it to the user.

[1884] Specific examples

[1885] As a concrete example, consider an elderly person talking about past memories. If the user inputs, "I have fond memories of the trips I took when I was younger," the generative AI will generate an empathetic response such as, "Which place left the most lasting impression on you?" Similarly, if an elderly person expresses a question about end-of-life planning, for example, by inputting, "What should I do to start preparing for the end of my life?", the generative AI will generate an answer such as, "It would be a good idea to start by writing a will and organizing your assets."

[1886] Example prompt sentence:

[1887] "I have fond memories of traveling when I was young."

[1888] "How do I start planning my end?"

[1889] In this way, the system can realize natural conversations with the elderly, and can comprehensively recognize emotions, respond with empathy, and provide information on end-of-life planning.

[1890] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1891] Step 1:

[1892] The user inputs text data using the terminal. For example, the user inputs "I miss the memories of traveling when I was young." The input text data is stored in the memory of the terminal.

[1893] Step 2:

[1894] The terminal sends the input text data to the server. Specifically, the terminal converts the text data into JSON format and sends it to the server using an HTTP POST request. The input is the user's text data, and the output is the data to be sent to the server.

[1895] Step 3:

[1896] The server receives text data sent from the terminal. The received data is stored in the server's memory. The input is the text data from the terminal, and the output is the data stored in the server's memory.

[1897] Step 4:

[1898] The server analyzes the received text data and generates a prompt to be input to the generative AI model. For example, if the received data is "I miss the memories of traveling when I was young," the prompt would be "The user is talking about their memories of traveling when they were young. What response would you like to generate?" The input is the text data stored on the server, and the output is the generated prompt.

[1899] Step 5:

[1900] The server inputs a prompt to the generative AI model. The generative AI model (e.g., GPT-4) generates an appropriate response based on the prompt. For example, it generates the response, "Which place is most memorable?" The input is the generated prompt, and the output is the response generated by the generative AI model.

[1901] Step 6:

[1902] The server sends the generated response to the terminal. Specifically, it converts the generated response into JSON format and sends it to the terminal as an HTTP response. The input is the response generated by the generative AI model, and the output is the data sent to the terminal.

[1903] Step 7:

[1904] The terminal receives the response data sent from the server and displays it to the user. For example, the terminal displays the message "Which place is the most memorable?" on the screen. The input is the response data from the server, and the output is the message displayed to the user.

[1905] In this way, the system can realize natural conversations with the elderly, and can comprehensively recognize emotions, respond with empathy, and provide information on end-of-life planning.

[1906] (Application example 2)

[1907] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1908] When elderly people spend time alone at home, it is necessary to provide an environment where they can live with peace of mind, reducing feelings of loneliness and anxiety. Also, a system that can respond quickly in emergencies is necessary. Furthermore, it is important to make it easy for elderly people to obtain information about end-of-life planning.

[1909] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a device means for interacting with the elderly, an emotion recognition means, a means for generating gestures and movements, a generative AI means for generating natural conversations, a means for empathizing with the elderly's story and asking questions, a means for recording memories and wills, a means for sorting belongings and preparing for a funeral, a means for providing information on end-of-life planning such as inheritance and tax consultations, and a means for automatically sending notifications in the event of an emergency. This provides an environment where the elderly can live in peace and enables rapid response in the event of an emergency. It also makes it easy to obtain information on end-of-life planning.

[1910] A "device for interacting with the elderly" is a device that includes hardware and software for conducting natural conversations with the elderly.

[1911] "Emotion recognition means" is a technology for analyzing and recognizing the emotions of elderly people from their facial expressions and tone of voice.

[1912] The "means for generating gestures and movements" is a technology that generates natural gestures and movements during conversations to facilitate communication with the elderly.

[1913] "Generative AI means for generating natural conversations" is an artificial intelligence technology for generating natural responses in dialogue with elderly people.

[1914] "Means for empathizing with and asking questions about what the elderly are saying" is a technology for generating empathetic responses and appropriate questions to what the elderly are saying.

[1915] "Means for recording memories and wills" is a technology for recording the memories and wills of elderly people in digital format.

[1916] "Tools for sorting out belongings and preparing for funerals" refers to technology that provides information and support for elderly people regarding sorting out belongings and preparing for funerals.

[1917] "Means of providing information on end-of-life planning, such as consultations on inheritance and taxes" refers to technology that provides information to help elderly people resolve questions about inheritance and taxes.

[1918] "Means for automatically sending notifications in emergencies" refers to technology that automatically sends notifications to family members and emergency contacts when an elderly person experiences an emergency.

[1919] This invention is a system for providing an environment where elderly people can live safely at home. The system includes a device for interacting with elderly people, emotion recognition means, gesture generation means, generative AI means for generating natural conversations, means for empathizing with elderly people's stories and asking questions, means for recording memories and wills, means for sorting belongings and preparing funerals, means for providing information on end-of-life planning such as consultations on inheritance and taxes, and means for automatically sending notifications in the event of an emergency.

[1920] System Program

[1921] The system program is configured as follows:

[1922] 1. Devices for interacting with the elderly:

[1923] It receives text and voice input from the elderly person and sends it to a generative AI.

[1924] The response from the generative AI is returned to the elderly person.

[1925] 2. Emotion recognition means:

[1926] It analyzes the facial expressions and tone of voice of elderly people to recognize their emotional state.

[1927] The recognition results are fed back to the generative AI to adjust the content of the dialogue.

[1928] 3. Means of generating gestures:

[1929] It generates natural gestures and hand movements during conversations with elderly people, facilitating communication.

[1930] 4. Generative AI methods for generating natural conversations:

[1931] Generate natural responses based on the elderly person's input.

[1932] Empathize with what the elderly are saying and generate appropriate questions.

[1933] 5. Means of recording memories and wills:

[1934] Record the memories and wills of the elderly in digital form.

[1935] 6. Arranging belongings and funeral arrangements:

[1936] Provides information and support for elderly people sorting out their belongings and preparing for funerals.

[1937] 7. Means of providing information on end-of-life planning, such as inheritance and tax consultations:

[1938] Provides information to help seniors resolve questions about inheritance and taxes.

[1939] 8. Means of sending automatic notifications in case of emergency:

[1940] When an elderly person has an emergency, notifications are automatically sent to family members and emergency contacts.

[1941] Hardware and software used

[1942] Hardware: Devices such as smartphones, tablets, and computers.

[1943] Software: Python, OpenAI API.

[1944] Processing flow

[1945] 1. The elderly person talks to the system using a smartphone or tablet.

[1946] 2. The system receives the elderly person's input and analyzes their emotional state using emotion recognition means.

[1947] 3. The analysis results are fed back to the generative AI to generate natural conversations.

[1948] 4. Return the generated response to the elderly person and generate gestures as needed.

[1949] 5. If an elderly person has questions about end-of-life planning, provide them with relevant information.

[1950] 6. Automatically notify family members and emergency contacts in case of an emergency.

[1951] Specific examples

[1952] Prompt Sentence Examples

[1953] Elderly person: I went on a trip with my family a long time ago, and I have fond memories of that time.

[1954] Example of the response generated

[1955] AI: That's a wonderful memory! Where did you travel to? What was...

Claims

1. A device for interacting with elderly people, a first means for identifying an emotional state of the senior citizen from the senior citizen's voice using an emotion recognition engine; A second means for receiving voice and text data input by the senior citizen; a third means for generating a first prompt sentence based on the emotional state of the elderly person and the input content, for outputting a response appropriate to the emotional state and the input content; a fourth means for inputting the generated first prompt sentence into a generative AI model to generate the response; a fifth means for providing the generated response to the senior; A system including:

2. and a sixth means for causing the device to perform a gesture associated with the generated response. The system of claim 1 .

3. The third means generates a second prompt sentence for outputting information related to end-of-life planning, including sorting of belongings, funeral preparations, inheritance, and tax consultations, based on the input content of the elderly person; The fourth means inputs the generated second prompt sentence into a generation AI model to generate information related to the end-of-life planning.

3. A system according to claim 1 or 2.

Images