System

Abstract

To provide a system for enabling an aged person to obtain mental stability by stimulating memory through interaction.SOLUTION: The system includes means for collecting personal-related data provided by a user, means for preprocessing and denoising and normalizing the collected data, means for training a generative AI model based on the preprocessed data, means for optimizing the trained model for the elderly, means for providing an interface for realizing interaction between the elderly and the generative AI model, means for recognizing a voice input of the elderly and analyzing the input to generate a response, means for reproducing the generated response to the elderly, and means for collecting feedback from the elderly and re-training the model.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Dementia in the elderly is a serious problem in modern society, where the birthrate is declining and the population is aging. In particular, declines in memory and communication skills not only significantly reduce the quality of life of elderly people, but also place a great burden on families who require care. Conventional measures rely on face-to-face caregiving, medical care, and rehabilitation, which require a lack of time and human resources. To solve this problem, effective dementia treatment methods using technology that can be easily used by elderly people at home are needed. [Means for solving the problem]

[0005] The present invention provides a system that combines an advanced generative AI model with virtual reality (VR) technology to prevent and improve dementia in the elderly. Specifically, the system includes a means for collecting personal data provided by users, a means for preprocessing the collected data, denoising and normalizing it, and a means for training a generative AI model based on the preprocessed data. It also includes a means for optimizing the trained model for the elderly and providing an interface for enabling dialogue between the elderly and the generative AI model. The system recognizes the elderly's voice input, analyzes the input, generates a response, and plays the generated response back to the elderly. Feedback from the elderly is collected and used to retrain the model, continuously improving the system's accuracy. This allows the elderly to stimulate their memory and achieve mental stability through dialogue.

[0006] "Personal data" refers to information provided by a user that relates to a specific individual, such as the tone of voice of family members, background, preferences, things they don't like, and dialects.

[0007] "Preprocessing" refers to processing the collected data, such as by removing noise and normalizing it, to prepare it in a form that is easier to analyze.

[0008] A "generative AI model" is an algorithm that uses machine learning and deep learning techniques to learn from data and automatically generate responses and decisions.

[0009] "Optimizing for the elderly" means adjusting the parameters of the generative AI model to suit the cognitive level, preferences, and voice characteristics of the elderly, so that it can provide responses that are appropriate for the elderly.

[0010] "Interface" refers to the user interface that allows the elderly person to interact with the generative AI model, and includes visual and audio elements.

[0011] "Recognizing voice input" means capturing the voice spoken by the elderly person as a digital signal and analyzing its content.

[0012] "Generating a response" refers to the process by which a generative AI model generates an appropriate reply based on the analyzed voice input.

[0013] "Playing the response to the senior" refers to using an audio output device to play the generated audio response to the senior.

[0014] "Gathering feedback" means recording the reactions and opinions of the elderly and using them to improve the system.

[0015] "Model retraining" refers to retraining a generative AI model based on collected feedback to improve its accuracy and performance. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention provides a system that combines a generative AI model with virtual reality (VR) technology to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing data related to individuals, and training and optimizing the generative AI model. It also provides an interface for realizing dialogue between the elderly and the generative AI model, recognizing the elderly's voice input, generating and playing back a response, and collecting feedback from the elderly for use in retraining the model.

[0038] Overall system picture

[0039] First, users (family members or friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data can include family memories, anecdotes about specific events, and everyday conversations. The server receives the provided data and performs preprocessing. This preprocessing includes noise reduction for the voice data and normalization for the text data.

[0040] The server then uses the preprocessed data to train a generative AI model, which is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the generative AI model to provide natural and friendly responses to the elderly.

[0041] Building the interface

[0042] The terminals (devices used by the elderly) will have user interfaces tailored to the elderly, providing large buttons and text that are visually and auditorily friendly, as well as voice-guided navigation.

[0043] When an elderly person initiates a dialogue with the generative AI model through the device, the device records the elderly person's voice input in real time and sends it to the server, which analyzes the voice data and generates an appropriate response, which is then played back by the device.

[0044] Gathering feedback and retraining the model

[0045] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions for the elderly.

[0046] Specific examples

[0047] Below is a concrete example of how the system actually works.

[0048] When an older person talks about a family trip they took long ago

[0049] Users (family and friends) upload audio and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When an elderly person speaks to the device, saying, "Tell me about your past trips," the device recognizes this and sends the audio data to the server.

[0050] The server analyzes the voice data and generates a response about a family trip. The generated response is then played back by the device as, "On a previous trip, we all went to a hot spring together as a family." The device then records the elderly person's response and sends feedback to the server. Based on this, the model is retrained to provide a more appropriate response in the next interaction.

[0051] In this way, elderly people can easily use the system at home, helping to prevent and improve dementia. The system provides new stimulation and joy to the lives of elderly people, supports their mental health, and contributes to reducing feelings of loneliness.

[0052] The processing flow will be explained below.

[0053] Step 1:

[0054] Users (family and friends) use a web portal or dedicated app to upload audio and text data related to the elderly, including family memories, anecdotes about specific events, and everyday conversations.

[0055] Step 2:

[0056] The server preprocesses the data it receives. Specifically, it performs noise removal for audio data, normalizes text data, and removes unnecessary parts.

[0057] Step 3:

[0058] The server stores the preprocessed data in a database, which is then used to train the generative AI model.

[0059] Step 4:

[0060] The server uses the stored data to train the generative AI model, extracting speech features from the voice data and using natural language processing to extract important keywords from the text data.

[0061] Step 5:

[0062] The server optimizes the trained generative AI model for seniors, adjusting the model parameters based on the seniors' cognitive level, voice characteristics, and individual preferences.

[0063] Step 6:

[0064] The device will create a user interface tailored to the elderly, providing large buttons and text that are visually and auditorily friendly, as well as voice-guided navigation.

[0065] Step 7:

[0066] The user (elderly person) initiates a dialogue with the generative AI model through the device. When the elderly person speaks to the device, the device records it in real time and sends it to the server.

[0067] Step 8:

[0068] The server analyzes the voice data sent and generates an appropriate response using a generative AI model.

[0069] Step 9:

[0070] The server sends the generated response to the terminal, which converts the response into speech and plays it back to the elderly person.

[0071] Step 10:

[0072] The device records the elderly person's reactions and feedback in real time and transmits them to a server.

[0073] Step 11:

[0074] The server uses the feedback collected to retrain the generative AI model and further optimize it for older people, improving the accuracy and performance of the system.

[0075] Through this process, seniors can enjoy interacting with generative AI models, stimulating their cognitive functions and promoting their mental health.

[0076] Example 1

[0077] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0078] Dialogue systems aimed at preventing and improving dementia in the elderly require mechanisms that are friendly to the elderly and provide appropriate responses. Conventional dialogue systems have difficulty generating appropriate responses based on the cognitive level and preferences of each individual elderly person, making it difficult to achieve natural dialogue in real time in response to elderly voice input. Furthermore, there have been insufficient methods for appropriately collecting feedback from the elderly and using it to retrain models. This has resulted in technical challenges for smoothly introducing dialogue systems into the daily lives of the elderly.

[0079] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0080] In this invention, the server includes a means for collecting personal information provided by the user, a means for preprocessing the collected information, removing unnecessary data, and standardizing the format, and a means for training a generative artificial intelligence model based on the preprocessed information. This makes it possible to provide natural and friendly dialogue responses to each elderly person. Furthermore, the system recognizes the elderly person's voice input, provides an interface for analyzing the input and generating an appropriate response, and outputs the generated response to the elderly person, thereby achieving natural dialogue in real time. Furthermore, responses from the elderly are collected and used to retrain the generative artificial intelligence model, thereby continuously improving the accuracy of the model. In this way, it is possible to provide an effective dialogue system that contributes to the prevention and improvement of dementia in the elderly.

[0081] A "user" is a person or group that provides information related to a particular individual.

[0082] "Information" is a general term for data related to an individual, such as the tone of voice of family members, background, interests, prohibited activities, and local dialects.

[0083] A "collection method" is a method or device used to collect information from a user.

[0084] "Preprocessing" refers to processing carried out to remove unnecessary data from collected information and standardize the format.

[0085] A "generative artificial intelligence model" is a machine learning model that is trained based on collected information and aims to generate natural-sounding dialogue responses.

[0086] A "training tool" refers to a method or technique for optimizing a generative artificial intelligence model for a specific purpose.

[0087] "Elderly optimization means" refers to methods and techniques for adjusting generative artificial intelligence models to provide friendly and natural dialogue responses to seniors.

[0088] "Connection means" refers to interfaces or devices that enable interaction between the elderly person and the generative artificial intelligence model.

[0089] "Voice input" refers to the voice or spoken words produced by the elderly person.

[0090] A "means for generating a response" is a method or technique for analyzing speech input and creating an appropriate reply.

[0091] "Output means" refers to a method or device for communicating the generated response to the elderly person.

[0092] Response collection tools are methods and techniques used to gather responses and feedback from older adults.

[0093] This invention provides a system that combines a generative artificial intelligence model with virtual reality (VR) technology to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing information related to an individual, and training and optimizing the generative artificial intelligence model. Furthermore, the system provides an interface for realizing dialogue between the elderly and the generative artificial intelligence model, recognizes the elderly's voice input, generates and plays back a response, and collects responses from the elderly for use in retraining the model.

[0094] Data collection methods

[0095] Users (elderly family members and friends) provide personal information, such as their family members' voices, backgrounds, interests, prohibited activities, and local dialects, through a dedicated web portal or application. This information includes memories of family trips and everyday conversations. Specifically, users open the application and upload images, audio files, and text. For example, they can upload photos and descriptions of past family trips.

[0096] Data preprocessing methods

[0097] The server receives the provided information and performs preprocessing, which includes denoising the audio data and standardising the format of the text data. Specifically, the server uses the Python library Librosa to denoise the audio data and the NLTK library to correct spelling and check grammar of the text data.

[0098] A means of training and optimizing generative artificial intelligence models

[0099] The server uses the preprocessed information to train a generative artificial intelligence model. Machine learning libraries such as TENSORFLOW® and PyTorch are used for training. The trained model is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences. This enables the model to provide natural and friendly responses to the elderly.

[0100] Building the user interface

[0101] Terminals (devices used by the elderly) will have user interfaces tailored to the elderly. These interfaces will include large buttons and text that take visual and hearing impairments into consideration, as well as voice-guided navigation. Specifically, terminals will be designed to take visual impairments into consideration, displaying buttons and text in large fonts and with clear contrast. They will also provide voice assistant functionality to make them easy for the elderly to operate.

[0102] Real-time interaction means

[0103] When an elderly person begins a dialogue with the generative AI model through the device, the device records the elderly person's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. For example, if an elderly person says, "Tell me about your old trips," the device records this and sends the voice data to the server. The server analyzes this voice data and "generates a response about family trips." The generated response is played back by the device as, "On your old trips, we all went to hot springs together as a family."

[0104] A means of gathering feedback and retraining the model

[0105] During the interaction, the device collects the elderly's reactions and feedback and sends it to the server. Based on this feedback, the server retrains the generative AI model. Specifically, the device asks whether the elderly is satisfied with the response and sends the result to the server. The server uses this feedback to update the model and provide more appropriate responses in the next interaction.

[0106] Specific examples

[0107] Below is a concrete example of how the system actually works.

[0108] When an older person talks about a family trip they took long ago

[0109] Users (family and friends) upload voice and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When the elderly person speaks to the device, saying, "Tell me about your old trips," the device recognizes this and sends the voice data to the server. The server analyzes the voice data and "generates a response about your family trip." The generated response is then played back by the device as, "On your old trips, we all went to a hot spring." The device then records the elderly person's response and sends feedback to the server. Based on this, the model is retrained to provide a more appropriate response in the next interaction.

[0110] Prompt Sentence Examples

[0111] "How are you feeling today?"

[0112] "Want to talk about past family trips?"

[0113] "Tell me some memories you have with your grandchildren."

[0114] In this way, the system can activate memories and conversational content of the elderly, helping to prevent and improve dementia. The system provides new stimulation and joy to the lives of the elderly, supporting their mental health and contributing to reducing loneliness.

[0115] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0116] Step 1:

[0117] Data collection

[0118] Users use a dedicated web portal or application to provide information about their family members, such as their voices, backgrounds, interests, prohibited activities, and local dialects, as well as upload photos, audio files, and text data.

[0119] Input: Audio data, text data, image data

[0120] Output: Collected information (stored in a database)

[0121] What happens: A user opens the application and uploads photos and descriptions of past family trips.

[0122] Step 2:

[0123] Data Preprocessing

[0124] The server preprocesses the collected information: first, it uses the Python library Librosa to denoise the audio data, and then it uses the NLTK library to standardize the format of the text data (for example, to perform spelling correction and grammar checks).

[0125] Input: Collected information (output of step 1)

[0126] Output: Preprocessed information (denoised audio data, normalized text data)

[0127] Specific operation: The server receives the audio data and uses Librosa to remove noise. It also processes the text data with NLTK and standardizes the format.

[0128] Step 3:

[0129] Training and optimizing generative AI models

[0130] The server trains a generative artificial intelligence model based on the preprocessed information using machine learning libraries such as TensorFlow and PyTorch, optimizing the model based on the elderly person's cognitive level, voice characteristics, and individual preferences.

[0131] Input: Preprocessed information (output of step 2)

[0132] Output: A trained generative artificial intelligence model

[0133] Specific operation: The server uses preprocessed voice and text data to train a generative artificial intelligence model and optimize it for the elderly.

[0134] Step 4:

[0135] Building the user interface

[0136] The device will have a user interface tailored to seniors, including large buttons and text for visual and hearing impairments, as well as voice-guided navigation.

[0137] Input: Design specifications, needs of the elderly

[0138] Output: User Interface (UI)

[0139] What it does: Your device displays buttons and text in large fonts and with clear contrast, and offers voice assistant functionality.

[0140] Step 5:

[0141] Real-time dialogue

[0142] When an elderly person initiates a dialogue with the generative AI model through the device, the device records the elderly person's voice input in real time and transmits it to the server, which analyzes the voice input and generates an appropriate response.

[0143] Input: Voice input by elderly people

[0144] Output: The generated response

[0145] Specific operation: When an elderly person says, "Tell me about your past travels," the device records this and sends the audio data to the server. The server analyzes the audio data, "generates a response about family trips," and sends it back to the device. The device then plays back the generated response.

[0146] Step 6:

[0147] Gathering feedback and retraining the model

[0148] During the interaction, the device collects the elderly's reactions and feedback and sends it to the server, which uses this feedback to retrain the generative artificial intelligence model.

[0149] Input: Older adults' reactions and feedback

[0150] Output: A retrained generative AI model

[0151] How it works: The device checks whether the elderly person is satisfied with the response and sends the result to the server. The server uses the feedback to update the model and provide a more appropriate response in the next interaction.

[0152] (Application example 1)

[0153] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0154] There is a need to provide an environment where elderly people can easily select and order meals while staying safe at home. It is also necessary to introduce dialogue systems that contribute to the prevention and improvement of dementia. However, existing systems have not been able to adequately respond to individual needs by taking into account the preferences and history of elderly people. Furthermore, in order for dialogue systems to generate natural and friendly responses to elderly voice input, advanced processing in speech recognition technology and generative AI models is required, posing a challenge to practical implementation.

[0155] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0156] In this invention, the server includes: means for collecting personal data provided by a user; means for preprocessing the collected data to remove noise and normalize it; means for training a generative AI model based on the preprocessed data; means for optimizing the trained model for the elderly; means for providing an interface for enabling dialogue between the elderly and the generative AI model; means for recognizing the elderly's voice input, analyzing the input, and generating a response; means for playing the generated response to the elderly; means for collecting feedback from the elderly and using it to retrain the model; means for presenting a menu based on the elderly's past eating history and preferences; and means for ordering meals based on the presented menu. This enables personalized meal suggestions and ordering based on the elderly's own preferences and history, and also provides an interactive system that is effective in preventing and improving dementia.

[0157] "Personal data provided by the user" refers to information related to the elderly person provided by the user, including the family member's tone of voice, upbringing, preferences, things they don't like, dialect, etc.

[0158] "Collected data" refers to audio data, text data, and the like that are uniquely associated with the senior, including information provided by the user.

[0159] "Preprocessing" refers to processing performed on collected data, such as noise removal and data normalization.

[0160] A "generative AI model" is an artificial intelligence model that has the ability to learn from large datasets and respond to specific tasks.

[0161] "Methods for optimizing for the elderly" refer to techniques for adjusting generative AI models based on the cognitive level, voice characteristics, and individual preferences of the elderly.

[0162] The "interface" is a means of providing an operating screen and voice guidance for elderly people to interact with the generated AI model.

[0163] "Means for recognizing a senior's voice input and analyzing the input to generate a response" refers to the overall process of analyzing a senior's voice and generating an appropriate response.

[0164] "Means for playing the generated response to the elderly" refers to a method for letting the elderly hear the voice response generated by the generative AI model.

[0165] "Means of collecting feedback from seniors and using it to retrain the model" refers to the process of improving the performance of generative AI models based on reactions and opinions obtained from interactions with seniors.

[0166] "Means for presenting menus based on past eating history and preferences" refers to a function that presents appropriate menus based on the elderly person's past eating history and preferences.

[0167] "Means for ordering food based on a presented menu" refers to the functionality of making selections from a presented menu and actually completing a food order.

[0168] This invention aims to prevent and improve dementia in the elderly using the following systems and processes.

[0169] Overall system picture

[0170] First, users (family and friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data includes past dietary history and preferences, the family member's voice tone, upbringing, things they don't like, dialects, etc. The server receives this data and performs noise reduction and data normalization.

[0171] The server then uses the preprocessed data to train a generative AI model, optimizing it based on the elderly person's cognitive level, voice characteristics, and individual preferences, allowing the trained generative AI model to generate natural and friendly responses in real time.

[0172] Building the interface

[0173] The terminal (the device used by the elderly) provides a user interface tailored to the elderly. This interface features large buttons, text, and voice-guided navigation that take visual and auditory impairments into consideration. When the elderly initiates a dialogue with the generative AI model through the terminal, the terminal records the elderly's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. This response is then played back by the terminal.

[0174] Food ordering function

[0175] A distinctive feature of this system is its ability to present menus based on the elderly person's past eating history and preferences. When the elderly person asks, "What's on the menu today?", the device recognizes this and sends the voice data to the server. The server then generates an optimal response based on this and suggests dishes suitable for the elderly. In this way, the application assists the elderly in the process of selecting and ordering meals.

[0176] Gathering feedback and retraining the model

[0177] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions and services for the elderly.

[0178] Hardware and software used

[0179] The system uses the following main hardware and software:

[0180] Smartphones / head-mounted displays: Devices used by the elderly.

[0181] TensorFlow / Keras: Used for training and inference of generative AI models.

[0182] SpeechRecognition: Provides speech recognition technology.

[0183] pyttsx3: Used to speak the generated response.

[0184] Adding specific examples

[0185] For example, if a user (an elderly person) says, "What's today's recommended menu?", the system will generate the following response, taking into account the elderly person's eating history and preferences:

[0186] "Today, I recommend the grilled eel and soup set. Would you like to order that?"

[0187] Prompt Sentence Examples

[0188] The prompt should be in the following format:

[0189] Senior citizen: "What's on the menu today?"

[0190] Generative AI: "Based on your past eating history and preferences, the recommended menu is a set of grilled eel and clear soup. Would you like to order that?"

[0191] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0192] Step 1:

[0193] Users (family and friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This includes past meal history, the family member's voice tone, upbringing, preferences, things they don't like, dialects, etc. This input data is sent to the server. The main purpose of this step is to collect the input data (voice and text data). The output is the initial data before it is preprocessed.

[0194] Step 2:

[0195] The server preprocesses the received data related to individuals, specifically by removing noise and normalizing the data. This process uses the acquired voice and text data as input and processes it through noise filtering, tokenization, and other processes. This results in clean data that the generative AI model can accurately learn from.

[0196] Step 3:

[0197] The server trains a generative AI model based on the preprocessed data. The server uses deep learning libraries such as TensorFlow and Keras to generate a model with voice response capabilities based on the input data. The output of this step is a highly specialized generative AI model.

[0198] Step 4:

[0199] The trained model is optimized for seniors. The server adjusts the model parameters based on the seniors' cognitive level, voice characteristics, and individual preferences. In this process, the generative AI model obtained in the previous step is further tuned so that it can generate responses appropriate for seniors. The output is a generative AI model optimized for seniors.

[0200] Step 5:

[0201] The device provides an interface for dialogue between the elderly person and the generative AI model. It displays an interface that takes visual and auditory impairments into consideration, including large buttons, text, and audio guides. The input is the elderly person's operations, and the output is a GUI (graphical user interface) for dialogue.

[0202] Step 6:

[0203] The elderly person inputs voice through the device. This voice input is recorded in real time and sent to the server. The input is the elderly person's voice, and the output is the recorded voice data. In this step, the voice is captured using the SpeechRecognition library.

[0204] Step 7:

[0205] The server analyzes the received voice data and generates an appropriate response. This process is performed using a generative AI model, and includes voice-to-text conversion, semantic analysis, and response generation. The input is the voice data, and the output is the generated response text.

[0206] Step 8:

[0207] The generated response is converted into speech and played back to the elderly. The device uses the pyttsx3 library to convert text into speech and play it back to the elderly. The input is the response text, and the output is the speech to be played to the elderly.

[0208] Step 9:

[0209] The device collects feedback from the elderly during the interaction. It records the elderly's reactions and evaluations and sends them to the server. The input is the elderly's feedback, and the output is the collected feedback data.

[0210] Step 10:

[0211] The server retrains the generative AI model based on the collected feedback data. In this step, the feedback data is used to fine-tune the model and learn it so that it can be reflected in the next conversation. The input is the feedback data, and the output is the retrained generative AI model.

[0212] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0213] The present invention provides a system that combines a generative AI model, virtual reality (VR) technology, and an emotion engine that recognizes user emotions to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing data related to individuals, and training and optimizing the generative AI model. Furthermore, the system provides an interface for realizing dialogue between the elderly and the generative AI model, recognizes the elderly's voice input, generates and plays a response, and collects feedback from the elderly and uses it to retrain the model. Furthermore, the emotion engine recognizes the user's emotions and adjusts the generative AI model's response based on the emotion.

[0214] Overall system picture

[0215] First, users (family members or friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data can include family memories, anecdotes about specific events, and everyday conversations. The server receives the provided data and performs preprocessing. This preprocessing includes noise reduction for the voice data and normalization for the text data.

[0216] The server then uses the preprocessed data to train a generative AI model, which is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the generative AI model to provide natural and friendly responses to the elderly.

[0217] Building an interface and emotion engine

[0218] The terminal (device used by the elderly) will have a user interface tailored to the elderly. This interface will provide navigation with large buttons, text, and voice guidance that take visual and auditory impairments into consideration. It will also incorporate an emotion engine that will analyze and recognize the emotional state of the elderly from voice input and facial expression data.

[0219] When an elderly person initiates a dialogue with the generative AI model through their device, the device records the elderly person's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. The emotion engine analyzes the elderly person's emotional state and adjusts the content and tone of the response. The generated response is then played back by the device.

[0220] Gathering feedback and retraining the model

[0221] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions for the elderly.

[0222] Specific examples

[0223] Below is a concrete example of how the system actually works.

[0224] When an older person talks about a family trip they took long ago

[0225] Users (family and friends) upload audio and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When an elderly person speaks to the device, saying, "Tell me about your past trips," the device recognizes this and sends the audio data to the server.

[0226] The server analyzes the voice data and generates a response about the family trip. At this time, the emotion engine reads the elderly person's emotions from their tone of voice and facial expressions. If it determines that the elderly person is having fun, for example, the response will be a positive one such as, "On our last trip, we all went to a hot spring together as a family. It was fun!" The generated response is then played back by the device.

[0227] The device then records the elderly person's responses and sends feedback to the server, which uses this feedback to retrain the model and provide more appropriate responses in the next interaction.

[0228] In this way, elderly people can easily use the system at home and it can help prevent and improve dementia. The introduction of the emotion engine makes the dialogue more natural and friendly, further supporting the mental health of the elderly.

[0229] The processing flow will be explained below.

[0230] Step 1:

[0231] Users (family and friends) use a web portal or dedicated app to upload voice and text data related to the elderly, including family memories, anecdotes about specific events, and everyday conversations.

[0232] Step 2:

[0233] The server preprocesses the received data, specifically removing noise from the audio data and normalizing and deleting unnecessary parts from the text data.

[0234] Step 3:

[0235] The server stores the preprocessed data in a database, which is then used to train the generative AI model.

[0236] Step 4:

[0237] The server uses the stored data to train the generative AI model, extracting speech features from the voice data and using natural language processing to extract important keywords from the text data.

[0238] Step 5:

[0239] The server optimizes the trained generative AI model for seniors, adjusting the model parameters based on the seniors' cognitive level, voice characteristics, and individual preferences.

[0240] Step 6:

[0241] The device will create a user interface tailored to the elderly, providing large buttons and text that are visually and auditorily friendly, as well as voice-guided navigation.

[0242] Step 7:

[0243] The device will be equipped with an emotion engine and configured to analyze and recognize the emotional state of the elderly person from voice input and facial expression data.

[0244] Step 8:

[0245] The user (elderly person) initiates a dialogue with the generative AI model through the device. When the elderly person speaks to the device, the device records it in real time and sends it to the server.

[0246] Step 9:

[0247] The server analyzes the received voice data, and the emotion engine estimates the elderly person's emotional state from their tone of voice and facial expressions, and provides this data to the generative AI model.

[0248] Step 10:

[0249] Based on the emotional state estimated by the emotion engine, the generative AI model generates an appropriate response. For example, if the server determines that the elderly person is having fun, it generates a positive response.

[0250] Step 11:

[0251] The server sends the generated response to the terminal, which converts the response into speech and plays it back to the elderly person.

[0252] Step 12:

[0253] The device records the elderly person's reactions and feedback in real time and transmits them to a server.

[0254] Step 13:

[0255] The server uses the feedback collected to retrain the generative AI model and further optimize it for older people, improving the accuracy and performance of the system.

[0256] This allows seniors to enjoy interacting with generative AI models, stimulating their cognitive functions and promoting their mental health. The introduction of an emotion engine makes interactions more natural and friendly.

[0257] Example 2

[0258] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0259] To prevent and improve dementia in the elderly, it is important to provide friendly and appropriate dialogue for each individual elderly person. However, conventional dialogue systems have difficulty providing optimal responses that reflect the emotional state and personal preferences of each elderly person. Furthermore, for elderly people to continue using the system, the dialogue needs to be natural and adapt to their emotions. This is required to maintain and improve the mental health of the elderly.

[0260] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0261] In this invention, the server includes means for collecting personal data provided by a user, means for preprocessing the collected data and performing noise reduction and normalization, means for training a generative AI model based on the preprocessed data, means for optimizing the trained model for the elderly, means for providing an interface for realizing a dialogue between the elderly and the generative AI model, means for recognizing the elderly's voice input, analyzing the input, and generating a response, means for playing the generated response to the elderly, means for collecting feedback from the elderly and using it to retrain the model, an emotion engine for recognizing the elderly's emotional state, and means for adjusting the response of the generative AI model based on the emotion engine. This makes it possible to provide a natural and friendly dialogue that adapts to emotions and improve the mental health of the elderly.

[0262] "User" refers to the entity that provides data, such as an individual, family member, or friend who uses the system.

[0263] "Personal data" refers to information related to an elderly person, such as their lifestyle, memories, tone of voice, upbringing, preferences, prohibitions, dialect, etc.

[0264] "Preprocessing" refers to the process of processing collected data, such as by removing noise and normalizing it, to prepare it in a form that can be used by a generative AI model.

[0265] A "generative AI model" is an artificial intelligence model that generates natural responses based on given data.

[0266] "Training" refers to the process of adjusting the parameters and algorithms of a generative AI model using collected data so that it can generate appropriate responses.

[0267] An "interface" is something that provides an operation screen and input means for a user to interact with a system.

[0268] The "emotion engine" is an engine that analyzes voice input and facial expression data to recognize the emotional state of the elderly.

[0269] "Feedback" refers to data about reactions, opinions, and behaviors obtained from seniors while using the system.

[0270] "Retraining" refers to the process of re-adjusting a generative AI model based on collected feedback data to improve its accuracy and performance.

[0271] "Means for generating a response" refers to the technology or method for analyzing the senior's voice input and generating an appropriate response.

[0272] A "database" is a collection of information that stores data that a generative AI model references when generating responses.

[0273] MODE FOR CARRYING OUT THE INVENTION

[0274] The present invention provides a system that combines a generative AI model, virtual reality (VR) technology, and an emotion engine that recognizes user emotions to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing data related to individuals, and training and optimizing the generative AI model. Furthermore, the system provides an interface for realizing dialogue between the elderly and the generative AI model, recognizes the elderly's voice input, generates and plays a response, and collects feedback from the elderly and uses it to retrain the model. Furthermore, the emotion engine recognizes the user's emotions and adjusts the generative AI model's response based on the emotion.

[0275] Overall system picture

[0276] First, users (family members or friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data can include family memories, anecdotes about specific events, and everyday conversations. The server receives the provided data and performs preprocessing. This preprocessing includes noise reduction for the voice data and normalization for the text data.

[0277] The server then uses the preprocessed data to train a generative AI model, which is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the generative AI model to provide natural and friendly responses to the elderly.

[0278] Building an interface and emotion engine

[0279] The terminal (device used by the elderly) will have a user interface tailored to the elderly. This interface will provide navigation with large buttons, text, and voice guidance that take visual and auditory impairments into consideration. It will also incorporate an emotion engine that will analyze and recognize the emotional state of the elderly from voice input and facial expression data.

[0280] When an elderly person initiates a dialogue with the generative AI model through their device, the device records the elderly person's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. The emotion engine analyzes the elderly person's emotional state and adjusts the content and tone of the response. The generated response is then played back by the device.

[0281] Gathering feedback and retraining the model

[0282] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions for the elderly.

[0283] Specific examples

[0284] Below is a concrete example of how the system actually works.

[0285] When an older person talks about a family trip they took long ago

[0286] Users (family and friends) upload audio and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When an elderly person speaks to the device, saying, "Tell me about your past trips," the device recognizes this and sends the audio data to the server.

[0287] The server analyzes the voice data and generates a response about the family trip. At this time, the emotion engine reads the elderly person's emotions from their tone of voice and facial expressions. If it determines that the elderly person is having fun, for example, the response will be a positive one such as, "On our last trip, we all went to a hot spring together as a family. It was fun!" The generated response is then played back by the device.

[0288] The device then records the elderly person's response and sends feedback to the server. The server uses this feedback to retrain the model and provide a more appropriate response the next time the conversation takes place. In this way, the elderly can easily use the system at home, helping to prevent and improve dementia. The introduction of an emotion engine makes conversations more natural and friendly, further supporting the mental health of the elderly.

[0289] Prompt Sentence Examples

[0290] An elderly person might speak into the device and say, "Tell me about your past travels."

[0291] This allows the system to respond to the user's emotions and provide a more friendly dialogue for the elderly.

[0292] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0293] Step 1: Data provision and collection

[0294] Users provide personal audio and text data through a dedicated web portal or application. For example, they upload stories about family trips or everyday conversations. As input, users send audio and text files to the server. This data is received by the server and saved in local storage. As output, saved audio and text data is generated.

[0295] Step 2: Data Preprocessing

[0296] The server performs preprocessing on the collected data. Specific operations include removing noise from the audio data and converting the audio into text using speech recognition technology. Normalization processing (standardization of notation, removal of unnecessary spaces, etc.) is performed on the text data. This generates preprocessed, clean text data.

[0297] Step 3: Training the generative AI model

[0298] The server trains a generative AI model using the preprocessed data. As input, clean text data is provided to the model. The generative AI model (e.g., GPT-4®) is used to learn appropriate responses to specific prompt sentences. As output, the generative AI model is able to generate natural and friendly responses for seniors.

[0299] Step 4: Optimize the model

[0300] The trained model is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences. The elderly person's profile information (cognitive level, preferences, etc.) is used as input. This results in the generation of a generative AI model optimized for the elderly person as the output.

[0301] Step 5: Providing an Interface

[0302] The terminal provides an interface that is easy for seniors to use. It is designed with consideration for visual and auditory impairments, with large buttons, text, and voice-guided navigation. The terminal accepts the operations of the seniors as input. An intuitive and easy-to-operate user interface is provided as output.

[0303] Step 6: Emotion Recognition with the Emotion Engine

[0304] The device collects voice input and facial expression data from the elderly person and uses an emotion engine to analyze and recognize their emotional state. Real-time voice and facial expression data are used as input. Specifically, data is acquired using the device's camera and microphone, and analyzed by the emotion engine. The output is data representing the recognized emotional state.

[0305] Step 7: Dialogue execution and response generation

[0306] When an elderly person speaks to the device, the device records the voice input in real time and sends it to the server. For example, the elderly person might say, "Tell me about your travels in the past." The recorded voice data is used as input. The server analyzes the voice data and generates an appropriate response using a generative AI model. At this time, the emotion engine adjusts the content and tone of the response based on the elderly person's emotional state. The output is a natural and friendly response that adapts to the emotion.

[0307] Step 8: Playing back the response

[0308] The generated response is played back by the device. Specifically, it is played back audibly using the device's speaker. The audio data of the generated response is used as input. The output is an appropriate audio response provided to the elderly person.

[0309] Step 9: Gather feedback

[0310] The device records the elderly person's reactions and feedback during the dialogue process. The input is the elderly person's remarks and changes in facial expressions. Specific operations include collecting data in real time using the device's sensors and analyzing it with an emotion engine. The collected feedback data is generated as output.

[0311] Step 10: Retrain the model

[0312] The server retrains the generative AI model based on the collected feedback. It uses the feedback data as input, allowing the generative AI model to continuously improve its accuracy and performance. The output is a retrained, highly accurate generative AI model.

[0313] Through the above processing steps, the system of the present invention realizes natural and friendly dialogue with the elderly, and can be useful in preventing and improving dementia.

[0314] (Application example 2)

[0315] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0316] When elderly people shop in physical stores, they often have difficulty selecting and using products, or finding the right products. To ensure that elderly people can shop with peace of mind, they need support that can provide appropriate advice and suggestions in real time. Furthermore, to prevent and improve dementia, personalized support using emotion recognition technology is needed.

[0317] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0318] In this invention, the server includes: means for collecting personal data provided by a user; means for preprocessing the collected data and performing noise reduction and normalization; means for training a generative AI model based on the preprocessed data; means for optimizing the trained model for the elderly; means for providing an interface for enabling dialogue between the elderly and the generative AI model; means for recognizing the elderly's voice input, analyzing the input, and generating a response; means for playing the generated response to the elderly; means for collecting feedback from the elderly and using it to retrain the model; means for adjusting the response of the generative AI model based on the elderly's emotions using an emotion engine that recognizes emotions from the elderly's facial expressions and voice; and means for recognizing emotions in real time and providing appropriate product suggestions and advice when the elderly selects products using a head-mounted display. This enables elderly people to receive appropriate emotion-based support in real time when shopping in physical stores.

[0319] "Users" refers to people who use the system, and in this context primarily includes elderly people and their families and friends.

[0320] "Personal data" refers to information about an older adult, including the user's tone of voice, background, preferences, prohibitions, dialect, etc.

[0321] "Means of collection" refers to the methods and technologies used to collect data provided by older adults and their families.

[0322] "Preprocessing" refers to the process of removing noise and normalizing collected data to make it easier to use.

[0323] "Noise reduction" refers to the process of removing unnecessary information from collected data.

[0324] "Normalization" refers to the process of converting collected data into a uniform format.

[0325] A "generative AI model" refers to an artificial intelligence model that is trained based on collected data and generates dialogue with elderly people.

[0326] "Training means" refers to the methods and techniques used to train and optimize generative AI models using collected data.

[0327] "Interface" refers to the point of contact between the user and the system that the elderly person uses to interact with the generative AI model.

[0328] "Means for recognizing voice input" refers to technologies and methods for recognizing and understanding what an older adult is saying.

[0329] "Means for analyzing" refers to techniques or methods for analyzing the senior's voice input and generating an appropriate response.

[0330] "Means for generating a response" refers to a technique or method for generating an appropriate reply based on the analyzed voice input.

[0331] "Means of playback" refers to the technology or method of outputting the generated response in audio or text format to convey it to the elderly.

[0332] "Feedback" refers to the reactions and opinions of the elderly during the dialogue, and is information used to improve the model.

[0333] The "emotion engine" refers to technology that recognizes emotions from the facial expressions and voice of elderly people and adjusts the response of the generative AI model.

[0334] A "head-mounted display" refers to a device worn by elderly people that displays information visually.

[0335] "Product suggestions and advice" refers to appropriate support provided in real time when seniors are choosing products.

[0336] This invention is a system for preventing and improving dementia in the elderly, and is composed of a generative AI model, virtual reality (VR) technology, and an emotion engine. The detailed configuration and operation of the system are described below.

[0337] Overall system configuration

[0338] The system mainly consists of the following components:

[0339] Means of collecting personal data provided by users

[0340] A means of preprocessing the collected data to remove noise and normalize it

[0341] A means of training generative AI models on preprocessed data

[0342] A means of optimizing trained models for older people

[0343] A means of providing an interface for enabling dialogue between elderly people and generative AI models

[0344] A means of recognizing an older adult's voice input and analyzing that input to generate a response

[0345] A means of playing back generated responses to older adults

[0346] A means of collecting feedback from seniors and using it to retrain the model

[0347] A means of adjusting the responses of generative AI models based on the emotions of the elderly, using an emotion engine that recognizes emotions from the elderly's facial expressions and voice.

[0348] A method to recognize emotions in real time and provide appropriate product suggestions and advice to elderly people when they are selecting products using a head-mounted display.

[0349] Program Implementation

[0350] The system program is configured as follows:

[0351] First, the server collects personal data provided by the user, including the elderly person's voice tone, background, preferences, restrictions, dialect, etc. Then, the server preprocesses the collected data, removing noise from the audio data and normalizing the text data.

[0352] Using the pre-processed data, the server trains a generative AI model that is optimized for the cognitive level, vocal characteristics, and individual preferences of seniors.

[0353] When the senior initiates a dialogue, the interface records the senior's voice input in real time and sends it to the server. The server analyzes the voice data and uses a generative AI model to generate an appropriate response. The emotion engine recognizes emotions from the senior's facial expressions and voice, and adjusts the content and tone of the response based on their emotional state.

[0354] The generated responses are then played back to the elderly, and feedback from the elderly is collected and used by the server to retrain the generative AI model.

[0355] Hardware and software used

[0356] Camera-equipped head-mounted display (HMD): Used by elderly people when selecting products in a store, it captures facial expressions and voice in real time.

[0357] Server: Collects user data, pre-processes it, trains generative AI models, analyzes voice input, generates responses, and retrains them.

[0358] Emotion engine: Recognizes emotions from the elderly's facial expressions and voice in real time and uses them to adjust the responses of generative AI models.

[0359] software:

[0360] OpenCV: Acquiring and displaying camera input

[0361] TensorFlow: Running the GPT-2 model

[0362] transformers: Hugging Face GPT-2 tokenizer and model

[0363] EmotionEngine (used as a temporary emotion recognition library)

[0364] Specific examples

[0365] For example, if an elderly person is wearing a head-mounted display and searching for a product in a physical store, the camera captures the elderly person's facial expression, and the emotion engine recognizes from that expression that the elderly person is having trouble. In this case, the generative AI model is prompted to "generate advice about the product that the elderly person is having trouble with." As a result, a response is generated that reads, "This product is a little complicated to use, but it's easy to use if you follow the instructions. Ask a staff member," and this response is played back to the elderly person.

[0366] Example prompt sentence:

[0367] "Generate suggestions for products that seniors enjoy."

[0368] "Generate advice on products that seniors have trouble with."

[0369] In this way, the system allows elderly people to shop in physical stores with peace of mind while also helping to prevent and improve dementia.

[0370] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0371] Step 1:

[0372] Collecting data relating to individuals

[0373] Users provide personal data (tone of voice, background, preferences, restrictions, dialect, etc.) through a dedicated web portal or application, which then inputs the data into the server, which receives it and stores it in a database.

[0374] Step 2:

[0375] Data Preprocessing

[0376] The server performs denoising and normalisation on the received data, which includes denoising audio data and normalising text data. The input is the data provided by the user and the output is the pre-processed, clean data.

[0377] Step 3:

[0378] Training generative AI models

[0379] The server uses the preprocessed data to train a generative AI model. Based on the preprocessed data input as training data, the generative AI model is optimized for elderly people. The output is a trained generative AI model.

[0380] Step 4:

[0381] Providing a dialogue interface with a generative AI model optimized for the elderly

[0382] The elderly person wears a head-mounted display (HMD) as a dialogue interface. The HMD provides a dialogue interface and supports dialogue between the elderly person and the generative AI model through voice input and visual information.

[0383] Step 5:

[0384] Recognition and analysis of elderly voice input

[0385] When an elderly person speaks about a product, the device (HMD) records this voice input in real time and sends it to the server. The server analyzes the voice input and recognizes what the elderly person is saying. The input is the elderly person's voice data, and the output is analyzed text data.

[0386] Step 6:

[0387] Emotion Recognition Using Emotion Engine

[0388] The server receives voice input and facial expression data of the elderly captured by the camera on the HMD. The emotion engine analyzes this data and recognizes the elderly's emotional state. The input is voice data and facial expression data, and the output is recognized emotional information.

[0389] Step 7:

[0390] Response Generation

[0391] The server inputs the recognized text data and emotion information into the generative AI model as a prompt. For example, a prompt such as "Generate suggestions for products that seniors enjoy" is sent. The generative AI model then generates an appropriate response based on this. The input is the prompt text, and the output is the generated response text.

[0392] Step 8:

[0393] Response playback

[0394] The device (HMD) receives the generated response sent from the server and plays it back to the elderly audibly and visually. For example, a message such as "This product is easy to use, please try it" is played back. The input is the response text, and the output is the played back audio message.

[0395] Step 9:

[0396] Collecting feedback

[0397] After receiving the response, the elderly person's reactions and additional comments are recorded by the device and sent to the server, which collects this feedback data and uses it to retrain the generative AI model. The input is the elderly person's reactions and feedback data, and the output is a dataset for retraining.

[0398] Through the above processing steps, the system can assist seniors in their shopping experience and provide appropriate emotion-based responses in real time.

[0399] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0400] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0401] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0402] [Second embodiment]

[0403] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0404] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0405] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0406] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0407] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0408] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0409] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0410] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0411] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0412] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0413] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0414] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0415] The present invention provides a system that combines a generative AI model with virtual reality (VR) technology to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing data related to individuals, and training and optimizing the generative AI model. It also provides an interface for realizing dialogue between the elderly and the generative AI model, recognizing the elderly's voice input, generating and playing back a response, and collecting feedback from the elderly for use in retraining the model.

[0416] Overall system picture

[0417] First, users (family members or friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data can include family memories, anecdotes about specific events, and everyday conversations. The server receives the provided data and performs preprocessing. This preprocessing includes noise reduction for the voice data and normalization for the text data.

[0418] The server then uses the preprocessed data to train a generative AI model, which is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the generative AI model to provide natural and friendly responses to the elderly.

[0419] Building the interface

[0420] The terminals (devices used by the elderly) will have user interfaces tailored to the elderly, providing large buttons and text that are visually and auditorily friendly, as well as voice-guided navigation.

[0421] When an elderly person initiates a dialogue with the generative AI model through the device, the device records the elderly person's voice input in real time and sends it to the server, which analyzes the voice data and generates an appropriate response, which is then played back by the device.

[0422] Gathering feedback and retraining the model

[0423] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions for the elderly.

[0424] Specific examples

[0425] Below is a concrete example of how the system actually works.

[0426] When an older person talks about a family trip they took long ago

[0427] Users (family and friends) upload audio and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When an elderly person speaks to the device, saying, "Tell me about your past trips," the device recognizes this and sends the audio data to the server.

[0428] The server analyzes the voice data and generates a response about a family trip. The generated response is then played back by the device as, "On a previous trip, we all went to a hot spring together as a family." The device then records the elderly person's response and sends feedback to the server. Based on this, the model is retrained to provide a more appropriate response in the next interaction.

[0429] In this way, elderly people can easily use the system at home, helping to prevent and improve dementia. The system provides new stimulation and joy to the lives of elderly people, supports their mental health, and contributes to reducing feelings of loneliness.

[0430] The processing flow will be explained below.

[0431] Step 1:

[0432] Users (family and friends) use a web portal or dedicated app to upload audio and text data related to the elderly, including family memories, anecdotes about specific events, and everyday conversations.

[0433] Step 2:

[0434] The server preprocesses the data it receives. Specifically, it performs noise removal for audio data, normalizes text data, and removes unnecessary parts.

[0435] Step 3:

[0436] The server stores the preprocessed data in a database, which is then used to train the generative AI model.

[0437] Step 4:

[0438] The server uses the stored data to train the generative AI model, extracting speech features from the voice data and using natural language processing to extract important keywords from the text data.

[0439] Step 5:

[0440] The server optimizes the trained generative AI model for seniors, adjusting the model parameters based on the seniors' cognitive level, voice characteristics, and individual preferences.

[0441] Step 6:

[0442] The device will create a user interface tailored to the elderly, providing large buttons and text that are visually and auditorily friendly, as well as voice-guided navigation.

[0443] Step 7:

[0444] The user (elderly person) initiates a dialogue with the generative AI model through the device. When the elderly person speaks to the device, the device records it in real time and sends it to the server.

[0445] Step 8:

[0446] The server analyzes the voice data sent and generates an appropriate response using a generative AI model.

[0447] Step 9:

[0448] The server sends the generated response to the terminal, which converts the response into speech and plays it back to the elderly person.

[0449] Step 10:

[0450] The device records the elderly person's reactions and feedback in real time and transmits them to a server.

[0451] Step 11:

[0452] The server uses the feedback collected to retrain the generative AI model and further optimize it for older people, improving the accuracy and performance of the system.

[0453] Through this process, seniors can enjoy interacting with generative AI models, stimulating their cognitive functions and promoting their mental health.

[0454] Example 1

[0455] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0456] Dialogue systems aimed at preventing and improving dementia in the elderly require mechanisms that are friendly to the elderly and provide appropriate responses. Conventional dialogue systems have difficulty generating appropriate responses based on the cognitive level and preferences of each individual elderly person, making it difficult to achieve natural dialogue in real time in response to elderly voice input. Furthermore, there have been insufficient methods for appropriately collecting feedback from the elderly and using it to retrain models. This has resulted in technical challenges for smoothly introducing dialogue systems into the daily lives of the elderly.

[0457] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0458] In this invention, the server includes a means for collecting personal information provided by the user, a means for preprocessing the collected information, removing unnecessary data, and standardizing the format, and a means for training a generative artificial intelligence model based on the preprocessed information. This makes it possible to provide natural and friendly dialogue responses to each elderly person. Furthermore, the system recognizes the elderly person's voice input, provides an interface for analyzing the input and generating an appropriate response, and outputs the generated response to the elderly person, thereby achieving natural dialogue in real time. Furthermore, responses from the elderly are collected and used to retrain the generative artificial intelligence model, thereby continuously improving the accuracy of the model. In this way, it is possible to provide an effective dialogue system that contributes to the prevention and improvement of dementia in the elderly.

[0459] A "user" is a person or group that provides information related to a particular individual.

[0460] "Information" is a general term for data related to an individual, such as the tone of voice of family members, background, interests, prohibited activities, and local dialects.

[0461] A "collection method" is a method or device used to collect information from a user.

[0462] "Preprocessing" refers to processing carried out to remove unnecessary data from collected information and standardize the format.

[0463] A "generative artificial intelligence model" is a machine learning model that is trained based on collected information and aims to generate natural-sounding dialogue responses.

[0464] A "training tool" refers to a method or technique for optimizing a generative artificial intelligence model for a specific purpose.

[0465] "Elderly optimization means" refers to methods and techniques for adjusting generative artificial intelligence models to provide friendly and natural dialogue responses to seniors.

[0466] "Connection means" refers to interfaces or devices that enable interaction between the elderly person and the generative artificial intelligence model.

[0467] "Voice input" refers to the voice or spoken words produced by the elderly person.

[0468] A "means for generating a response" is a method or technique for analyzing speech input and creating an appropriate reply.

[0469] "Output means" refers to a method or device for communicating the generated response to the elderly person.

[0470] Response collection tools are methods and techniques used to gather responses and feedback from older adults.

[0471] This invention provides a system that combines a generative artificial intelligence model with virtual reality (VR) technology to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing information related to an individual, and training and optimizing the generative artificial intelligence model. Furthermore, the system provides an interface for realizing dialogue between the elderly and the generative artificial intelligence model, recognizes the elderly's voice input, generates and plays back a response, and collects responses from the elderly for use in retraining the model.

[0472] Data collection methods

[0473] Users (elderly family members and friends) provide personal information, such as their family members' voices, backgrounds, interests, prohibited activities, and local dialects, through a dedicated web portal or application. This information includes memories of family trips and everyday conversations. Specifically, users open the application and upload images, audio files, and text. For example, they can upload photos and descriptions of past family trips.

[0474] Data preprocessing methods

[0475] The server receives the provided information and performs preprocessing, which includes denoising the audio data and standardising the format of the text data. Specifically, the server uses the Python library Librosa to denoise the audio data and the NLTK library to correct spelling and check grammar of the text data.

[0476] A means of training and optimizing generative artificial intelligence models

[0477] The server uses the preprocessed information to train a generative artificial intelligence model using machine learning libraries such as TensorFlow and PyTorch. The trained model is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the model to provide natural and friendly responses to the elderly.

[0478] Building the user interface

[0479] Terminals (devices used by the elderly) will have user interfaces tailored to the elderly. These interfaces will include large buttons and text that take visual and hearing impairments into consideration, as well as voice-guided navigation. Specifically, terminals will be designed to take visual impairments into consideration, displaying buttons and text in large fonts and with clear contrast. They will also provide voice assistant functionality to make them easy for the elderly to operate.

[0480] Real-time interaction means

[0481] When an elderly person begins a dialogue with the generative AI model through the device, the device records the elderly person's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. For example, if an elderly person says, "Tell me about your old trips," the device records this and sends the voice data to the server. The server analyzes this voice data and "generates a response about family trips." The generated response is played back by the device as, "On your old trips, we all went to hot springs together as a family."

[0482] A means of gathering feedback and retraining the model

[0483] During the interaction, the device collects the elderly's reactions and feedback and sends it to the server. Based on this feedback, the server retrains the generative AI model. Specifically, the device asks whether the elderly is satisfied with the response and sends the result to the server. The server uses this feedback to update the model and provide more appropriate responses in the next interaction.

[0484] Specific examples

[0485] Below is a concrete example of how the system actually works.

[0486] When an older person talks about a family trip they took long ago

[0487] Users (family and friends) upload voice and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When the elderly person speaks to the device, saying, "Tell me about your old trips," the device recognizes this and sends the voice data to the server. The server analyzes the voice data and "generates a response about your family trip." The generated response is then played back by the device as, "On your old trips, we all went to a hot spring." The device then records the elderly person's response and sends feedback to the server. Based on this, the model is retrained to provide a more appropriate response in the next interaction.

[0488] Prompt Sentence Examples

[0489] "How are you feeling today?"

[0490] "Want to talk about past family trips?"

[0491] "Tell me some memories you have with your grandchildren."

[0492] In this way, the system can activate memories and conversational content of the elderly, helping to prevent and improve dementia. The system provides new stimulation and joy to the lives of the elderly, supporting their mental health and contributing to reducing loneliness.

[0493] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0494] Step 1:

[0495] Data collection

[0496] Users use a dedicated web portal or application to provide information about their family members, such as their voices, backgrounds, interests, prohibited activities, and local dialects, as well as upload photos, audio files, and text data.

[0497] Input: Audio data, text data, image data

[0498] Output: Collected information (stored in a database)

[0499] What happens: A user opens the application and uploads photos and descriptions of past family trips.

[0500] Step 2:

[0501] Data Preprocessing

[0502] The server preprocesses the collected information: first, it uses the Python library Librosa to denoise the audio data, and then it uses the NLTK library to standardize the format of the text data (for example, to perform spelling correction and grammar checks).

[0503] Input: Collected information (output of step 1)

[0504] Output: Preprocessed information (denoised audio data, normalized text data)

[0505] Specific operation: The server receives the audio data and uses Librosa to remove noise. It also processes the text data with NLTK and standardizes the format.

[0506] Step 3:

[0507] Training and optimizing generative AI models

[0508] The server trains a generative artificial intelligence model based on the preprocessed information using machine learning libraries such as TensorFlow and PyTorch, optimizing the model based on the elderly person's cognitive level, voice characteristics, and individual preferences.

[0509] Input: Preprocessed information (output of step 2)

[0510] Output: A trained generative artificial intelligence model

[0511] Specific operation: The server uses preprocessed voice and text data to train a generative artificial intelligence model and optimize it for the elderly.

[0512] Step 4:

[0513] Building the user interface

[0514] The device will have a user interface tailored to seniors, including large buttons and text for visual and hearing impairments, as well as voice-guided navigation.

[0515] Input: Design specifications, needs of the elderly

[0516] Output: User Interface (UI)

[0517] What it does: Your device displays buttons and text in large fonts and with clear contrast, and offers voice assistant functionality.

[0518] Step 5:

[0519] Real-time dialogue

[0520] When an elderly person initiates a dialogue with the generative AI model through the device, the device records the elderly person's voice input in real time and transmits it to the server, which analyzes the voice input and generates an appropriate response.

[0521] Input: Voice input by elderly people

[0522] Output: The generated response

[0523] Specific operation: When an elderly person says, "Tell me about your past travels," the device records this and sends the audio data to the server. The server analyzes the audio data, "generates a response about family trips," and sends it back to the device. The device then plays back the generated response.

[0524] Step 6:

[0525] Gathering feedback and retraining the model

[0526] During the interaction, the device collects the elderly's reactions and feedback and sends it to the server, which uses this feedback to retrain the generative artificial intelligence model.

[0527] Input: Older adults' reactions and feedback

[0528] Output: A retrained generative AI model

[0529] How it works: The device checks whether the elderly person is satisfied with the response and sends the result to the server. The server uses the feedback to update the model and provide a more appropriate response in the next interaction.

[0530] (Application example 1)

[0531] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0532] There is a need to provide an environment where elderly people can easily select and order meals while staying safe at home. It is also necessary to introduce dialogue systems that contribute to the prevention and improvement of dementia. However, existing systems have not been able to adequately respond to individual needs by taking into account the preferences and history of elderly people. Furthermore, in order for dialogue systems to generate natural and friendly responses to elderly voice input, advanced processing in speech recognition technology and generative AI models is required, posing a challenge to practical implementation.

[0533] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0534] In this invention, the server includes: means for collecting personal data provided by a user; means for preprocessing the collected data to remove noise and normalize it; means for training a generative AI model based on the preprocessed data; means for optimizing the trained model for the elderly; means for providing an interface for enabling dialogue between the elderly and the generative AI model; means for recognizing the elderly's voice input, analyzing the input, and generating a response; means for playing the generated response to the elderly; means for collecting feedback from the elderly and using it to retrain the model; means for presenting a menu based on the elderly's past eating history and preferences; and means for ordering meals based on the presented menu. This enables personalized meal suggestions and ordering based on the elderly's own preferences and history, and also provides an interactive system that is effective in preventing and improving dementia.

[0535] "Personal data provided by the user" refers to information related to the elderly person provided by the user, including the family member's tone of voice, upbringing, preferences, things they don't like, dialect, etc.

[0536] "Collected data" refers to audio data, text data, and the like that are uniquely associated with the senior, including information provided by the user.

[0537] "Preprocessing" refers to processing performed on collected data, such as noise removal and data normalization.

[0538] A "generative AI model" is an artificial intelligence model that has the ability to learn from large datasets and respond to specific tasks.

[0539] "Methods for optimizing for the elderly" refer to techniques for adjusting generative AI models based on the cognitive level, voice characteristics, and individual preferences of the elderly.

[0540] The "interface" is a means of providing an operating screen and voice guidance for elderly people to interact with the generated AI model.

[0541] "Means for recognizing a senior's voice input and analyzing the input to generate a response" refers to the overall process of analyzing a senior's voice and generating an appropriate response.

[0542] "Means for playing the generated response to the elderly" refers to a method for letting the elderly hear the voice response generated by the generative AI model.

[0543] "Means of collecting feedback from seniors and using it to retrain the model" refers to the process of improving the performance of generative AI models based on reactions and opinions obtained from interactions with seniors.

[0544] "Means for presenting menus based on past eating history and preferences" refers to a function that presents appropriate menus based on the elderly person's past eating history and preferences.

[0545] "Means for ordering food based on a presented menu" refers to the functionality of making selections from a presented menu and actually completing a food order.

[0546] This invention aims to prevent and improve dementia in the elderly using the following systems and processes.

[0547] Overall system picture

[0548] First, users (family and friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data includes past dietary history and preferences, the family member's voice tone, upbringing, things they don't like, dialects, etc. The server receives this data and performs noise reduction and data normalization.

[0549] The server then uses the preprocessed data to train a generative AI model, optimizing it based on the elderly person's cognitive level, voice characteristics, and individual preferences, allowing the trained generative AI model to generate natural and friendly responses in real time.

[0550] Building the interface

[0551] The terminal (the device used by the elderly) provides a user interface tailored to the elderly. This interface features large buttons, text, and voice-guided navigation that take visual and auditory impairments into consideration. When the elderly initiates a dialogue with the generative AI model through the terminal, the terminal records the elderly's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. This response is then played back by the terminal.

[0552] Food ordering function

[0553] A distinctive feature of this system is its ability to present menus based on the elderly person's past eating history and preferences. When the elderly person asks, "What's on the menu today?", the device recognizes this and sends the voice data to the server. The server then generates an optimal response based on this and suggests dishes suitable for the elderly. In this way, the application assists the elderly in the process of selecting and ordering meals.

[0554] Gathering feedback and retraining the model

[0555] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions and services for the elderly.

[0556] Hardware and software used

[0557] The system uses the following main hardware and software:

[0558] Smartphones / head-mounted displays: Devices used by the elderly.

[0559] TensorFlow / Keras: Used for training and inference of generative AI models.

[0560] SpeechRecognition: Provides speech recognition technology.

[0561] pyttsx3: Used to speak the generated response.

[0562] Adding specific examples

[0563] For example, if a user (an elderly person) says, "What's today's recommended menu?", the system will generate the following response, taking into account the elderly person's eating history and preferences:

[0564] "Today, I recommend the grilled eel and soup set. Would you like to order that?"

[0565] Prompt Sentence Examples

[0566] The prompt should be in the following format:

[0567] Senior citizen: "What's on the menu today?"

[0568] Generative AI: "Based on your past eating history and preferences, the recommended menu is a set of grilled eel and clear soup. Would you like to order that?"

[0569] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0570] Step 1:

[0571] Users (family and friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This includes past meal history, the family member's voice tone, upbringing, preferences, things they don't like, dialects, etc. This input data is sent to the server. The main purpose of this step is to collect the input data (voice and text data). The output is the initial data before it is preprocessed.

[0572] Step 2:

[0573] The server preprocesses the received data related to individuals, specifically by removing noise and normalizing the data. This process uses the acquired voice and text data as input and processes it through noise filtering, tokenization, and other processes. This results in clean data that the generative AI model can accurately learn from.

[0574] Step 3:

[0575] The server trains a generative AI model based on the preprocessed data. The server uses deep learning libraries such as TensorFlow and Keras to generate a model with voice response capabilities based on the input data. The output of this step is a highly specialized generative AI model.

[0576] Step 4:

[0577] The trained model is optimized for seniors. The server adjusts the model parameters based on the seniors' cognitive level, voice characteristics, and individual preferences. In this process, the generative AI model obtained in the previous step is further tuned so that it can generate responses appropriate for seniors. The output is a generative AI model optimized for seniors.

[0578] Step 5:

[0579] The device provides an interface for dialogue between the elderly person and the generative AI model. It displays an interface that takes visual and auditory impairments into consideration, including large buttons, text, and audio guides. The input is the elderly person's operations, and the output is a GUI (graphical user interface) for dialogue.

[0580] Step 6:

[0581] The elderly person inputs voice through the device. This voice input is recorded in real time and sent to the server. The input is the elderly person's voice, and the output is the recorded voice data. In this step, the voice is captured using the SpeechRecognition library.

[0582] Step 7:

[0583] The server analyzes the received voice data and generates an appropriate response. This process is performed using a generative AI model, and includes voice-to-text conversion, semantic analysis, and response generation. The input is the voice data, and the output is the generated response text.

[0584] Step 8:

[0585] The generated response is converted into speech and played back to the elderly. The device uses the pyttsx3 library to convert text into speech and play it back to the elderly. The input is the response text, and the output is the speech to be played to the elderly.

[0586] Step 9:

[0587] The device collects feedback from the elderly during the interaction. It records the elderly's reactions and evaluations and sends them to the server. The input is the elderly's feedback, and the output is the collected feedback data.

[0588] Step 10:

[0589] The server retrains the generative AI model based on the collected feedback data. In this step, the feedback data is used to fine-tune the model and learn it so that it can be reflected in the next conversation. The input is the feedback data, and the output is the retrained generative AI model.

[0590] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0591] The present invention provides a system that combines a generative AI model, virtual reality (VR) technology, and an emotion engine that recognizes user emotions to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing data related to individuals, and training and optimizing the generative AI model. Furthermore, the system provides an interface for realizing dialogue between the elderly and the generative AI model, recognizes the elderly's voice input, generates and plays a response, and collects feedback from the elderly and uses it to retrain the model. Furthermore, the emotion engine recognizes the user's emotions and adjusts the generative AI model's response based on the emotion.

[0592] Overall system picture

[0593] First, users (family members or friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data can include family memories, anecdotes about specific events, and everyday conversations. The server receives the provided data and performs preprocessing. This preprocessing includes noise reduction for the voice data and normalization for the text data.

[0594] The server then uses the preprocessed data to train a generative AI model, which is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the generative AI model to provide natural and friendly responses to the elderly.

[0595] Building an interface and emotion engine

[0596] The terminal (device used by the elderly) will have a user interface tailored to the elderly. This interface will provide navigation with large buttons, text, and voice guidance that take visual and auditory impairments into consideration. It will also incorporate an emotion engine that will analyze and recognize the emotional state of the elderly from voice input and facial expression data.

[0597] When an elderly person initiates a dialogue with the generative AI model through their device, the device records the elderly person's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. The emotion engine analyzes the elderly person's emotional state and adjusts the content and tone of the response. The generated response is then played back by the device.

[0598] Gathering feedback and retraining the model

[0599] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions for the elderly.

[0600] Specific examples

[0601] Below is a concrete example of how the system actually works.

[0602] When an older person talks about a family trip they took long ago

[0603] Users (family and friends) upload audio and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When an elderly person speaks to the device, saying, "Tell me about your past trips," the device recognizes this and sends the audio data to the server.

[0604] The server analyzes the voice data and generates a response about the family trip. At this time, the emotion engine reads the elderly person's emotions from their tone of voice and facial expressions. If it determines that the elderly person is having fun, for example, the response will be a positive one such as, "On our last trip, we all went to a hot spring together as a family. It was fun!" The generated response is then played back by the device.

[0605] The device then records the elderly person's responses and sends feedback to the server, which uses this feedback to retrain the model and provide more appropriate responses in the next interaction.

[0606] In this way, elderly people can easily use the system at home and it can help prevent and improve dementia. The introduction of the emotion engine makes the dialogue more natural and friendly, further supporting the mental health of the elderly.

[0607] The processing flow will be explained below.

[0608] Step 1:

[0609] Users (family and friends) use a web portal or dedicated app to upload voice and text data related to the elderly, including family memories, anecdotes about specific events, and everyday conversations.

[0610] Step 2:

[0611] The server preprocesses the received data, specifically removing noise from the audio data and normalizing and deleting unnecessary parts from the text data.

[0612] Step 3:

[0613] The server stores the preprocessed data in a database, which is then used to train the generative AI model.

[0614] Step 4:

[0615] The server uses the stored data to train the generative AI model, extracting speech features from the voice data and using natural language processing to extract important keywords from the text data.

[0616] Step 5:

[0617] The server optimizes the trained generative AI model for seniors, adjusting the model parameters based on the seniors' cognitive level, voice characteristics, and individual preferences.

[0618] Step 6:

[0619] The device will create a user interface tailored to the elderly, providing large buttons and text that are visually and auditorily friendly, as well as voice-guided navigation.

[0620] Step 7:

[0621] The device will be equipped with an emotion engine and configured to analyze and recognize the emotional state of the elderly person from voice input and facial expression data.

[0622] Step 8:

[0623] The user (elderly person) initiates a dialogue with the generative AI model through the device. When the elderly person speaks to the device, the device records it in real time and sends it to the server.

[0624] Step 9:

[0625] The server analyzes the received voice data, and the emotion engine estimates the elderly person's emotional state from their tone of voice and facial expressions, and provides this data to the generative AI model.

[0626] Step 10:

[0627] Based on the emotional state estimated by the emotion engine, the generative AI model generates an appropriate response. For example, if the server determines that the elderly person is having fun, it generates a positive response.

[0628] Step 11:

[0629] The server sends the generated response to the terminal, which converts the response into speech and plays it back to the elderly person.

[0630] Step 12:

[0631] The device records the elderly person's reactions and feedback in real time and transmits them to a server.

[0632] Step 13:

[0633] The server uses the feedback collected to retrain the generative AI model and further optimize it for older people, improving the accuracy and performance of the system.

[0634] This allows seniors to enjoy interacting with generative AI models, stimulating their cognitive functions and promoting their mental health. The introduction of an emotion engine makes interactions more natural and friendly.

[0635] Example 2

[0636] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0637] To prevent and improve dementia in the elderly, it is important to provide friendly and appropriate dialogue for each individual elderly person. However, conventional dialogue systems have difficulty providing optimal responses that reflect the emotional state and personal preferences of each elderly person. Furthermore, for elderly people to continue using the system, the dialogue needs to be natural and adapt to their emotions. This is required to maintain and improve the mental health of the elderly.

[0638] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0639] In this invention, the server includes means for collecting personal data provided by a user, means for preprocessing the collected data and performing noise reduction and normalization, means for training a generative AI model based on the preprocessed data, means for optimizing the trained model for the elderly, means for providing an interface for realizing a dialogue between the elderly and the generative AI model, means for recognizing the elderly's voice input, analyzing the input, and generating a response, means for playing the generated response to the elderly, means for collecting feedback from the elderly and using it to retrain the model, an emotion engine for recognizing the elderly's emotional state, and means for adjusting the response of the generative AI model based on the emotion engine. This makes it possible to provide a natural and friendly dialogue that adapts to emotions and improve the mental health of the elderly.

[0640] "User" refers to the entity that provides data, such as an individual, family member, or friend who uses the system.

[0641] "Personal data" refers to information related to an elderly person, such as their lifestyle, memories, tone of voice, upbringing, preferences, prohibitions, dialect, etc.

[0642] "Preprocessing" refers to the process of processing collected data, such as by removing noise and normalizing it, to prepare it in a form that can be used by a generative AI model.

[0643] A "generative AI model" is an artificial intelligence model that generates natural responses based on given data.

[0644] "Training" refers to the process of adjusting the parameters and algorithms of a generative AI model using collected data so that it can generate appropriate responses.

[0645] An "interface" is something that provides an operation screen and input means for a user to interact with a system.

[0646] The "emotion engine" is an engine that analyzes voice input and facial expression data to recognize the emotional state of the elderly.

[0647] "Feedback" refers to data about reactions, opinions, and behaviors obtained from seniors while using the system.

[0648] "Retraining" refers to the process of re-adjusting a generative AI model based on collected feedback data to improve its accuracy and performance.

[0649] "Means for generating a response" refers to the technology or method for analyzing the senior's voice input and generating an appropriate response.

[0650] A "database" is a collection of information that stores data that a generative AI model references when generating responses.

[0651] MODE FOR CARRYING OUT THE INVENTION

[0652] The present invention provides a system that combines a generative AI model, virtual reality (VR) technology, and an emotion engine that recognizes user emotions to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing data related to individuals, and training and optimizing the generative AI model. Furthermore, the system provides an interface for realizing dialogue between the elderly and the generative AI model, recognizes the elderly's voice input, generates and plays a response, and collects feedback from the elderly and uses it to retrain the model. Furthermore, the emotion engine recognizes the user's emotions and adjusts the generative AI model's response based on the emotion.

[0653] Overall system picture

[0654] First, users (family members or friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data can include family memories, anecdotes about specific events, and everyday conversations. The server receives the provided data and performs preprocessing. This preprocessing includes noise reduction for the voice data and normalization for the text data.

[0655] The server then uses the preprocessed data to train a generative AI model, which is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the generative AI model to provide natural and friendly responses to the elderly.

[0656] Building an interface and emotion engine

[0657] The terminal (device used by the elderly) will have a user interface tailored to the elderly. This interface will provide navigation with large buttons, text, and voice guidance that take visual and auditory impairments into consideration. It will also incorporate an emotion engine that will analyze and recognize the emotional state of the elderly from voice input and facial expression data.

[0658] When an elderly person initiates a dialogue with the generative AI model through their device, the device records the elderly person's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. The emotion engine analyzes the elderly person's emotional state and adjusts the content and tone of the response. The generated response is then played back by the device.

[0659] Gathering feedback and retraining the model

[0660] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions for the elderly.

[0661] Specific examples

[0662] Below is a concrete example of how the system actually works.

[0663] When an older person talks about a family trip they took long ago

[0664] Users (family and friends) upload audio and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When an elderly person speaks to the device, saying, "Tell me about your past trips," the device recognizes this and sends the audio data to the server.

[0665] The server analyzes the voice data and generates a response about the family trip. At this time, the emotion engine reads the elderly person's emotions from their tone of voice and facial expressions. If it determines that the elderly person is having fun, for example, the response will be a positive one such as, "On our last trip, we all went to a hot spring together as a family. It was fun!" The generated response is then played back by the device.

[0666] The device then records the elderly person's response and sends feedback to the server. The server uses this feedback to retrain the model and provide a more appropriate response the next time the conversation takes place. In this way, the elderly can easily use the system at home, helping to prevent and improve dementia. The introduction of an emotion engine makes conversations more natural and friendly, further supporting the mental health of the elderly.

[0667] Prompt Sentence Examples

[0668] An elderly person might speak into the device and say, "Tell me about your past travels."

[0669] This allows the system to respond to the user's emotions and provide a more friendly dialogue for the elderly.

[0670] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0671] Step 1: Data provision and collection

[0672] Users provide personal audio and text data through a dedicated web portal or application. For example, they upload stories about family trips or everyday conversations. As input, users send audio and text files to the server. This data is received by the server and saved in local storage. As output, saved audio and text data is generated.

[0673] Step 2: Data Preprocessing

[0674] The server performs preprocessing on the collected data. Specific operations include removing noise from the audio data and converting the audio into text using speech recognition technology. Normalization processing (standardization of notation, removal of unnecessary spaces, etc.) is performed on the text data. This generates preprocessed, clean text data.

[0675] Step 3: Training the generative AI model

[0676] The server trains a generative AI model using the preprocessed data. As input, clean text data is provided to the model. The generative AI model (e.g., GPT-4) is used to learn appropriate responses to specific prompt sentences. As output, the generative AI model is able to generate natural and friendly responses for seniors.

[0677] Step 4: Optimize the model

[0678] The trained model is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences. The elderly person's profile information (cognitive level, preferences, etc.) is used as input. This results in the generation of a generative AI model optimized for the elderly person as the output.

[0679] Step 5: Providing an Interface

[0680] The terminal provides an interface that is easy for seniors to use. It is designed with consideration for visual and auditory impairments, with large buttons, text, and voice-guided navigation. The terminal accepts the operations of the seniors as input. An intuitive and easy-to-operate user interface is provided as output.

[0681] Step 6: Emotion Recognition with the Emotion Engine

[0682] The device collects voice input and facial expression data from the elderly person and uses an emotion engine to analyze and recognize their emotional state. Real-time voice and facial expression data are used as input. Specifically, data is acquired using the device's camera and microphone, and analyzed by the emotion engine. The output is data representing the recognized emotional state.

[0683] Step 7: Dialogue execution and response generation

[0684] When an elderly person speaks to the device, the device records the voice input in real time and sends it to the server. For example, the elderly person might say, "Tell me about your travels in the past." The recorded voice data is used as input. The server analyzes the voice data and generates an appropriate response using a generative AI model. At this time, the emotion engine adjusts the content and tone of the response based on the elderly person's emotional state. The output is a natural and friendly response that adapts to the emotion.

[0685] Step 8: Playing back the response

[0686] The generated response is played back by the device. Specifically, it is played back audibly using the device's speaker. The audio data of the generated response is used as input. The output is an appropriate audio response provided to the elderly person.

[0687] Step 9: Gather feedback

[0688] The device records the elderly person's reactions and feedback during the dialogue process. The input is the elderly person's remarks and changes in facial expressions. Specific operations include collecting data in real time using the device's sensors and analyzing it with an emotion engine. The collected feedback data is generated as output.

[0689] Step 10: Retrain the model

[0690] The server retrains the generative AI model based on the collected feedback. It uses the feedback data as input, allowing the generative AI model to continuously improve its accuracy and performance. The output is a retrained, highly accurate generative AI model.

[0691] Through the above processing steps, the system of the present invention realizes natural and friendly dialogue with the elderly, and can be useful in preventing and improving dementia.

[0692] (Application example 2)

[0693] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0694] When elderly people shop in physical stores, they often have difficulty selecting and using products, or finding the right products. To ensure that elderly people can shop with peace of mind, they need support that can provide appropriate advice and suggestions in real time. Furthermore, to prevent and improve dementia, personalized support using emotion recognition technology is needed.

[0695] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0696] In this invention, the server includes: means for collecting personal data provided by a user; means for preprocessing the collected data and performing noise reduction and normalization; means for training a generative AI model based on the preprocessed data; means for optimizing the trained model for the elderly; means for providing an interface for enabling dialogue between the elderly and the generative AI model; means for recognizing the elderly's voice input, analyzing the input, and generating a response; means for playing the generated response to the elderly; means for collecting feedback from the elderly and using it to retrain the model; means for adjusting the response of the generative AI model based on the elderly's emotions using an emotion engine that recognizes emotions from the elderly's facial expressions and voice; and means for recognizing emotions in real time and providing appropriate product suggestions and advice when the elderly selects products using a head-mounted display. This enables elderly people to receive appropriate emotion-based support in real time when shopping in physical stores.

[0697] "Users" refers to people who use the system, and in this context primarily includes elderly people and their families and friends.

[0698] "Personal data" refers to information about an older adult, including the user's tone of voice, background, preferences, prohibitions, dialect, etc.

[0699] "Means of collection" refers to the methods and technologies used to collect data provided by older adults and their families.

[0700] "Preprocessing" refers to the process of removing noise and normalizing collected data to make it easier to use.

[0701] "Noise reduction" refers to the process of removing unnecessary information from collected data.

[0702] "Normalization" refers to the process of converting collected data into a uniform format.

[0703] A "generative AI model" refers to an artificial intelligence model that is trained based on collected data and generates dialogue with elderly people.

[0704] "Training means" refers to the methods and techniques used to train and optimize generative AI models using collected data.

[0705] "Interface" refers to the point of contact between the user and the system that the elderly person uses to interact with the generative AI model.

[0706] "Means for recognizing voice input" refers to technologies and methods for recognizing and understanding what an older adult is saying.

[0707] "Means for analyzing" refers to techniques or methods for analyzing the senior's voice input and generating an appropriate response.

[0708] "Means for generating a response" refers to a technique or method for generating an appropriate reply based on the analyzed voice input.

[0709] "Means of playback" refers to the technology or method of outputting the generated response in audio or text format to convey it to the elderly.

[0710] "Feedback" refers to the reactions and opinions of the elderly during the dialogue, and is information used to improve the model.

[0711] The "emotion engine" refers to technology that recognizes emotions from the facial expressions and voice of elderly people and adjusts the response of the generative AI model.

[0712] A "head-mounted display" refers to a device worn by elderly people that displays information visually.

[0713] "Product suggestions and advice" refers to appropriate support provided in real time when seniors are choosing products.

[0714] This invention is a system for preventing and improving dementia in the elderly, and is composed of a generative AI model, virtual reality (VR) technology, and an emotion engine. The detailed configuration and operation of the system are described below.

[0715] Overall system configuration

[0716] The system mainly consists of the following components:

[0717] Means of collecting personal data provided by users

[0718] A means of preprocessing the collected data to remove noise and normalize it

[0719] A means of training generative AI models on preprocessed data

[0720] A means of optimizing trained models for older people

[0721] A means of providing an interface for enabling dialogue between elderly people and generative AI models

[0722] A means of recognizing an older adult's voice input and analyzing that input to generate a response

[0723] A means of playing back generated responses to older adults

[0724] A means of collecting feedback from seniors and using it to retrain the model

[0725] A means of adjusting the responses of generative AI models based on the emotions of the elderly, using an emotion engine that recognizes emotions from the elderly's facial expressions and voice.

[0726] A method to recognize emotions in real time and provide appropriate product suggestions and advice to elderly people when they are selecting products using a head-mounted display.

[0727] Program Implementation

[0728] The system program is configured as follows:

[0729] First, the server collects personal data provided by the user, including the elderly person's voice tone, background, preferences, restrictions, dialect, etc. Then, the server preprocesses the collected data, removing noise from the audio data and normalizing the text data.

[0730] Using the pre-processed data, the server trains a generative AI model that is optimized for the cognitive level, vocal characteristics, and individual preferences of seniors.

[0731] When the senior initiates a dialogue, the interface records the senior's voice input in real time and sends it to the server. The server analyzes the voice data and uses a generative AI model to generate an appropriate response. The emotion engine recognizes emotions from the senior's facial expressions and voice, and adjusts the content and tone of the response based on their emotional state.

[0732] The generated responses are then played back to the elderly, and feedback from the elderly is collected and used by the server to retrain the generative AI model.

[0733] Hardware and software used

[0734] Camera-equipped head-mounted display (HMD): Used by elderly people when selecting products in a store, it captures facial expressions and voice in real time.

[0735] Server: Collects user data, pre-processes it, trains generative AI models, analyzes voice input, generates responses, and retrains them.

[0736] Emotion engine: Recognizes emotions from the elderly's facial expressions and voice in real time and uses them to adjust the responses of generative AI models.

[0737] software:

[0738] OpenCV: Acquiring and displaying camera input

[0739] TensorFlow: Running the GPT-2 model

[0740] transformers: Hugging Face GPT-2 tokenizer and model

[0741] EmotionEngine (used as a temporary emotion recognition library)

[0742] Specific examples

[0743] For example, if an elderly person is wearing a head-mounted display and searching for a product in a physical store, the camera captures the elderly person's facial expression, and the emotion engine recognizes from that expression that the elderly person is having trouble. In this case, the generative AI model is prompted to "generate advice about the product that the elderly person is having trouble with." As a result, a response is generated that reads, "This product is a little complicated to use, but it's easy to use if you follow the instructions. Ask a staff member," and this response is played back to the elderly person.

[0744] Example prompt sentence:

[0745] "Generate suggestions for products that seniors enjoy."

[0746] "Generate advice on products that seniors have trouble with."

[0747] In this way, the system allows elderly people to shop in physical stores with peace of mind while also helping to prevent and improve dementia.

[0748] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0749] Step 1:

[0750] Collecting data relating to individuals

[0751] Users provide personal data (tone of voice, background, preferences, restrictions, dialect, etc.) through a dedicated web portal or application, which then inputs the data into the server, which receives it and stores it in a database.

[0752] Step 2:

[0753] Data Preprocessing

[0754] The server performs denoising and normalisation on the received data, which includes denoising audio data and normalising text data. The input is the data provided by the user and the output is the pre-processed, clean data.

[0755] Step 3:

[0756] Training generative AI models

[0757] The server uses the preprocessed data to train a generative AI model. Based on the preprocessed data input as training data, the generative AI model is optimized for elderly people. The output is a trained generative AI model.

[0758] Step 4:

[0759] Providing a dialogue interface with a generative AI model optimized for the elderly

[0760] The elderly person wears a head-mounted display (HMD) as a dialogue interface. The HMD provides a dialogue interface and supports dialogue between the elderly person and the generative AI model through voice input and visual information.

[0761] Step 5:

[0762] Recognition and analysis of elderly voice input

[0763] When an elderly person speaks about a product, the device (HMD) records this voice input in real time and sends it to the server. The server analyzes the voice input and recognizes what the elderly person is saying. The input is the elderly person's voice data, and the output is analyzed text data.

[0764] Step 6:

[0765] Emotion Recognition Using Emotion Engine

[0766] The server receives voice input and facial expression data of the elderly captured by the camera on the HMD. The emotion engine analyzes this data and recognizes the elderly's emotional state. The input is voice data and facial expression data, and the output is recognized emotional information.

[0767] Step 7:

[0768] Response Generation

[0769] The server inputs the recognized text data and emotion information into the generative AI model as a prompt. For example, a prompt such as "Generate suggestions for products that seniors enjoy" is sent. The generative AI model then generates an appropriate response based on this. The input is the prompt text, and the output is the generated response text.

[0770] Step 8:

[0771] Response playback

[0772] The device (HMD) receives the generated response sent from the server and plays it back to the elderly audibly and visually. For example, a message such as "This product is easy to use, please try it" is played back. The input is the response text, and the output is the played back audio message.

[0773] Step 9:

[0774] Collecting feedback

[0775] After receiving the response, the elderly person's reactions and additional comments are recorded by the device and sent to the server, which collects this feedback data and uses it to retrain the generative AI model. The input is the elderly person's reactions and feedback data, and the output is a dataset for retraining.

[0776] Through the above processing steps, the system can assist seniors in their shopping experience and provide appropriate emotion-based responses in real time.

[0777] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0778] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0779] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0780] [Third embodiment]

[0781] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0782] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0783] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0784] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0785] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0786] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0787] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0788] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0789] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0790] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0791] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0792] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0793] The present invention provides a system that combines a generative AI model with virtual reality (VR) technology to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing data related to individuals, and training and optimizing the generative AI model. It also provides an interface for realizing dialogue between the elderly and the generative AI model, recognizing the elderly's voice input, generating and playing back a response, and collecting feedback from the elderly for use in retraining the model.

[0794] Overall system picture

[0795] First, users (family members or friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data can include family memories, anecdotes about specific events, and everyday conversations. The server receives the provided data and performs preprocessing. This preprocessing includes noise reduction for the voice data and normalization for the text data.

[0796] The server then uses the preprocessed data to train a generative AI model, which is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the generative AI model to provide natural and friendly responses to the elderly.

[0797] Building the interface

[0798] The terminals (devices used by the elderly) will have user interfaces tailored to the elderly, providing large buttons and text that are visually and auditorily friendly, as well as voice-guided navigation.

[0799] When an elderly person initiates a dialogue with the generative AI model through the device, the device records the elderly person's voice input in real time and sends it to the server, which analyzes the voice data and generates an appropriate response, which is then played back by the device.

[0800] Gathering feedback and retraining the model

[0801] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions for the elderly.

[0802] Specific examples

[0803] Below is a concrete example of how the system actually works.

[0804] When an older person talks about a family trip they took long ago

[0805] Users (family and friends) upload audio and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When an elderly person speaks to the device, saying, "Tell me about your past trips," the device recognizes this and sends the audio data to the server.

[0806] The server analyzes the voice data and generates a response about a family trip. The generated response is then played back by the device as, "On a previous trip, we all went to a hot spring together as a family." The device then records the elderly person's response and sends feedback to the server. Based on this, the model is retrained to provide a more appropriate response in the next interaction.

[0807] In this way, elderly people can easily use the system at home, helping to prevent and improve dementia. The system provides new stimulation and joy to the lives of elderly people, supports their mental health, and contributes to reducing feelings of loneliness.

[0808] The processing flow will be explained below.

[0809] Step 1:

[0810] Users (family and friends) use a web portal or dedicated app to upload audio and text data related to the elderly, including family memories, anecdotes about specific events, and everyday conversations.

[0811] Step 2:

[0812] The server preprocesses the data it receives. Specifically, it performs noise removal for audio data, normalizes text data, and removes unnecessary parts.

[0813] Step 3:

[0814] The server stores the preprocessed data in a database, which is then used to train the generative AI model.

[0815] Step 4:

[0816] The server uses the stored data to train the generative AI model, extracting speech features from the voice data and using natural language processing to extract important keywords from the text data.

[0817] Step 5:

[0818] The server optimizes the trained generative AI model for seniors, adjusting the model parameters based on the seniors' cognitive level, voice characteristics, and individual preferences.

[0819] Step 6:

[0820] The device will create a user interface tailored to the elderly, providing large buttons and text that are visually and auditorily friendly, as well as voice-guided navigation.

[0821] Step 7:

[0822] The user (elderly person) initiates a dialogue with the generative AI model through the device. When the elderly person speaks to the device, the device records it in real time and sends it to the server.

[0823] Step 8:

[0824] The server analyzes the voice data sent and generates an appropriate response using a generative AI model.

[0825] Step 9:

[0826] The server sends the generated response to the terminal, which converts the response into speech and plays it back to the elderly person.

[0827] Step 10:

[0828] The device records the elderly person's reactions and feedback in real time and transmits them to a server.

[0829] Step 11:

[0830] The server uses the feedback collected to retrain the generative AI model and further optimize it for older people, improving the accuracy and performance of the system.

[0831] Through this process, seniors can enjoy interacting with generative AI models, stimulating their cognitive functions and promoting their mental health.

[0832] Example 1

[0833] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0834] Dialogue systems aimed at preventing and improving dementia in the elderly require mechanisms that are friendly to the elderly and provide appropriate responses. Conventional dialogue systems have difficulty generating appropriate responses based on the cognitive level and preferences of each individual elderly person, making it difficult to achieve natural dialogue in real time in response to elderly voice input. Furthermore, there have been insufficient methods for appropriately collecting feedback from the elderly and using it to retrain models. This has resulted in technical challenges for smoothly introducing dialogue systems into the daily lives of the elderly.

[0835] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0836] In this invention, the server includes a means for collecting personal information provided by the user, a means for preprocessing the collected information, removing unnecessary data, and standardizing the format, and a means for training a generative artificial intelligence model based on the preprocessed information. This makes it possible to provide natural and friendly dialogue responses to each elderly person. Furthermore, the system recognizes the elderly person's voice input, provides an interface for analyzing the input and generating an appropriate response, and outputs the generated response to the elderly person, thereby achieving natural dialogue in real time. Furthermore, responses from the elderly are collected and used to retrain the generative artificial intelligence model, thereby continuously improving the accuracy of the model. In this way, it is possible to provide an effective dialogue system that contributes to the prevention and improvement of dementia in the elderly.

[0837] A "user" is a person or group that provides information related to a particular individual.

[0838] "Information" is a general term for data related to an individual, such as the tone of voice of family members, background, interests, prohibited activities, and local dialects.

[0839] A "collection method" is a method or device used to collect information from a user.

[0840] "Preprocessing" refers to processing carried out to remove unnecessary data from collected information and standardize the format.

[0841] A "generative artificial intelligence model" is a machine learning model that is trained based on collected information and aims to generate natural-sounding dialogue responses.

[0842] A "training tool" refers to a method or technique for optimizing a generative artificial intelligence model for a specific purpose.

[0843] "Elderly optimization means" refers to methods and techniques for adjusting generative artificial intelligence models to provide friendly and natural dialogue responses to seniors.

[0844] "Connection means" refers to interfaces or devices that enable interaction between the elderly person and the generative artificial intelligence model.

[0845] "Voice input" refers to the voice or spoken words produced by the elderly person.

[0846] A "means for generating a response" is a method or technique for analyzing speech input and creating an appropriate reply.

[0847] "Output means" refers to a method or device for communicating the generated response to the elderly person.

[0848] Response collection tools are methods and techniques used to gather responses and feedback from older adults.

[0849] This invention provides a system that combines a generative artificial intelligence model with virtual reality (VR) technology to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing information related to an individual, and training and optimizing the generative artificial intelligence model. Furthermore, the system provides an interface for realizing dialogue between the elderly and the generative artificial intelligence model, recognizes the elderly's voice input, generates and plays back a response, and collects responses from the elderly for use in retraining the model.

[0850] Data collection methods

[0851] Users (elderly family members and friends) provide personal information, such as their family members' voices, backgrounds, interests, prohibited activities, and local dialects, through a dedicated web portal or application. This information includes memories of family trips and everyday conversations. Specifically, users open the application and upload images, audio files, and text. For example, they can upload photos and descriptions of past family trips.

[0852] Data preprocessing methods

[0853] The server receives the provided information and performs preprocessing, which includes denoising the audio data and standardising the format of the text data. Specifically, the server uses the Python library Librosa to denoise the audio data and the NLTK library to correct spelling and check grammar of the text data.

[0854] A means of training and optimizing generative artificial intelligence models

[0855] The server uses the preprocessed information to train a generative artificial intelligence model using machine learning libraries such as TensorFlow and PyTorch. The trained model is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the model to provide natural and friendly responses to the elderly.

[0856] Building the user interface

[0857] Terminals (devices used by the elderly) will have user interfaces tailored to the elderly. These interfaces will include large buttons and text that take visual and hearing impairments into consideration, as well as voice-guided navigation. Specifically, terminals will be designed to take visual impairments into consideration, displaying buttons and text in large fonts and with clear contrast. They will also provide voice assistant functionality to make them easy for the elderly to operate.

[0858] Real-time interaction means

[0859] When an elderly person begins a dialogue with the generative AI model through the device, the device records the elderly person's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. For example, if an elderly person says, "Tell me about your old trips," the device records this and sends the voice data to the server. The server analyzes this voice data and "generates a response about family trips." The generated response is played back by the device as, "On your old trips, we all went to hot springs together as a family."

[0860] A means of gathering feedback and retraining the model

[0861] During the interaction, the device collects the elderly's reactions and feedback and sends it to the server. Based on this feedback, the server retrains the generative AI model. Specifically, the device asks whether the elderly is satisfied with the response and sends the result to the server. The server uses this feedback to update the model and provide more appropriate responses in the next interaction.

[0862] Specific examples

[0863] Below is a concrete example of how the system actually works.

[0864] When an older person talks about a family trip they took long ago

[0865] Users (family and friends) upload voice and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When the elderly person speaks to the device, saying, "Tell me about your old trips," the device recognizes this and sends the voice data to the server. The server analyzes the voice data and "generates a response about your family trip." The generated response is then played back by the device as, "On your old trips, we all went to a hot spring." The device then records the elderly person's response and sends feedback to the server. Based on this, the model is retrained to provide a more appropriate response in the next interaction.

[0866] Prompt Sentence Examples

[0867] "How are you feeling today?"

[0868] "Want to talk about past family trips?"

[0869] "Tell me some memories you have with your grandchildren."

[0870] In this way, the system can activate memories and conversational content of the elderly, helping to prevent and improve dementia. The system provides new stimulation and joy to the lives of the elderly, supporting their mental health and contributing to reducing loneliness.

[0871] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0872] Step 1:

[0873] Data collection

[0874] Users use a dedicated web portal or application to provide information about their family members, such as their voices, backgrounds, interests, prohibited activities, and local dialects, as well as upload photos, audio files, and text data.

[0875] Input: Audio data, text data, image data

[0876] Output: Collected information (stored in a database)

[0877] What happens: A user opens the application and uploads photos and descriptions of past family trips.

[0878] Step 2:

[0879] Data Preprocessing

[0880] The server preprocesses the collected information: first, it uses the Python library Librosa to denoise the audio data, and then it uses the NLTK library to standardize the format of the text data (for example, to perform spelling correction and grammar checks).

[0881] Input: Collected information (output of step 1)

[0882] Output: Preprocessed information (denoised audio data, normalized text data)

[0883] Specific operation: The server receives the audio data and uses Librosa to remove noise. It also processes the text data with NLTK and standardizes the format.

[0884] Step 3:

[0885] Training and optimizing generative AI models

[0886] The server trains a generative artificial intelligence model based on the preprocessed information using machine learning libraries such as TensorFlow and PyTorch, optimizing the model based on the elderly person's cognitive level, voice characteristics, and individual preferences.

[0887] Input: Preprocessed information (output of step 2)

[0888] Output: A trained generative artificial intelligence model

[0889] Specific operation: The server uses preprocessed voice and text data to train a generative artificial intelligence model and optimize it for the elderly.

[0890] Step 4:

[0891] Building the user interface

[0892] The device will have a user interface tailored to seniors, including large buttons and text for visual and hearing impairments, as well as voice-guided navigation.

[0893] Input: Design specifications, needs of the elderly

[0894] Output: User Interface (UI)

[0895] What it does: Your device displays buttons and text in large fonts and with clear contrast, and offers voice assistant functionality.

[0896] Step 5:

[0897] Real-time dialogue

[0898] When an elderly person initiates a dialogue with the generative AI model through the device, the device records the elderly person's voice input in real time and transmits it to the server, which analyzes the voice input and generates an appropriate response.

[0899] Input: Voice input by elderly people

[0900] Output: The generated response

[0901] Specific operation: When an elderly person says, "Tell me about your past travels," the device records this and sends the audio data to the server. The server analyzes the audio data, "generates a response about family trips," and sends it back to the device. The device then plays back the generated response.

[0902] Step 6:

[0903] Gathering feedback and retraining the model

[0904] During the interaction, the device collects the elderly's reactions and feedback and sends it to the server, which uses this feedback to retrain the generative artificial intelligence model.

[0905] Input: Older adults' reactions and feedback

[0906] Output: A retrained generative AI model

[0907] How it works: The device checks whether the elderly person is satisfied with the response and sends the result to the server. The server uses the feedback to update the model and provide a more appropriate response in the next interaction.

[0908] (Application example 1)

[0909] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0910] There is a need to provide an environment where elderly people can easily select and order meals while staying safe at home. It is also necessary to introduce dialogue systems that contribute to the prevention and improvement of dementia. However, existing systems have not been able to adequately respond to individual needs by taking into account the preferences and history of elderly people. Furthermore, in order for dialogue systems to generate natural and friendly responses to elderly voice input, advanced processing in speech recognition technology and generative AI models is required, posing a challenge to practical implementation.

[0911] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0912] In this invention, the server includes: means for collecting personal data provided by a user; means for preprocessing the collected data to remove noise and normalize it; means for training a generative AI model based on the preprocessed data; means for optimizing the trained model for the elderly; means for providing an interface for enabling dialogue between the elderly and the generative AI model; means for recognizing the elderly's voice input, analyzing the input, and generating a response; means for playing the generated response to the elderly; means for collecting feedback from the elderly and using it to retrain the model; means for presenting a menu based on the elderly's past eating history and preferences; and means for ordering meals based on the presented menu. This enables personalized meal suggestions and ordering based on the elderly's own preferences and history, and also provides an interactive system that is effective in preventing and improving dementia.

[0913] "Personal data provided by the user" refers to information related to the elderly person provided by the user, including the family member's tone of voice, upbringing, preferences, things they don't like, dialect, etc.

[0914] "Collected data" refers to audio data, text data, and the like that are uniquely associated with the senior, including information provided by the user.

[0915] "Preprocessing" refers to processing performed on collected data, such as noise removal and data normalization.

[0916] A "generative AI model" is an artificial intelligence model that has the ability to learn from large datasets and respond to specific tasks.

[0917] "Methods for optimizing for the elderly" refer to techniques for adjusting generative AI models based on the cognitive level, voice characteristics, and individual preferences of the elderly.

[0918] The "interface" is a means of providing an operating screen and voice guidance for elderly people to interact with the generated AI model.

[0919] "Means for recognizing a senior's voice input and analyzing the input to generate a response" refers to the overall process of analyzing a senior's voice and generating an appropriate response.

[0920] "Means for playing the generated response to the elderly" refers to a method for letting the elderly hear the voice response generated by the generative AI model.

[0921] "Means of collecting feedback from seniors and using it to retrain the model" refers to the process of improving the performance of generative AI models based on reactions and opinions obtained from interactions with seniors.

[0922] "Means for presenting menus based on past eating history and preferences" refers to a function that presents appropriate menus based on the elderly person's past eating history and preferences.

[0923] "Means for ordering food based on a presented menu" refers to the functionality of making selections from a presented menu and actually completing a food order.

[0924] This invention aims to prevent and improve dementia in the elderly using the following systems and processes.

[0925] Overall system picture

[0926] First, users (family and friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data includes past dietary history and preferences, the family member's voice tone, upbringing, things they don't like, dialects, etc. The server receives this data and performs noise reduction and data normalization.

[0927] The server then uses the preprocessed data to train a generative AI model, optimizing it based on the elderly person's cognitive level, voice characteristics, and individual preferences, allowing the trained generative AI model to generate natural and friendly responses in real time.

[0928] Building the interface

[0929] The terminal (the device used by the elderly) provides a user interface tailored to the elderly. This interface features large buttons, text, and voice-guided navigation that take visual and auditory impairments into consideration. When the elderly initiates a dialogue with the generative AI model through the terminal, the terminal records the elderly's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. This response is then played back by the terminal.

[0930] Food ordering function

[0931] A distinctive feature of this system is its ability to present menus based on the elderly person's past eating history and preferences. When the elderly person asks, "What's on the menu today?", the device recognizes this and sends the voice data to the server. The server then generates an optimal response based on this and suggests dishes suitable for the elderly. In this way, the application assists the elderly in the process of selecting and ordering meals.

[0932] Gathering feedback and retraining the model

[0933] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions and services for the elderly.

[0934] Hardware and software used

[0935] The system uses the following main hardware and software:

[0936] Smartphones / head-mounted displays: Devices used by the elderly.

[0937] TensorFlow / Keras: Used for training and inference of generative AI models.

[0938] SpeechRecognition: Provides speech recognition technology.

[0939] pyttsx3: Used to speak the generated response.

[0940] Adding specific examples

[0941] For example, if a user (an elderly person) says, "What's today's recommended menu?", the system will generate the following response, taking into account the elderly person's eating history and preferences:

[0942] "Today, I recommend the grilled eel and soup set. Would you like to order that?"

[0943] Prompt Sentence Examples

[0944] The prompt should be in the following format:

[0945] Senior citizen: "What's on the menu today?"

[0946] Generative AI: "Based on your past eating history and preferences, the recommended menu is a set of grilled eel and clear soup. Would you like to order that?"

[0947] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0948] Step 1:

[0949] Users (family and friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This includes past meal history, the family member's voice tone, upbringing, preferences, things they don't like, dialects, etc. This input data is sent to the server. The main purpose of this step is to collect the input data (voice and text data). The output is the initial data before it is preprocessed.

[0950] Step 2:

[0951] The server preprocesses the received data related to individuals, specifically by removing noise and normalizing the data. This process uses the acquired voice and text data as input and processes it through noise filtering, tokenization, and other processes. This results in clean data that the generative AI model can accurately learn from.

[0952] Step 3:

[0953] The server trains a generative AI model based on the preprocessed data. The server uses deep learning libraries such as TensorFlow and Keras to generate a model with voice response capabilities based on the input data. The output of this step is a highly specialized generative AI model.

[0954] Step 4:

[0955] The trained model is optimized for seniors. The server adjusts the model parameters based on the seniors' cognitive level, voice characteristics, and individual preferences. In this process, the generative AI model obtained in the previous step is further tuned so that it can generate responses appropriate for seniors. The output is a generative AI model optimized for seniors.

[0956] Step 5:

[0957] The device provides an interface for dialogue between the elderly person and the generative AI model. It displays an interface that takes visual and auditory impairments into consideration, including large buttons, text, and audio guides. The input is the elderly person's operations, and the output is a GUI (graphical user interface) for dialogue.

[0958] Step 6:

[0959] The elderly person inputs voice through the device. This voice input is recorded in real time and sent to the server. The input is the elderly person's voice, and the output is the recorded voice data. In this step, the voice is captured using the SpeechRecognition library.

[0960] Step 7:

[0961] The server analyzes the received voice data and generates an appropriate response. This process is performed using a generative AI model, and includes voice-to-text conversion, semantic analysis, and response generation. The input is the voice data, and the output is the generated response text.

[0962] Step 8:

[0963] The generated response is converted into speech and played back to the elderly. The device uses the pyttsx3 library to convert text into speech and play it back to the elderly. The input is the response text, and the output is the speech to be played to the elderly.

[0964] Step 9:

[0965] The device collects feedback from the elderly during the interaction. It records the elderly's reactions and evaluations and sends them to the server. The input is the elderly's feedback, and the output is the collected feedback data.

[0966] Step 10:

[0967] The server retrains the generative AI model based on the collected feedback data. In this step, the feedback data is used to fine-tune the model and learn it so that it can be reflected in the next conversation. The input is the feedback data, and the output is the retrained generative AI model.

[0968] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0969] The present invention provides a system that combines a generative AI model, virtual reality (VR) technology, and an emotion engine that recognizes user emotions to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing data related to individuals, and training and optimizing the generative AI model. Furthermore, the system provides an interface for realizing dialogue between the elderly and the generative AI model, recognizes the elderly's voice input, generates and plays a response, and collects feedback from the elderly and uses it to retrain the model. Furthermore, the emotion engine recognizes the user's emotions and adjusts the generative AI model's response based on the emotion.

[0970] Overall system picture

[0971] First, users (family members or friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data can include family memories, anecdotes about specific events, and everyday conversations. The server receives the provided data and performs preprocessing. This preprocessing includes noise reduction for the voice data and normalization for the text data.

[0972] The server then uses the preprocessed data to train a generative AI model, which is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the generative AI model to provide natural and friendly responses to the elderly.

[0973] Building an interface and emotion engine

[0974] The terminal (device used by the elderly) will have a user interface tailored to the elderly. This interface will provide navigation with large buttons, text, and voice guidance that take visual and auditory impairments into consideration. It will also incorporate an emotion engine that will analyze and recognize the emotional state of the elderly from voice input and facial expression data.

[0975] When an elderly person initiates a dialogue with the generative AI model through their device, the device records the elderly person's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. The emotion engine analyzes the elderly person's emotional state and adjusts the content and tone of the response. The generated response is then played back by the device.

[0976] Gathering feedback and retraining the model

[0977] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions for the elderly.

[0978] Specific examples

[0979] Below is a concrete example of how the system actually works.

[0980] When an older person talks about a family trip they took long ago

[0981] Users (family and friends) upload audio and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When an elderly person speaks to the device, saying, "Tell me about your past trips," the device recognizes this and sends the audio data to the server.

[0982] The server analyzes the voice data and generates a response about the family trip. At this time, the emotion engine reads the elderly person's emotions from their tone of voice and facial expressions. If it determines that the elderly person is having fun, for example, the response will be a positive one such as, "On our last trip, we all went to a hot spring together as a family. It was fun!" The generated response is then played back by the device.

[0983] The device then records the elderly person's responses and sends feedback to the server, which uses this feedback to retrain the model and provide more appropriate responses in the next interaction.

[0984] In this way, elderly people can easily use the system at home and it can help prevent and improve dementia. The introduction of the emotion engine makes the dialogue more natural and friendly, further supporting the mental health of the elderly.

[0985] The processing flow will be explained below.

[0986] Step 1:

[0987] Users (family and friends) use a web portal or dedicated app to upload voice and text data related to the elderly, including family memories, anecdotes about specific events, and everyday conversations.

[0988] Step 2:

[0989] The server preprocesses the received data, specifically removing noise from the audio data and normalizing and deleting unnecessary parts from the text data.

[0990] Step 3:

[0991] The server stores the preprocessed data in a database, which is then used to train the generative AI model.

[0992] Step 4:

[0993] The server uses the stored data to train the generative AI model, extracting speech features from the voice data and using natural language processing to extract important keywords from the text data.

[0994] Step 5:

[0995] The server optimizes the trained generative AI model for seniors, adjusting the model parameters based on the seniors' cognitive level, voice characteristics, and individual preferences.

[0996] Step 6:

[0997] The device will create a user interface tailored to the elderly, providing large buttons and text that are visually and auditorily friendly, as well as voice-guided navigation.

[0998] Step 7:

[0999] The device will be equipped with an emotion engine and configured to analyze and recognize the emotional state of the elderly person from voice input and facial expression data.

[1000] Step 8:

[1001] The user (elderly person) initiates a dialogue with the generative AI model through the device. When the elderly person speaks to the device, the device records it in real time and sends it to the server.

[1002] Step 9:

[1003] The server analyzes the received voice data, and the emotion engine estimates the elderly person's emotional state from their tone of voice and facial expressions, and provides this data to the generative AI model.

[1004] Step 10:

[1005] Based on the emotional state estimated by the emotion engine, the generative AI model generates an appropriate response. For example, if the server determines that the elderly person is having fun, it generates a positive response.

[1006] Step 11:

[1007] The server sends the generated response to the terminal, which converts the response into speech and plays it back to the elderly person.

[1008] Step 12:

[1009] The device records the elderly person's reactions and feedback in real time and transmits them to a server.

[1010] Step 13:

[1011] The server uses the feedback collected to retrain the generative AI model and further optimize it for older people, improving the accuracy and performance of the system.

[1012] This allows seniors to enjoy interacting with generative AI models, stimulating their cognitive functions and promoting their mental health. The introduction of an emotion engine makes interactions more natural and friendly.

[1013] Example 2

[1014] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1015] To prevent and improve dementia in the elderly, it is important to provide friendly and appropriate dialogue for each individual elderly person. However, conventional dialogue systems have difficulty providing optimal responses that reflect the emotional state and personal preferences of each elderly person. Furthermore, for elderly people to continue using the system, the dialogue needs to be natural and adapt to their emotions. This is required to maintain and improve the mental health of the elderly.

[1016] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1017] In this invention, the server includes means for collecting personal data provided by a user, means for preprocessing the collected data and performing noise reduction and normalization, means for training a generative AI model based on the preprocessed data, means for optimizing the trained model for the elderly, means for providing an interface for realizing a dialogue between the elderly and the generative AI model, means for recognizing the elderly's voice input, analyzing the input, and generating a response, means for playing the generated response to the elderly, means for collecting feedback from the elderly and using it to retrain the model, an emotion engine for recognizing the elderly's emotional state, and means for adjusting the response of the generative AI model based on the emotion engine. This makes it possible to provide a natural and friendly dialogue that adapts to emotions and improve the mental health of the elderly.

[1018] "User" refers to the entity that provides data, such as an individual, family member, or friend who uses the system.

[1019] "Personal data" refers to information related to an elderly person, such as their lifestyle, memories, tone of voice, upbringing, preferences, prohibitions, dialect, etc.

[1020] "Preprocessing" refers to the process of processing collected data, such as by removing noise and normalizing it, to prepare it in a form that can be used by a generative AI model.

[1021] A "generative AI model" is an artificial intelligence model that generates natural responses based on given data.

[1022] "Training" refers to the process of adjusting the parameters and algorithms of a generative AI model using collected data so that it can generate appropriate responses.

[1023] An "interface" is something that provides an operation screen and input means for a user to interact with a system.

[1024] The "emotion engine" is an engine that analyzes voice input and facial expression data to recognize the emotional state of the elderly.

[1025] "Feedback" refers to data about reactions, opinions, and behaviors obtained from seniors while using the system.

[1026] "Retraining" refers to the process of re-adjusting a generative AI model based on collected feedback data to improve its accuracy and performance.

[1027] "Means for generating a response" refers to the technology or method for analyzing the senior's voice input and generating an appropriate response.

[1028] A "database" is a collection of information that stores data that a generative AI model references when generating responses.

[1029] MODE FOR CARRYING OUT THE INVENTION

[1030] The present invention provides a system that combines a generative AI model, virtual reality (VR) technology, and an emotion engine that recognizes user emotions to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing data related to individuals, and training and optimizing the generative AI model. Furthermore, the system provides an interface for realizing dialogue between the elderly and the generative AI model, recognizes the elderly's voice input, generates and plays a response, and collects feedback from the elderly and uses it to retrain the model. Furthermore, the emotion engine recognizes the user's emotions and adjusts the generative AI model's response based on the emotion.

[1031] Overall system picture

[1032] First, users (family members or friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data can include family memories, anecdotes about specific events, and everyday conversations. The server receives the provided data and performs preprocessing. This preprocessing includes noise reduction for the voice data and normalization for the text data.

[1033] The server then uses the preprocessed data to train a generative AI model, which is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the generative AI model to provide natural and friendly responses to the elderly.

[1034] Building an interface and emotion engine

[1035] The terminal (device used by the elderly) will have a user interface tailored to the elderly. This interface will provide navigation with large buttons, text, and voice guidance that take visual and auditory impairments into consideration. It will also incorporate an emotion engine that will analyze and recognize the emotional state of the elderly from voice input and facial expression data.

[1036] When an elderly person initiates a dialogue with the generative AI model through their device, the device records the elderly person's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. The emotion engine analyzes the elderly person's emotional state and adjusts the content and tone of the response. The generated response is then played back by the device.

[1037] Gathering feedback and retraining the model

[1038] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions for the elderly.

[1039] Specific examples

[1040] Below is a concrete example of how the system actually works.

[1041] When an older person talks about a family trip they took long ago

[1042] Users (family and friends) upload audio and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When an elderly person speaks to the device, saying, "Tell me about your past trips," the device recognizes this and sends the audio data to the server.

[1043] The server analyzes the voice data and generates a response about the family trip. At this time, the emotion engine reads the elderly person's emotions from their tone of voice and facial expressions. If it determines that the elderly person is having fun, for example, the response will be a positive one such as, "On our last trip, we all went to a hot spring together as a family. It was fun!" The generated response is then played back by the device.

[1044] The device then records the elderly person's response and sends feedback to the server. The server uses this feedback to retrain the model and provide a more appropriate response the next time the conversation takes place. In this way, the elderly can easily use the system at home, helping to prevent and improve dementia. The introduction of an emotion engine makes conversations more natural and friendly, further supporting the mental health of the elderly.

[1045] Prompt Sentence Examples

[1046] An elderly person might speak into the device and say, "Tell me about your past travels."

[1047] This allows the system to respond to the user's emotions and provide a more friendly dialogue for the elderly.

[1048] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1049] Step 1: Data provision and collection

[1050] Users provide personal audio and text data through a dedicated web portal or application. For example, they upload stories about family trips or everyday conversations. As input, users send audio and text files to the server. This data is received by the server and saved in local storage. As output, saved audio and text data is generated.

[1051] Step 2: Data Preprocessing

[1052] The server performs preprocessing on the collected data. Specific operations include removing noise from the audio data and converting the audio into text using speech recognition technology. Normalization processing (standardization of notation, removal of unnecessary spaces, etc.) is performed on the text data. This generates preprocessed, clean text data.

[1053] Step 3: Training the generative AI model

[1054] The server trains a generative AI model using the preprocessed data. As input, clean text data is provided to the model. The generative AI model (e.g., GPT-4) is used to learn appropriate responses to specific prompt sentences. As output, the generative AI model is able to generate natural and friendly responses for seniors.

[1055] Step 4: Optimize the model

[1056] The trained model is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences. The elderly person's profile information (cognitive level, preferences, etc.) is used as input. This results in the generation of a generative AI model optimized for the elderly person as the output.

[1057] Step 5: Providing an Interface

[1058] The terminal provides an interface that is easy for seniors to use. It is designed with consideration for visual and auditory impairments, with large buttons, text, and voice-guided navigation. The terminal accepts the operations of the seniors as input. An intuitive and easy-to-operate user interface is provided as output.

[1059] Step 6: Emotion Recognition with the Emotion Engine

[1060] The device collects voice input and facial expression data from the elderly person and uses an emotion engine to analyze and recognize their emotional state. Real-time voice and facial expression data are used as input. Specifically, data is acquired using the device's camera and microphone, and analyzed by the emotion engine. The output is data representing the recognized emotional state.

[1061] Step 7: Dialogue execution and response generation

[1062] When an elderly person speaks to the device, the device records the voice input in real time and sends it to the server. For example, the elderly person might say, "Tell me about your travels in the past." The recorded voice data is used as input. The server analyzes the voice data and generates an appropriate response using a generative AI model. At this time, the emotion engine adjusts the content and tone of the response based on the elderly person's emotional state. The output is a natural and friendly response that adapts to the emotion.

[1063] Step 8: Playing back the response

[1064] The generated response is played back by the device. Specifically, it is played back audibly using the device's speaker. The audio data of the generated response is used as input. The output is an appropriate audio response provided to the elderly person.

[1065] Step 9: Gather feedback

[1066] The device records the elderly person's reactions and feedback during the dialogue process. The input is the elderly person's remarks and changes in facial expressions. Specific operations include collecting data in real time using the device's sensors and analyzing it with an emotion engine. The collected feedback data is generated as output.

[1067] Step 10: Retrain the model

[1068] The server retrains the generative AI model based on the collected feedback. It uses the feedback data as input, allowing the generative AI model to continuously improve its accuracy and performance. The output is a retrained, highly accurate generative AI model.

[1069] Through the above processing steps, the system of the present invention realizes natural and friendly dialogue with the elderly, and can be useful in preventing and improving dementia.

[1070] (Application example 2)

[1071] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1072] When elderly people shop in physical stores, they often have difficulty selecting and using products, or finding the right products. To ensure that elderly people can shop with peace of mind, they need support that can provide appropriate advice and suggestions in real time. Furthermore, to prevent and improve dementia, personalized support using emotion recognition technology is needed.

[1073] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1074] In this invention, the server includes: means for collecting personal data provided by a user; means for preprocessing the collected data and performing noise reduction and normalization; means for training a generative AI model based on the preprocessed data; means for optimizing the trained model for the elderly; means for providing an interface for enabling dialogue between the elderly and the generative AI model; means for recognizing the elderly's voice input, analyzing the input, and generating a response; means for playing the generated response to the elderly; means for collecting feedback from the elderly and using it to retrain the model; means for adjusting the response of the generative AI model based on the elderly's emotions using an emotion engine that recognizes emotions from the elderly's facial expressions and voice; and means for recognizing emotions in real time and providing appropriate product suggestions and advice when the elderly selects products using a head-mounted display. This enables elderly people to receive appropriate emotion-based support in real time when shopping in physical stores.

[1075] "Users" refers to people who use the system, and in this context primarily includes elderly people and their families and friends.

[1076] "Personal data" refers to information about an older adult, including the user's tone of voice, background, preferences, prohibitions, dialect, etc.

[1077] "Means of collection" refers to the methods and technologies used to collect data provided by older adults and their families.

[1078] "Preprocessing" refers to the process of removing noise and normalizing collected data to make it easier to use.

[1079] "Noise reduction" refers to the process of removing unnecessary information from collected data.

[1080] "Normalization" refers to the process of converting collected data into a uniform format.

[1081] A "generative AI model" refers to an artificial intelligence model that is trained based on collected data and generates dialogue with elderly people.

[1082] "Training means" refers to the methods and techniques used to train and optimize generative AI models using collected data.

[1083] "Interface" refers to the point of contact between the user and the system that the elderly person uses to interact with the generative AI model.

[1084] "Means for recognizing voice input" refers to technologies and methods for recognizing and understanding what an older adult is saying.

[1085] "Means for analyzing" refers to techniques or methods for analyzing the senior's voice input and generating an appropriate response.

[1086] "Means for generating a response" refers to a technique or method for generating an appropriate reply based on the analyzed voice input.

[1087] "Means of playback" refers to the technology or method of outputting the generated response in audio or text format to convey it to the elderly.

[1088] "Feedback" refers to the reactions and opinions of the elderly during the dialogue, and is information used to improve the model.

[1089] The "emotion engine" refers to technology that recognizes emotions from the facial expressions and voice of elderly people and adjusts the response of the generative AI model.

[1090] A "head-mounted display" refers to a device worn by elderly people that displays information visually.

[1091] "Product suggestions and advice" refers to appropriate support provided in real time when seniors are choosing products.

[1092] This invention is a system for preventing and improving dementia in the elderly, and is composed of a generative AI model, virtual reality (VR) technology, and an emotion engine. The detailed configuration and operation of the system are described below.

[1093] Overall system configuration

[1094] The system mainly consists of the following components:

[1095] Means of collecting personal data provided by users

[1096] A means of preprocessing the collected data to remove noise and normalize it

[1097] A means of training generative AI models on preprocessed data

[1098] A means of optimizing trained models for older people

[1099] A means of providing an interface for enabling dialogue between elderly people and generative AI models

[1100] A means of recognizing an older adult's voice input and analyzing that input to generate a response

[1101] A means of playing back generated responses to older adults

[1102] A means of collecting feedback from seniors and using it to retrain the model

[1103] A means of adjusting the responses of generative AI models based on the emotions of the elderly, using an emotion engine that recognizes emotions from the elderly's facial expressions and voice.

[1104] A method to recognize emotions in real time and provide appropriate product suggestions and advice to elderly people when they are selecting products using a head-mounted display.

[1105] Program Implementation

[1106] The system program is configured as follows:

[1107] First, the server collects personal data provided by the user, including the elderly person's voice tone, background, preferences, restrictions, dialect, etc. Then, the server preprocesses the collected data, removing noise from the audio data and normalizing the text data.

[1108] Using the pre-processed data, the server trains a generative AI model that is optimized for the cognitive level, vocal characteristics, and individual preferences of seniors.

[1109] When the senior initiates a dialogue, the interface records the senior's voice input in real time and sends it to the server. The server analyzes the voice data and uses a generative AI model to generate an appropriate response. The emotion engine recognizes emotions from the senior's facial expressions and voice, and adjusts the content and tone of the response based on their emotional state.

[1110] The generated responses are then played back to the elderly, and feedback from the elderly is collected and used by the server to retrain the generative AI model.

[1111] Hardware and software used

[1112] Camera-equipped head-mounted display (HMD): Used by elderly people when selecting products in a store, it captures facial expressions and voice in real time.

[1113] Server: Collects user data, pre-processes it, trains generative AI models, analyzes voice input, generates responses, and retrains them.

[1114] Emotion engine: Recognizes emotions from the elderly's facial expressions and voice in real time and uses them to adjust the responses of generative AI models.

[1115] software:

[1116] OpenCV: Acquiring and displaying camera input

[1117] TensorFlow: Running the GPT-2 model

[1118] transformers: Hugging Face GPT-2 tokenizer and model

[1119] EmotionEngine (used as a temporary emotion recognition library)

[1120] Specific examples

[1121] For example, if an elderly person is wearing a head-mounted display and searching for a product in a physical store, the camera captures the elderly person's facial expression, and the emotion engine recognizes from that expression that the elderly person is having trouble. In this case, the generative AI model is prompted to "generate advice about the product that the elderly person is having trouble with." As a result, a response is generated that reads, "This product is a little complicated to use, but it's easy to use if you follow the instructions. Ask a staff member," and this response is played back to the elderly person.

[1122] Example prompt sentence:

[1123] "Generate suggestions for products that seniors enjoy."

[1124] "Generate advice on products that seniors have trouble with."

[1125] In this way, the system allows elderly people to shop in physical stores with peace of mind while also helping to prevent and improve dementia.

[1126] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1127] Step 1:

[1128] Collecting data relating to individuals

[1129] Users provide personal data (tone of voice, background, preferences, restrictions, dialect, etc.) through a dedicated web portal or application, which then inputs the data into the server, which receives it and stores it in a database.

[1130] Step 2:

[1131] Data Preprocessing

[1132] The server performs denoising and normalisation on the received data, which includes denoising audio data and normalising text data. The input is the data provided by the user and the output is the pre-processed, clean data.

[1133] Step 3:

[1134] Training generative AI models

[1135] The server uses the preprocessed data to train a generative AI model. Based on the preprocessed data input as training data, the generative AI model is optimized for elderly people. The output is a trained generative AI model.

[1136] Step 4:

[1137] Providing a dialogue interface with a generative AI model optimized for the elderly

[1138] The elderly person wears a head-mounted display (HMD) as a dialogue interface. The HMD provides a dialogue interface and supports dialogue between the elderly person and the generative AI model through voice input and visual information.

[1139] Step 5:

[1140] Recognition and analysis of elderly voice input

[1141] When an elderly person speaks about a product, the device (HMD) records this voice input in real time and sends it to the server. The server analyzes the voice input and recognizes what the elderly person is saying. The input is the elderly person's voice data, and the output is analyzed text data.

[1142] Step 6:

[1143] Emotion Recognition Using Emotion Engine

[1144] The server receives voice input and facial expression data of the elderly captured by the camera on the HMD. The emotion engine analyzes this data and recognizes the elderly's emotional state. The input is voice data and facial expression data, and the output is recognized emotional information.

[1145] Step 7:

[1146] Response Generation

[1147] The server inputs the recognized text data and emotion information into the generative AI model as a prompt. For example, a prompt such as "Generate suggestions for products that seniors enjoy" is sent. The generative AI model then generates an appropriate response based on this. The input is the prompt text, and the output is the generated response text.

[1148] Step 8:

[1149] Response playback

[1150] The device (HMD) receives the generated response sent from the server and plays it back to the elderly audibly and visually. For example, a message such as "This product is easy to use, please try it" is played back. The input is the response text, and the output is the played back audio message.

[1151] Step 9:

[1152] Collecting feedback

[1153] After receiving the response, the elderly person's reactions and additional comments are recorded by the device and sent to the server, which collects this feedback data and uses it to retrain the generative AI model. The input is the elderly person's reactions and feedback data, and the output is a dataset for retraining.

[1154] Through the above processing steps, the system can assist seniors in their shopping experience and provide appropriate emotion-based responses in real time.

[1155] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1156] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1157] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1158] [Fourth embodiment]

[1159] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1160] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1161] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1162] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1163] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1164] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1165] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1166] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1167] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1168] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1169] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1170] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1171] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1172] The present invention provides a system that combines a generative AI model with virtual reality (VR) technology to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing data related to individuals, and training and optimizing the generative AI model. It also provides an interface for realizing dialogue between the elderly and the generative AI model, recognizing the elderly's voice input, generating and playing back a response, and collecting feedback from the elderly for use in retraining the model.

[1173] Overall system picture

[1174] First, users (family members or friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data can include family memories, anecdotes about specific events, and everyday conversations. The server receives the provided data and performs preprocessing. This preprocessing includes noise reduction for the voice data and normalization for the text data.

[1175] The server then uses the preprocessed data to train a generative AI model, which is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the generative AI model to provide natural and friendly responses to the elderly.

[1176] Building the interface

[1177] The terminals (devices used by the elderly) will have user interfaces tailored to the elderly, providing large buttons and text that are visually and auditorily friendly, as well as voice-guided navigation.

[1178] When an elderly person initiates a dialogue with the generative AI model through the device, the device records the elderly person's voice input in real time and sends it to the server, which analyzes the voice data and generates an appropriate response, which is then played back by the device.

[1179] Gathering feedback and retraining the model

[1180] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions for the elderly.

[1181] Specific examples

[1182] Below is a concrete example of how the system actually works.

[1183] When an older person talks about a family trip they took long ago

[1184] Users (family and friends) upload audio and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When an elderly person speaks to the device, saying, "Tell me about your past trips," the device recognizes this and sends the audio data to the server.

[1185] The server analyzes the voice data and generates a response about a family trip. The generated response is then played back by the device as, "On a previous trip, we all went to a hot spring together as a family." The device then records the elderly person's response and sends feedback to the server. Based on this, the model is retrained to provide a more appropriate response in the next interaction.

[1186] In this way, elderly people can easily use the system at home, helping to prevent and improve dementia. The system provides new stimulation and joy to the lives of elderly people, supports their mental health, and contributes to reducing feelings of loneliness.

[1187] The processing flow will be explained below.

[1188] Step 1:

[1189] Users (family and friends) use a web portal or dedicated app to upload audio and text data related to the elderly, including family memories, anecdotes about specific events, and everyday conversations.

[1190] Step 2:

[1191] The server preprocesses the data it receives. Specifically, it performs noise removal for audio data, normalizes text data, and removes unnecessary parts.

[1192] Step 3:

[1193] The server stores the preprocessed data in a database, which is then used to train the generative AI model.

[1194] Step 4:

[1195] The server uses the stored data to train the generative AI model, extracting speech features from the voice data and using natural language processing to extract important keywords from the text data.

[1196] Step 5:

[1197] The server optimizes the trained generative AI model for seniors, adjusting the model parameters based on the seniors' cognitive level, voice characteristics, and individual preferences.

[1198] Step 6:

[1199] The device will create a user interface tailored to the elderly, providing large buttons and text that are visually and auditorily friendly, as well as voice-guided navigation.

[1200] Step 7:

[1201] The user (elderly person) initiates a dialogue with the generative AI model through the device. When the elderly person speaks to the device, the device records it in real time and sends it to the server.

[1202] Step 8:

[1203] The server analyzes the voice data sent and generates an appropriate response using a generative AI model.

[1204] Step 9:

[1205] The server sends the generated response to the terminal, which converts the response into speech and plays it back to the elderly person.

[1206] Step 10:

[1207] The device records the elderly person's reactions and feedback in real time and transmits them to a server.

[1208] Step 11:

[1209] The server uses the feedback collected to retrain the generative AI model and further optimize it for older people, improving the accuracy and performance of the system.

[1210] Through this process, seniors can enjoy interacting with generative AI models, stimulating their cognitive functions and promoting their mental health.

[1211] Example 1

[1212] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1213] Dialogue systems aimed at preventing and improving dementia in the elderly require mechanisms that are friendly to the elderly and provide appropriate responses. Conventional dialogue systems have difficulty generating appropriate responses based on the cognitive level and preferences of each individual elderly person, making it difficult to achieve natural dialogue in real time in response to elderly voice input. Furthermore, there have been insufficient methods for appropriately collecting feedback from the elderly and using it to retrain models. This has resulted in technical challenges for smoothly introducing dialogue systems into the daily lives of the elderly.

[1214] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1215] In this invention, the server includes a means for collecting personal information provided by the user, a means for preprocessing the collected information, removing unnecessary data, and standardizing the format, and a means for training a generative artificial intelligence model based on the preprocessed information. This makes it possible to provide natural and friendly dialogue responses to each elderly person. Furthermore, the system recognizes the elderly person's voice input, provides an interface for analyzing the input and generating an appropriate response, and outputs the generated response to the elderly person, thereby achieving natural dialogue in real time. Furthermore, responses from the elderly are collected and used to retrain the generative artificial intelligence model, thereby continuously improving the accuracy of the model. In this way, it is possible to provide an effective dialogue system that contributes to the prevention and improvement of dementia in the elderly.

[1216] A "user" is a person or group that provides information related to a particular individual.

[1217] "Information" is a general term for data related to an individual, such as the tone of voice of family members, background, interests, prohibited activities, and local dialects.

[1218] A "collection method" is a method or device used to collect information from a user.

[1219] "Preprocessing" refers to processing carried out to remove unnecessary data from collected information and standardize the format.

[1220] A "generative artificial intelligence model" is a machine learning model that is trained based on collected information and aims to generate natural-sounding dialogue responses.

[1221] A "training tool" refers to a method or technique for optimizing a generative artificial intelligence model for a specific purpose.

[1222] "Elderly optimization means" refers to methods and techniques for adjusting generative artificial intelligence models to provide friendly and natural dialogue responses to seniors.

[1223] "Connection means" refers to interfaces or devices that enable interaction between the elderly person and the generative artificial intelligence model.

[1224] "Voice input" refers to the voice or spoken words produced by the elderly person.

[1225] A "means for generating a response" is a method or technique for analyzing speech input and creating an appropriate reply.

[1226] "Output means" refers to a method or device for communicating the generated response to the elderly person.

[1227] Response collection tools are methods and techniques used to gather responses and feedback from older adults.

[1228] This invention provides a system that combines a generative artificial intelligence model with virtual reality (VR) technology to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing information related to an individual, and training and optimizing the generative artificial intelligence model. Furthermore, the system provides an interface for realizing dialogue between the elderly and the generative artificial intelligence model, recognizes the elderly's voice input, generates and plays back a response, and collects responses from the elderly for use in retraining the model.

[1229] Data collection methods

[1230] Users (elderly family members and friends) provide personal information, such as their family members' voices, backgrounds, interests, prohibited activities, and local dialects, through a dedicated web portal or application. This information includes memories of family trips and everyday conversations. Specifically, users open the application and upload images, audio files, and text. For example, they can upload photos and descriptions of past family trips.

[1231] Data preprocessing methods

[1232] The server receives the provided information and performs preprocessing, which includes denoising the audio data and standardising the format of the text data. Specifically, the server uses the Python library Librosa to denoise the audio data and the NLTK library to correct spelling and check grammar of the text data.

[1233] A means of training and optimizing generative artificial intelligence models

[1234] The server uses the preprocessed information to train a generative artificial intelligence model using machine learning libraries such as TensorFlow and PyTorch. The trained model is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the model to provide natural and friendly responses to the elderly.

[1235] Building the user interface

[1236] Terminals (devices used by the elderly) will have user interfaces tailored to the elderly. These interfaces will include large buttons and text that take visual and hearing impairments into consideration, as well as voice-guided navigation. Specifically, terminals will be designed to take visual impairments into consideration, displaying buttons and text in large fonts and with clear contrast. They will also provide voice assistant functionality to make them easy for the elderly to operate.

[1237] Real-time interaction means

[1238] When an elderly person begins a dialogue with the generative AI model through the device, the device records the elderly person's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. For example, if an elderly person says, "Tell me about your old trips," the device records this and sends the voice data to the server. The server analyzes this voice data and "generates a response about family trips." The generated response is played back by the device as, "On your old trips, we all went to hot springs together as a family."

[1239] A means of gathering feedback and retraining the model

[1240] During the interaction, the device collects the elderly's reactions and feedback and sends it to the server. Based on this feedback, the server retrains the generative AI model. Specifically, the device asks whether the elderly is satisfied with the response and sends the result to the server. The server uses this feedback to update the model and provide more appropriate responses in the next interaction.

[1241] Specific examples

[1242] Below is a concrete example of how the system actually works.

[1243] When an older person talks about a family trip they took long ago

[1244] Users (family and friends) upload voice and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When the elderly person speaks to the device, saying, "Tell me about your old trips," the device recognizes this and sends the voice data to the server. The server analyzes the voice data and "generates a response about your family trip." The generated response is then played back by the device as, "On your old trips, we all went to a hot spring." The device then records the elderly person's response and sends feedback to the server. Based on this, the model is retrained to provide a more appropriate response in the next interaction.

[1245] Prompt Sentence Examples

[1246] "How are you feeling today?"

[1247] "Want to talk about past family trips?"

[1248] "Tell me some memories you have with your grandchildren."

[1249] In this way, the system can activate memories and conversational content of the elderly, helping to prevent and improve dementia. The system provides new stimulation and joy to the lives of the elderly, supporting their mental health and contributing to reducing loneliness.

[1250] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1251] Step 1:

[1252] Data collection

[1253] Users use a dedicated web portal or application to provide information about their family members, such as their voices, backgrounds, interests, prohibited activities, and local dialects, as well as upload photos, audio files, and text data.

[1254] Input: Audio data, text data, image data

[1255] Output: Collected information (stored in a database)

[1256] What happens: A user opens the application and uploads photos and descriptions of past family trips.

[1257] Step 2:

[1258] Data Preprocessing

[1259] The server preprocesses the collected information: first, it uses the Python library Librosa to denoise the audio data, and then it uses the NLTK library to standardize the format of the text data (for example, to perform spelling correction and grammar checks).

[1260] Input: Collected information (output of step 1)

[1261] Output: Preprocessed information (denoised audio data, normalized text data)

[1262] Specific operation: The server receives the audio data and uses Librosa to remove noise. It also processes the text data with NLTK and standardizes the format.

[1263] Step 3:

[1264] Training and optimizing generative AI models

[1265] The server trains a generative artificial intelligence model based on the preprocessed information using machine learning libraries such as TensorFlow and PyTorch, optimizing the model based on the elderly person's cognitive level, voice characteristics, and individual preferences.

[1266] Input: Preprocessed information (output of step 2)

[1267] Output: A trained generative artificial intelligence model

[1268] Specific operation: The server uses preprocessed voice and text data to train a generative artificial intelligence model and optimize it for the elderly.

[1269] Step 4:

[1270] Building the user interface

[1271] The device will have a user interface tailored to seniors, including large buttons and text for visual and hearing impairments, as well as voice-guided navigation.

[1272] Input: Design specifications, needs of the elderly

[1273] Output: User Interface (UI)

[1274] What it does: Your device displays buttons and text in large fonts and with clear contrast, and offers voice assistant functionality.

[1275] Step 5:

[1276] Real-time dialogue

[1277] When an elderly person initiates a dialogue with the generative AI model through the device, the device records the elderly person's voice input in real time and transmits it to the server, which analyzes the voice input and generates an appropriate response.

[1278] Input: Voice input by elderly people

[1279] Output: The generated response

[1280] Specific operation: When an elderly person says, "Tell me about your past travels," the device records this and sends the audio data to the server. The server analyzes the audio data, "generates a response about family trips," and sends it back to the device. The device then plays back the generated response.

[1281] Step 6:

[1282] Gathering feedback and retraining the model

[1283] During the interaction, the device collects the elderly's reactions and feedback and sends it to the server, which uses this feedback to retrain the generative artificial intelligence model.

[1284] Input: Older adults' reactions and feedback

[1285] Output: A retrained generative AI model

[1286] How it works: The device checks whether the elderly person is satisfied with the response and sends the result to the server. The server uses the feedback to update the model and provide a more appropriate response in the next interaction.

[1287] (Application example 1)

[1288] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1289] There is a need to provide an environment where elderly people can easily select and order meals while staying safe at home. It is also necessary to introduce dialogue systems that contribute to the prevention and improvement of dementia. However, existing systems have not been able to adequately respond to individual needs by taking into account the preferences and history of elderly people. Furthermore, in order for dialogue systems to generate natural and friendly responses to elderly voice input, advanced processing in speech recognition technology and generative AI models is required, posing a challenge to practical implementation.

[1290] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1291] In this invention, the server includes: means for collecting personal data provided by a user; means for preprocessing the collected data to remove noise and normalize it; means for training a generative AI model based on the preprocessed data; means for optimizing the trained model for the elderly; means for providing an interface for enabling dialogue between the elderly and the generative AI model; means for recognizing the elderly's voice input, analyzing the input, and generating a response; means for playing the generated response to the elderly; means for collecting feedback from the elderly and using it to retrain the model; means for presenting a menu based on the elderly's past eating history and preferences; and means for ordering meals based on the presented menu. This enables personalized meal suggestions and ordering based on the elderly's own preferences and history, and also provides an interactive system that is effective in preventing and improving dementia.

[1292] "Personal data provided by the user" refers to information related to the elderly person provided by the user, including the family member's tone of voice, upbringing, preferences, things they don't like, dialect, etc.

[1293] "Collected data" refers to audio data, text data, and the like that are uniquely associated with the senior, including information provided by the user.

[1294] "Preprocessing" refers to processing performed on collected data, such as noise removal and data normalization.

[1295] A "generative AI model" is an artificial intelligence model that has the ability to learn from large datasets and respond to specific tasks.

[1296] "Methods for optimizing for the elderly" refer to techniques for adjusting generative AI models based on the cognitive level, voice characteristics, and individual preferences of the elderly.

[1297] The "interface" is a means of providing an operating screen and voice guidance for elderly people to interact with the generated AI model.

[1298] "Means for recognizing a senior's voice input and analyzing the input to generate a response" refers to the overall process of analyzing a senior's voice and generating an appropriate response.

[1299] "Means for playing the generated response to the elderly" refers to a method for letting the elderly hear the voice response generated by the generative AI model.

[1300] "Means of collecting feedback from seniors and using it to retrain the model" refers to the process of improving the performance of generative AI models based on reactions and opinions obtained from interactions with seniors.

[1301] "Means for presenting menus based on past eating history and preferences" refers to a function that presents appropriate menus based on the elderly person's past eating history and preferences.

[1302] "Means for ordering food based on a presented menu" refers to the functionality of making selections from a presented menu and actually completing a food order.

[1303] This invention aims to prevent and improve dementia in the elderly using the following systems and processes.

[1304] Overall system picture

[1305] First, users (family and friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data includes past dietary history and preferences, the family member's voice tone, upbringing, things they don't like, dialects, etc. The server receives this data and performs noise reduction and data normalization.

[1306] The server then uses the preprocessed data to train a generative AI model, optimizing it based on the elderly person's cognitive level, voice characteristics, and individual preferences, allowing the trained generative AI model to generate natural and friendly responses in real time.

[1307] Building the interface

[1308] The terminal (the device used by the elderly) provides a user interface tailored to the elderly. This interface features large buttons, text, and voice-guided navigation that take visual and auditory impairments into consideration. When the elderly initiates a dialogue with the generative AI model through the terminal, the terminal records the elderly's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. This response is then played back by the terminal.

[1309] Food ordering function

[1310] A distinctive feature of this system is its ability to present menus based on the elderly person's past eating history and preferences. When the elderly person asks, "What's on the menu today?", the device recognizes this and sends the voice data to the server. The server then generates an optimal response based on this and suggests dishes suitable for the elderly. In this way, the application assists the elderly in the process of selecting and ordering meals.

[1311] Gathering feedback and retraining the model

[1312] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions and services for the elderly.

[1313] Hardware and software used

[1314] The system uses the following main hardware and software:

[1315] Smartphones / head-mounted displays: Devices used by the elderly.

[1316] TensorFlow / Keras: Used for training and inference of generative AI models.

[1317] SpeechRecognition: Provides speech recognition technology.

[1318] pyttsx3: Used to speak the generated response.

[1319] Adding specific examples

[1320] For example, if a user (an elderly person) says, "What's today's recommended menu?", the system will generate the following response, taking into account the elderly person's eating history and preferences:

[1321] "Today, I recommend the grilled eel and soup set. Would you like to order that?"

[1322] Prompt Sentence Examples

[1323] The prompt should be in the following format:

[1324] Senior citizen: "What's on the menu today?"

[1325] Generative AI: "Based on your past eating history and preferences, the recommended menu is a set of grilled eel and clear soup. Would you like to order that?"

[1326] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1327] Step 1:

[1328] Users (family and friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This includes past meal history, the family member's voice tone, upbringing, preferences, things they don't like, dialects, etc. This input data is sent to the server. The main purpose of this step is to collect the input data (voice and text data). The output is the initial data before it is preprocessed.

[1329] Step 2:

[1330] The server preprocesses the received data related to individuals, specifically by removing noise and normalizing the data. This process uses the acquired voice and text data as input and processes it through noise filtering, tokenization, and other processes. This results in clean data that the generative AI model can accurately learn from.

[1331] Step 3:

[1332] The server trains a generative AI model based on the preprocessed data. The server uses deep learning libraries such as TensorFlow and Keras to generate a model with voice response capabilities based on the input data. The output of this step is a highly specialized generative AI model.

[1333] Step 4:

[1334] The trained model is optimized for seniors. The server adjusts the model parameters based on the seniors' cognitive level, voice characteristics, and individual preferences. In this process, the generative AI model obtained in the previous step is further tuned so that it can generate responses appropriate for seniors. The output is a generative AI model optimized for seniors.

[1335] Step 5:

[1336] The device provides an interface for dialogue between the elderly person and the generative AI model. It displays an interface that takes visual and auditory impairments into consideration, including large buttons, text, and audio guides. The input is the elderly person's operations, and the output is a GUI (graphical user interface) for dialogue.

[1337] Step 6:

[1338] The elderly person inputs voice through the device. This voice input is recorded in real time and sent to the server. The input is the elderly person's voice, and the output is the recorded voice data. In this step, the voice is captured using the SpeechRecognition library.

[1339] Step 7:

[1340] The server analyzes the received voice data and generates an appropriate response. This process is performed using a generative AI model, and includes voice-to-text conversion, semantic analysis, and response generation. The input is the voice data, and the output is the generated response text.

[1341] Step 8:

[1342] The generated response is converted into speech and played back to the elderly. The device uses the pyttsx3 library to convert text into speech and play it back to the elderly. The input is the response text, and the output is the speech to be played to the elderly.

[1343] Step 9:

[1344] The device collects feedback from the elderly during the interaction. It records the elderly's reactions and evaluations and sends them to the server. The input is the elderly's feedback, and the output is the collected feedback data.

[1345] Step 10:

[1346] The server retrains the generative AI model based on the collected feedback data. In this step, the feedback data is used to fine-tune the model and learn it so that it can be reflected in the next conversation. The input is the feedback data, and the output is the retrained generative AI model.

[1347] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1348] The present invention provides a system that combines a generative AI model, virtual reality (VR) technology, and an emotion engine that recognizes user emotions to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing data related to individuals, and training and optimizing the generative AI model. Furthermore, the system provides an interface for realizing dialogue between the elderly and the generative AI model, recognizes the elderly's voice input, generates and plays a response, and collects feedback from the elderly and uses it to retrain the model. Furthermore, the emotion engine recognizes the user's emotions and adjusts the generative AI model's response based on the emotion.

[1349] Overall system picture

[1350] First, users (family members or friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data can include family memories, anecdotes about specific events, and everyday conversations. The server receives the provided data and performs preprocessing. This preprocessing includes noise reduction for the voice data and normalization for the text data.

[1351] The server then uses the preprocessed data to train a generative AI model, which is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the generative AI model to provide natural and friendly responses to the elderly.

[1352] Building an interface and emotion engine

[1353] The terminal (device used by the elderly) will have a user interface tailored to the elderly. This interface will provide navigation with large buttons, text, and voice guidance that take visual and auditory impairments into consideration. It will also incorporate an emotion engine that will analyze and recognize the emotional state of the elderly from voice input and facial expression data.

[1354] When an elderly person initiates a dialogue with the generative AI model through their device, the device records the elderly person's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. The emotion engine analyzes the elderly person's emotional state and adjusts the content and tone of the response. The generated response is then played back by the device.

[1355] Gathering feedback and retraining the model

[1356] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions for the elderly.

[1357] Specific examples

[1358] Below is a concrete example of how the system actually works.

[1359] When an older person talks about a family trip they took long ago

[1360] Users (family and friends) upload audio and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When an elderly person speaks to the device, saying, "Tell me about your past trips," the device recognizes this and sends the audio data to the server.

[1361] The server analyzes the voice data and generates a response about the family trip. At this time, the emotion engine reads the elderly person's emotions from their tone of voice and facial expressions. If it determines that the elderly person is having fun, for example, the response will be a positive one such as, "On our last trip, we all went to a hot spring together as a family. It was fun!" The generated response is then played back by the device.

[1362] The device then records the elderly person's responses and sends feedback to the server, which uses this feedback to retrain the model and provide more appropriate responses in the next interaction.

[1363] In this way, elderly people can easily use the system at home and it can help prevent and improve dementia. The introduction of the emotion engine makes the dialogue more natural and friendly, further supporting the mental health of the elderly.

[1364] The processing flow will be explained below.

[1365] Step 1:

[1366] Users (family and friends) use a web portal or dedicated app to upload voice and text data related to the elderly, including family memories, anecdotes about specific events, and everyday conversations.

[1367] Step 2:

[1368] The server preprocesses the received data, specifically removing noise from the audio data and normalizing and deleting unnecessary parts from the text data.

[1369] Step 3:

[1370] The server stores the preprocessed data in a database, which is then used to train the generative AI model.

[1371] Step 4:

[1372] The server uses the stored data to train the generative AI model, extracting speech features from the voice data and using natural language processing to extract important keywords from the text data.

[1373] Step 5:

[1374] The server optimizes the trained generative AI model for seniors, adjusting the model parameters based on the seniors' cognitive level, voice characteristics, and individual preferences.

[1375] Step 6:

[1376] The device will create a user interface tailored to the elderly, providing large buttons and text that are visually and auditorily friendly, as well as voice-guided navigation.

[1377] Step 7:

[1378] The device will be equipped with an emotion engine and configured to analyze and recognize the emotional state of the elderly person from voice input and facial expression data.

[1379] Step 8:

[1380] The user (elderly person) initiates a dialogue with the generative AI model through the device. When the elderly person speaks to the device, the device records it in real time and sends it to the server.

[1381] Step 9:

[1382] The server analyzes the received voice data, and the emotion engine estimates the elderly person's emotional state from their tone of voice and facial expressions, and provides this data to the generative AI model.

[1383] Step 10:

[1384] Based on the emotional state estimated by the emotion engine, the generative AI model generates an appropriate response. For example, if the server determines that the elderly person is having fun, it generates a positive response.

[1385] Step 11:

[1386] The server sends the generated response to the terminal, which converts the response into speech and plays it back to the elderly person.

[1387] Step 12:

[1388] The device records the elderly person's reactions and feedback in real time and transmits them to a server.

[1389] Step 13:

[1390] The server uses the feedback collected to retrain the generative AI model and further optimize it for older people, improving the accuracy and performance of the system.

[1391] This allows seniors to enjoy interacting with generative AI models, stimulating their cognitive functions and promoting their mental health. The introduction of an emotion engine makes interactions more natural and friendly.

[1392] Example 2

[1393] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1394] To prevent and improve dementia in the elderly, it is important to provide friendly and appropriate dialogue for each individual elderly person. However, conventional dialogue systems have difficulty providing optimal responses that reflect the emotional state and personal preferences of each elderly person. Furthermore, for elderly people to continue using the system, the dialogue needs to be natural and adapt to their emotions. This is required to maintain and improve the mental health of the elderly.

[1395] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1396] In this invention, the server includes means for collecting personal data provided by a user, means for preprocessing the collected data and performing noise reduction and normalization, means for training a generative AI model based on the preprocessed data, means for optimizing the trained model for the elderly, means for providing an interface for realizing a dialogue between the elderly and the generative AI model, means for recognizing the elderly's voice input, analyzing the input, and generating a response, means for playing the generated response to the elderly, means for collecting feedback from the elderly and using it to retrain the model, an emotion engine for recognizing the elderly's emotional state, and means for adjusting the response of the generative AI model based on the emotion engine. This makes it possible to provide a natural and friendly dialogue that adapts to emotions and improve the mental health of the elderly.

[1397] "User" refers to the entity that provides data, such as an individual, family member, or friend who uses the system.

[1398] "Personal data" refers to information related to an elderly person, such as their lifestyle, memories, tone of voice, upbringing, preferences, prohibitions, dialect, etc.

[1399] "Preprocessing" refers to the process of processing collected data, such as by removing noise and normalizing it, to prepare it in a form that can be used by a generative AI model.

[1400] A "generative AI model" is an artificial intelligence model that generates natural responses based on given data.

[1401] "Training" refers to the process of adjusting the parameters and algorithms of a generative AI model using collected data so that it can generate appropriate responses.

[1402] An "interface" is something that provides an operation screen and input means for a user to interact with a system.

[1403] The "emotion engine" is an engine that analyzes voice input and facial expression data to recognize the emotional state of the elderly.

[1404] "Feedback" refers to data about reactions, opinions, and behaviors obtained from seniors while using the system.

[1405] "Retraining" refers to the process of re-adjusting a generative AI model based on collected feedback data to improve its accuracy and performance.

[1406] "Means for generating a response" refers to the technology or method for analyzing the senior's voice input and generating an appropriate response.

[1407] A "database" is a collection of information that stores data that a generative AI model references when generating responses.

[1408] MODE FOR CARRYING OUT THE INVENTION

[1409] The present invention provides a system that combines a generative AI model, virtual reality (VR) technology, and an emotion engine that recognizes user emotions to prevent and improve dementia in the elderly. The system includes means for collecting and preprocessing data related to individuals, and training and optimizing the generative AI model. Furthermore, the system provides an interface for realizing dialogue between the elderly and the generative AI model, recognizes the elderly's voice input, generates and plays a response, and collects feedback from the elderly and uses it to retrain the model. Furthermore, the emotion engine recognizes the user's emotions and adjusts the generative AI model's response based on the emotion.

[1410] Overall system picture

[1411] First, users (family members or friends of elderly people) provide voice and text data related to the individual through a dedicated web portal or application. This data can include family memories, anecdotes about specific events, and everyday conversations. The server receives the provided data and performs preprocessing. This preprocessing includes noise reduction for the voice data and normalization for the text data.

[1412] The server then uses the preprocessed data to train a generative AI model, which is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences, enabling the generative AI model to provide natural and friendly responses to the elderly.

[1413] Building an interface and emotion engine

[1414] The terminal (device used by the elderly) will have a user interface tailored to the elderly. This interface will provide navigation with large buttons, text, and voice guidance that take visual and auditory impairments into consideration. It will also incorporate an emotion engine that will analyze and recognize the emotional state of the elderly from voice input and facial expression data.

[1415] When an elderly person initiates a dialogue with the generative AI model through their device, the device records the elderly person's voice input in real time and sends it to the server. The server analyzes the voice data and generates an appropriate response. The emotion engine analyzes the elderly person's emotional state and adjusts the content and tone of the response. The generated response is then played back by the device.

[1416] Gathering feedback and retraining the model

[1417] As the elderly interact with the generative AI model, the device collects their reactions and feedback, which is then sent to the server and used to retrain the generative AI model, allowing the model to continuously improve its accuracy and performance and provide more friendly interactions for the elderly.

[1418] Specific examples

[1419] Below is a concrete example of how the system actually works.

[1420] When an older person talks about a family trip they took long ago

[1421] Users (family and friends) upload audio and text data, including photos and anecdotes about family trips. The server preprocesses the data and trains and optimizes the generative AI model. When an elderly person speaks to the device, saying, "Tell me about your past trips," the device recognizes this and sends the audio data to the server.

[1422] The server analyzes the voice data and generates a response about the family trip. At this time, the emotion engine reads the elderly person's emotions from their tone of voice and facial expressions. If it determines that the elderly person is having fun, for example, the response will be a positive one such as, "On our last trip, we all went to a hot spring together as a family. It was fun!" The generated response is then played back by the device.

[1423] The device then records the elderly person's response and sends feedback to the server. The server uses this feedback to retrain the model and provide a more appropriate response the next time the conversation takes place. In this way, the elderly can easily use the system at home, helping to prevent and improve dementia. The introduction of an emotion engine makes conversations more natural and friendly, further supporting the mental health of the elderly.

[1424] Prompt Sentence Examples

[1425] An elderly person might speak into the device and say, "Tell me about your past travels."

[1426] This allows the system to respond to the user's emotions and provide a more friendly dialogue for the elderly.

[1427] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1428] Step 1: Data provision and collection

[1429] Users provide personal audio and text data through a dedicated web portal or application. For example, they upload stories about family trips or everyday conversations. As input, users send audio and text files to the server. This data is received by the server and saved in local storage. As output, saved audio and text data is generated.

[1430] Step 2: Data Preprocessing

[1431] The server performs preprocessing on the collected data. Specific operations include removing noise from the audio data and converting the audio into text using speech recognition technology. Normalization processing (standardization of notation, removal of unnecessary spaces, etc.) is performed on the text data. This generates preprocessed, clean text data.

[1432] Step 3: Training the generative AI model

[1433] The server trains a generative AI model using the preprocessed data. As input, clean text data is provided to the model. The generative AI model (e.g., GPT-4) is used to learn appropriate responses to specific prompt sentences. As output, the generative AI model is able to generate natural and friendly responses for seniors.

[1434] Step 4: Optimize the model

[1435] The trained model is optimized based on the elderly person's cognitive level, voice characteristics, and individual preferences. The elderly person's profile information (cognitive level, preferences, etc.) is used as input. This results in the generation of a generative AI model optimized for the elderly person as the output.

[1436] Step 5: Providing an Interface

[1437] The terminal provides an interface that is easy for seniors to use. It is designed with consideration for visual and auditory impairments, with large buttons, text, and voice-guided navigation. The terminal accepts the operations of the seniors as input. An intuitive and easy-to-operate user interface is provided as output.

[1438] Step 6: Emotion Recognition with the Emotion Engine

[1439] The device collects voice input and facial expression data from the elderly person and uses an emotion engine to analyze and recognize their emotional state. Real-time voice and facial expression data are used as input. Specifically, data is acquired using the device's camera and microphone, and analyzed by the emotion engine. The output is data representing the recognized emotional state.

[1440] Step 7: Dialogue execution and response generation

[1441] When an elderly person speaks to the device, the device records the voice input in real time and sends it to the server. For example, the elderly person might say, "Tell me about your travels in the past." The recorded voice data is used as input. The server analyzes the voice data and generates an appropriate response using a generative AI model. At this time, the emotion engine adjusts the content and tone of the response based on the elderly person's emotional state. The output is a natural and friendly response that adapts to the emotion.

[1442] Step 8: Playing back the response

[1443] The generated response is played back by the device. Specifically, it is played back audibly using the device's speaker. The audio data of the generated response is used as input. The output is an appropriate audio response provided to the elderly person.

[1444] Step 9: Gather feedback

[1445] The device records the elderly person's reactions and feedback during the dialogue process. The input is the elderly person's remarks and changes in facial expressions. Specific operations include collecting data in real time using the device's sensors and analyzing it with an emotion engine. The collected feedback data is generated as output.

[1446] Step 10: Retrain the model

[1447] The server retrains the generative AI model based on the collected feedback. It uses the feedback data as input, allowing the generative AI model to continuously improve its accuracy and performance. The output is a retrained, highly accurate generative AI model.

[1448] Through the above processing steps, the system of the present invention realizes natural and friendly dialogue with the elderly, and can be useful in preventing and improving dementia.

[1449] (Application example 2)

[1450] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1451] When elderly people shop in physical stores, they often have difficulty selecting and using products, or finding the right products. To ensure that elderly people can shop with peace of mind, they need support that can provide appropriate advice and suggestions in real time. Furthermore, to prevent and improve dementia, personalized support using emotion recognition technology is needed.

[1452] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1453] In this invention, the server includes: means for collecting personal data provided by a user; means for preprocessing the collected data and performing noise reduction and normalization; means for training a generative AI model based on the preprocessed data; means for optimizing the trained model for the elderly; means for providing an interface for enabling dialogue between the elderly and the generative AI model; means for recognizing the elderly's voice input, analyzing the input, and generating a response; means for playing the generated response to the elderly; means for collecting feedback from the elderly and using it to retrain the model; means for adjusting the response of the generative AI model based on the elderly's emotions using an emotion engine that recognizes emotions from the elderly's facial expressions and voice; and means for recognizing emotions in real time and providing appropriate product suggestions and advice when the elderly selects products using a head-mounted display. This enables elderly people to receive appropriate emotion-based support in real time when shopping in physical stores.

[1454] "Users" refers to people who use the system, and in this context primarily includes elderly people and their families and friends.

[1455] "Personal data" refers to information about an older adult, including the user's tone of voice, background, preferences, prohibitions, dialect, etc.

[1456] "Means of collection" refers to the methods and technologies used to collect data provided by older adults and their families.

[1457] "Preprocessing" refers to the process of removing noise and normalizing collected data to make it easier to use.

[1458] "Noise reduction" refers to the process of removing unnecessary information from collected data.

[1459] "Normalization" refers to the process of converting collected data into a uniform format.

[1460] A "generative AI model" refers to an artificial intelligence model that is trained based on collected data and generates dialogue with elderly people.

[1461] "Training means" refers to the methods and techniques used to train and optimize generative AI models using collected data.

[1462] "Interface" refers to the point of contact between the user and the system that the elderly person uses to interact with the generative AI model.

[1463] "Means for recognizing voice input" refers to technologies and methods for recognizing and understanding what an older adult is saying.

[1464] "Means for analyzing" refers to techniques or methods for analyzing the senior's voice input and generating an appropriate response.

[1465] "Means for generating a response" refers to a technique or method for generating an appropriate reply based on the analyzed voice input.

[1466] "Means of playback" refers to the technology or method of outputting the generated response in audio or text format to convey it to the elderly.

[1467] "Feedback" refers to the reactions and opinions of the elderly during the dialogue, and is information used to improve the model.

[1468] The "emotion engine" refers to technology that recognizes emotions from the facial expressions and voice of elderly people and adjusts the response of the generative AI model.

[1469] A "head-mounted display" refers to a device worn by elderly people that displays information visually.

[1470] "Product suggestions and advice" refers to appropriate support provided in real time when seniors are choosing products.

[1471] This invention is a system for preventing and improving dementia in the elderly, and is composed of a generative AI model, virtual reality (VR) technology, and an emotion engine. The detailed configuration and operation of the system are described below.

[1472] Overall system configuration

[1473] The system mainly consists of the following components:

[1474] Means of collecting personal data provided by users

[1475] A means of preprocessing the collected data to remove noise and normalize it

[1476] A means of training generative AI models on preprocessed data

[1477] A means of optimizing trained models for older people

[1478] A means of providing an interface for enabling dialogue between elderly people and generative AI models

[1479] A means of recognizing an older adult's voice input and analyzing that input to generate a response

[1480] A means of playing back generated responses to older adults

[1481] A means of collecting feedback from seniors and using it to retrain the model

[1482] A means of adjusting the responses of generative AI models based on the emotions of the elderly, using an emotion engine that recognizes emotions from the elderly's facial expressions and voice.

[1483] A method to recognize emotions in real time and provide appropriate product suggestions and advice to elderly people when they are selecting products using a head-mounted display.

[1484] Program Implementation

[1485] The system program is configured as follows:

[1486] First, the server collects personal data provided by the user, including the elderly person's voice tone, background, preferences, restrictions, dialect, etc. Then, the server preprocesses the collected data, removing noise from the audio data and normalizing the text data.

[1487] Using the pre-processed data, the server trains a generative AI model that is optimized for the cognitive level, vocal characteristics, and individual preferences of seniors.

[1488] When the senior initiates a dialogue, the interface records the senior's voice input in real time and sends it to the server. The server analyzes the voice data and uses a generative AI model to generate an appropriate response. The emotion engine recognizes emotions from the senior's facial expressions and voice, and adjusts the content and tone of the response based on their emotional state.

[1489] The generated responses are then played back to the elderly, and feedback from the elderly is collected and used by the server to retrain the generative AI model.

[1490] Hardware and software used

[1491] Camera-equipped head-mounted display (HMD): Used by elderly people when selecting products in a store, it captures facial expressions and voice in real time.

[1492] Server: Collects user data, pre-processes it, trains generative AI models, analyzes voice input, generates responses, and retrains them.

[1493] Emotion engine: Recognizes emotions from the elderly's facial expressions and voice in real time and uses them to adjust the responses of generative AI models.

[1494] software:

[1495] OpenCV: Acquiring and displaying camera input

[1496] TensorFlow: Running the GPT-2 model

[1497] transformers: Hugging Face GPT-2 tokenizer and model

[1498] EmotionEngine (used as a temporary emotion recognition library)

[1499] Specific examples

[1500] For example, if an elderly person is wearing a head-mounted display and searching for a product in a physical store, the camera captures the elderly person's facial expression, and the emotion engine recognizes from that expression that the elderly person is having trouble. In this case, the generative AI model is prompted to "generate advice about the product that the elderly person is having trouble with." As a result, a response is generated that reads, "This product is a little complicated to use, but it's easy to use if you follow the instructions. Ask a staff member," and this response is played back to the elderly person.

[1501] Example prompt sentence:

[1502] "Generate suggestions for products that seniors enjoy."

[1503] "Generate advice on products that seniors have trouble with."

[1504] In this way, the system allows elderly people to shop in physical stores with peace of mind while also helping to prevent and improve dementia.

[1505] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1506] Step 1:

[1507] Collecting data relating to individuals

[1508] Users provide personal data (tone of voice, background, preferences, restrictions, dialect, etc.) through a dedicated web portal or application, which then inputs the data into the server, which receives it and stores it in a database.

[1509] Step 2:

[1510] Data Preprocessing

[1511] The server performs denoising and normalisation on the received data, which includes denoising audio data and normalising text data. The input is the data provided by the user and the output is the pre-processed, clean data.

[1512] Step 3:

[1513] Training generative AI models

[1514] The server uses the preprocessed data to train a generative AI model. Based on the preprocessed data input as training data, the generative AI model is optimized for elderly people. The output is a trained generative AI model.

[1515] Step 4:

[1516] Providing a dialogue interface with a generative AI model optimized for the elderly

[1517] The elderly person wears a head-mounted display (HMD) as a dialogue interface. The HMD provides a dialogue interface and supports dialogue between the elderly person and the generative AI model through voice input and visual information.

[1518] Step 5:

[1519] Recognition and analysis of elderly voice input

[1520] When an elderly person speaks about a product, the device (HMD) records this voice input in real time and sends it to the server. The server analyzes the voice input and recognizes what the elderly person is saying. The input is the elderly person's voice data, and the output is analyzed text data.

[1521] Step 6:

[1522] Emotion Recognition Using Emotion Engine

[1523] The server receives voice input and facial expression data of the elderly captured by the camera on the HMD. The emotion engine analyzes this data and recognizes the elderly's emotional state. The input is voice data and facial expression data, and the output is recognized emotional information.

[1524] Step 7:

[1525] Response Generation

[1526] The server inputs the recognized text data and emotion information into the generative AI model as a prompt. For example, a prompt such as "Generate suggestions for products that seniors enjoy" is sent. The generative AI model then generates an appropriate response based on this. The input is the prompt text, and the output is the generated response text.

[1527] Step 8:

[1528] Response playback

[1529] The device (HMD) receives the generated response sent from the server and plays it back to the elderly audibly and visually. For example, a message such as "This product is easy to use, please try it" is played back. The input is the response text, and the output is the played back audio message.

[1530] Step 9:

[1531] Collecting feedback

[1532] After receiving the response, the elderly person's reactions and additional comments are recorded by the device and sent to the server, which collects this feedback data and uses it to retrain the generative AI model. The input is the elderly person's reactions and feedback data, and the output is a dataset for retraining.

[1533] Through the above processing steps, the system can assist seniors in their shopping experience and provide appropriate emotion-based responses in real time.

[1534] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1535] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1536] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1537] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1538] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1539] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1540] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1541] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1542] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1543] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1544] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1545] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1546] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1547] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1548] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1549] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1550] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1551] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1552] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1553] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1554] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1555] The following is further disclosed regarding the above embodiment.

[1556] (Claim 1)

[1557] A means for collecting personally identifiable information provided by users;

[1558] means for pre-processing the collected data to denoise and normalize it;

[1559] a means for training a generative AI model based on the preprocessed data; and

[1560] A means of optimizing the trained model for older adults;

[1561] A means for providing an interface for realizing a dialogue between the elderly person and the generative AI model;

[1562] means for recognizing a voice input from the senior citizen and analyzing the input to generate a response;

[1563] a means for playing back the generated response to the elderly person;

[1564] A means of collecting feedback from seniors and using it to retrain the model; and

[1565] A system including:

[1566] (Claim 2)

[1567] 2. The system of claim 1, wherein the data relating to the individual includes family members' voices, backgrounds, preferences, taboos, and dialects.

[1568] (Claim 3)

[1569] 10. The system of claim 1, wherein the generative AI model is configured to generate responses in real time based on data stored in a database.

[1570] "Example 1"

[1571] (Claim 1)

[1572] A means for collecting personally identifiable information provided by users;

[1573] A means of preprocessing the collected information to remove unnecessary data and standardize the format;

[1574] means for training a generative artificial intelligence model based on the preprocessed information;

[1575] A means of optimizing the trained model for older adults;

[1576] A connection means for realizing a dialogue between the elderly person and the generative artificial intelligence model;

[1577] means for recognizing a voice input from the senior citizen and analyzing the input to generate a response;

[1578] a means for outputting the generated response to the senior citizen;

[1579] A means of collecting responses from seniors and using them to retrain the model;

[1580] A system including:

[1581] (Claim 2)

[1582] 2. The system of claim 1, wherein the information related to the individual includes family members' voices, background, interests, prohibited activities, and local dialects.

[1583] (Claim 3)

[1584] 10. The system of claim 1, wherein the generative artificial intelligence model is configured to generate responses in real time based on information stored in an information database.

[1585] "Application Example 1"

[1586] (Claim 1)

[1587] A means for collecting personally identifiable information provided by users;

[1588] means for pre-processing the collected data to denoise and normalize it;

[1589] a means for training a generative AI model based on the preprocessed data; and

[1590] A means of optimizing the trained model for older adults;

[1591] A means for providing an interface for realizing a dialogue between the elderly person and the generative AI model;

[1592] means for recognizing a voice input from the senior citizen and analyzing the input to generate a response;

[1593] a means for playing back the generated response to the elderly person;

[1594] A means of collecting feedback from seniors and using it to retrain the model; and

[1595] a means for presenting menus based on past eating history and preferences;

[1596] a means for ordering a meal based on a presented menu;

[1597] A system including:

[1598] (Claim 2)

[1599] 2. The system of claim 1, wherein the data relating to the individual includes family members' voices, backgrounds, preferences, taboos, and dialects.

[1600] (Claim 3)

[1601] 10. The system of claim 1, wherein the generative AI model is configured to generate responses in real time based on data stored in a database.

[1602] "Example 2: Combining Emotion Engines"

[1603] (Claim 1)

[1604] A means for collecting personally identifiable information provided by users;

[1605] means for pre-processing the collected data to denoise and normalize it;

[1606] a means for training a generative AI model based on the preprocessed data; and ...

Claims

1. A means for collecting personally identifiable information provided by users; means for pre-processing the collected data to denoise and normalize it; means for training a generative AI model based on the preprocessed data; A means of optimizing the trained model for older adults; A means for providing an interface for realizing a dialogue between the elderly person and the generative AI model; means for recognizing a voice input from the senior citizen and analyzing the input to generate a response; a means for playing back the generated response to the elderly person; A means of collecting feedback from seniors and using it to retrain the model; and A system including:

2. The system of claim 1 , wherein the data relating to the individual includes family members' voice tones, backgrounds, preferences, prohibited matters, and dialects.

3. 10. The system of claim 1, wherein the generative AI model is configured to generate responses in real time based on data stored in a database.

Images