system

Abstract

Provide a system. 【Solution means】 Input means for acquiring voice input from a user and converting it into text data, Dialogue generation means for generating an appropriate response using natural language processing based on the generated text data, Response output means for converting the generated response into voice and presenting it to the user, Task generation means for acquiring user performance data and generating training tasks according to the user's cognitive function, Analysis means for analyzing the acquired life log data and evaluating the health status, Notification means for generating and presenting a notification for promoting improvement of lifestyle based on the analysis result, Evaluation means for monitoring changes in cognitive function by data analysis and generating a notification for prompting consultation with a medical expert if necessary, A system including.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, as the aging process progresses, the incidence of dementia is increasing, and its prevention and early detection have become important issues. Despite the urgent social need to reduce loneliness and promote the maintenance of cognitive functions in the elderly, there is a lack of efficient technologies and systems for providing consistent support. Furthermore, there is a need to provide means for effectively promoting health management and improving lifestyle habits in daily life.

Means for Solving the Problems

[0005] This invention features a dialogue function that converts voice input from the user into text data and generates and presents appropriate responses through natural language processing. It also provides a task generation means for generating training tasks suitable for the user's cognitive abilities and acquires and stores user performance data. Furthermore, it includes a function to collect and analyze life log data, evaluate health status, and notify appropriate feedback to encourage lifestyle improvements. This enables monitoring and early detection of the user's cognitive function and, if necessary, encourages consultation with a specialist. Additionally, by improving the quality of dialogue, it provides a system that supports independent living while reducing the user's feelings of isolation.

[0006] "User" refers to an individual who uses the system, and in this invention, it is primarily aimed at elderly people.

[0007] "Voice input" refers to the process by which a system receives the voice spoken by a user, and the means by which that voice is digitized and processed.

[0008] "Text data" refers to character data converted from voice input, and natural language processing is performed based on this data.

[0009] "Natural language processing" is a technology that enables computers to understand and process human language, and in this invention, it is used for dialogue generation.

[0010] "Response output" refers to the process by which the system presents the response generated by natural language processing to the user, and is usually done in the form of speech or text.

[0011] "Performance data" refers to data that includes training results and activity records obtained by users using the system.

[0012] "Task generation" refers to the process by which the system creates training problems that are appropriate for the user's cognitive abilities.

[0013] "Life log data" refers to information about a user's daily activities, including steps taken, sleep duration, and lifestyle habits such as meals.

[0014] "Analysis" is the process of evaluating collected data and deriving useful conclusions or suggestions.

[0015] "Notifications" refer to messages and alerts presented to users based on analysis results, and are intended to encourage improved behavior. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12]It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine. **Mode for Carrying Out the Invention**

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the language used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] The system according to the present invention primarily targets the elderly and supports the prevention and early detection of dementia by providing communication through daily conversation, cognitive function training, and management of daily routines.

[0038] This system is implemented by coordinating a terminal and a server. The terminal receives voice input from the user and converts this voice data into text. The server uses natural language processing techniques based on the text data to generate an appropriate response to the user's input. The terminal receives this response, converts it back into voice, and sends it back to the user.

[0039] For example, if a user says to the device, "I think I'll go for a walk today," the device converts this speech into text and sends it to the server. The server generates a response such as, "That's great! The weather's nice, so it'll be good exercise," which the device then delivers to the user as audio. This process helps maintain cognitive function through everyday conversation.

[0040] Furthermore, the server utilizes the user's past performance data to generate brain training tasks of appropriate difficulty. The terminal can then present these tasks to the user and receive their answers. For example, the server might generate a math problem and ask the user, "What is 3 + 5?" If the user answers "8," the server determines whether the answer is correct or incorrect and provides appropriate feedback. This allows for continuous monitoring of the user's cognitive abilities.

[0041] Furthermore, the user's health status and lifestyle habits are evaluated through the collection and analysis of life log data. For example, based on data such as steps taken and sleep duration, the server generates advice on a healthy lifestyle, and the device notifies the user. These functions support the user's daily life and contribute to maintaining their health.

[0042] This system can also generate notifications to encourage users and their families to consult a specialist if cognitive decline is suspected. This allows for early access to appropriate medical intervention.

[0043] With the above configuration, the system of the present invention comprehensively supports the maintenance and improvement of cognitive function in the elderly, and contributes to improving the quality of life for users.

[0044] The following describes the processing flow.

[0045] Step 1:

[0046] The device captures the user's voice using a microphone and converts the acquired audio into text data using a speech recognition engine. During this process, pre-processing is performed to reduce noise and improve speech clarity.

[0047] Step 2:

[0048] The terminal sends the converted text data to the server. Error checking and data compression techniques are used during this data transfer to ensure communication stability.

[0049] Step 3:

[0050] The server analyzes the received text data and uses natural language processing techniques to understand the user's intent. This process applies algorithms to improve semantic analysis and contextual understanding of the text.

[0051] Step 4:

[0052] The server generates an appropriate response based on the analysis results. This response generation references the conversation history and user profile to provide the user with personalized content.

[0053] Step 5:

[0054] The server sends the generated response to the terminal. Here too, data compression is performed as needed to improve the efficiency of data transfer.

[0055] Step 6:

[0056] The terminal converts the received response text into speech data using a speech synthesis engine. During this process, adjustments are made to improve the naturalness and clarity of the speech.

[0057] Step 7:

[0058] The device plays the converted voice response through its speaker and communicates it to the user. The volume and speed of the voice are also adjusted as needed to ensure the response is played correctly.

[0059] Step 8:

[0060] The server references the user's past performance data and generates training tasks appropriate for their cognitive abilities. The task generation process utilizes the user's score history and trend analysis.

[0061] Step 9:

[0062] The device presents the generated task to the user and prompts them to input their answer. The user's answer can be received in either audio or text format.

[0063] Step 10:

[0064] The terminal sends the user's answer to the server, which then determines whether the answer is correct or incorrect. The evaluation of the answer is performed by comparing it to pre-configured correct answer data.

[0065] Step 11:

[0066] The server generates feedback for the user based on the evaluation results of the answers. This feedback will be used to generate future assignments.

[0067] Step 12:

[0068] The device notifies the user of feedback and provides advice and future goals necessary for improving cognitive function. Notifications are made via voice or on-screen display.

[0069] Step 13:

[0070] The server analyzes the life log data sent by the user and performs a health status assessment. Based on the analysis results, it generates a notification encouraging lifestyle improvements and sends it to the device.

[0071] (Example 1)

[0072] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0073] For elderly users, there is a need for comprehensive support that includes maintaining communication through everyday conversation, training cognitive function, and monitoring their health status. Furthermore, technologies are needed to detect cognitive decline early and facilitate appropriate medical intervention. However, an effective system that can consistently provide all of these is currently lacking.

[0074] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0075] In this invention, the server includes an acquisition means that acquires information of the user who inputs voice and converts it into text data, a generation means that generates an appropriate response using language processing technology based on the generated text data, and an analysis means that analyzes lifestyle record data and evaluates the user's health status. This makes it possible to maintain and improve cognitive function through the user's daily conversations, as well as to monitor the user's health status and promote early medical intervention.

[0076] "Information about the user who inputs voice data" refers to information used to record and analyze the voice data emitted by the user.

[0077] "Means for acquiring data and converting it into text data" refers to a technology or device for converting input speech into text information word for word.

[0078] "Language processing technology" is a technology that generates responses in natural language based on input from users.

[0079] "Generation means" refers to technologies and devices used to perform specific processing and generate necessary information or responses.

[0080] "Lifestyle record data" refers to data that records the user's behavior and health status, and is used for analysis to maintain a healthy lifestyle.

[0081] "Analysis means" refers to a technology or device that analyzes the user's current state and future trends based on collected data.

[0082] "Assessing health status" means diagnosing a user's physical condition and lifestyle habits based on data and providing advice for maintaining their health.

[0083] "Generating a response" means creating appropriate replies or information in response to user input.

[0084] This invention is a system for maintaining cognitive function and managing health through daily communication, targeting elderly individuals and others. This system is realized through the collaboration of a terminal and a server.

[0085] The terminal is a device for receiving voice input from the user. This device uses speech recognition technology to convert the user's voice into text. Here, the speech recognition technology uses general-purpose speech recognition software (e.g., industry-standard speech recognition tools). For example, if the user says, "I think I'll go for a walk today," that voice will be converted into text data.

[0086] The server generates an appropriate response based on the received text data using natural language processing techniques. Natural language processing utilizes generative AI models (e.g., widely used language models in the industry). An example of a prompt is, "Generate a response based on user utterance." The server uses this technique to generate a response such as, "That's great! The weather's nice, so it'll be good exercise."

[0087] The generated text response is returned to the terminal, which then uses speech synthesis technology to convert the text into speech. This speech synthesis technology uses common speech synthesis software (e.g., the synthesis tool used in the system). This provides the response to the user as speech.

[0088] Furthermore, the server has the ability to analyze the user's past performance data and life log data. This allows it to provide cognitive training tasks and health advice. For example, it can provide feedback such as, "Solve the following calculation: 7 + 6" or "You took 10,000 steps today. That's a great pace!"

[0089] In this way, the server and terminal work together to support the user's cognitive functions and create a system that can provide medical advice early on when necessary.

[0090] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0091] Step 1:

[0092] The device receives voice input from the user. Voice data is acquired via the built-in microphone. This input data is then used for subsequent speech recognition processing.

[0093] Step 2:

[0094] The terminal uses speech recognition technology to convert received audio data into text data. Industry-standard speech recognition software is used for processing, analyzing the speech patterns. This text data is then output and ready for transmission to the server.

[0095] Step 3:

[0096] The terminal sends the generated text data to the server. The server receives this text data as input and prepares to generate a response.

[0097] Step 4:

[0098] The server uses a generative AI model to generate an appropriate response based on the received text data. Here, natural language processing techniques are used to analyze the context within the text and select the optimal response to the user's statement. This response is then output as text data.

[0099] Step 5:

[0100] The server sends the generated response as text data to the terminal. The terminal receives this and prepares for the next speech conversion process.

[0101] Step 6:

[0102] The terminal converts the received text data into speech using speech synthesis technology. This process utilizes commonly available speech synthesis software. The resulting speech response data is then output.

[0103] Step 7:

[0104] The device provides the user with a response via voice. This allows the user to receive feedback from the system, completing the interaction.

[0105] Step 8:

[0106] The server analyzes the user's past performance data and generates new cognitive training tasks. This process takes user behavior data as input and outputs tasks of appropriate difficulty.

[0107] Step 9:

[0108] The server sends the generated task to the terminal. The terminal presents the task to the user and prepares to receive the answer.

[0109] Step 10:

[0110] The terminal receives the user's answer and sends it back to the server. The server receives the answer data as input, performs correctness checks, and generates feedback.

[0111] Step 11:

[0112] The server analyzes the life log data to assess the user's health status and generates appropriate advice and notifications. This output provides concrete suggestions for improving the user's lifestyle.

[0113] Step 12:

[0114] The server sends the analysis results to the terminal, and the terminal provides a notification to the user, completing the support for the entire service.

[0115] (Application Example 1)

[0116] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0117] With the aging of modern society, cognitive decline and health problems are becoming increasingly serious. There is a need for communication support to prevent social isolation among the elderly, as well as means for daily health management and maintaining and improving cognitive function. However, conventional methods struggle to provide comprehensive support tailored to individual users.

[0118] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0119] In this invention, the server includes input means for converting voice input into text data, dialogue generation means for generating appropriate responses using natural language processing, and task generation means for generating training tasks according to the user's cognitive function. This makes it possible to promote the user's daily communication, support health maintenance, and improve cognitive function.

[0120] "Voice input" is a technology that acquires voice signals emitted by a user as digital information.

[0121] "Text data" refers to data obtained by analyzing voice input and converting it into corresponding text information.

[0122] "Natural language processing" is a technology that enables computers to understand, generate, and manage human language.

[0123] A "dialogue generation means" is a function that generates appropriate responses or answers based on acquired text data.

[0124] A "training task" is a problem or activity provided to activate a user's cognitive functions.

[0125] "Lifestyle log data" refers to data related to a user's daily activities and health status.

[0126] "Analysis tools for evaluating health status" refers to a function that analyzes acquired lifestyle log data to evaluate the user's health.

[0127] "Notification methods" refer to technologies used to inform users of analysis results and instructions.

[0128] "Evaluation tools" are functions that monitor changes in the user's cognitive function and prompt action as needed.

[0129] "Lifestyle management tools" are technologies that manage users' schedules and activities and provide information related to maintaining their health.

[0130] This system is designed to support the daily lives of the elderly. The terminal receives voice input from the user and converts it into text data. Google® Speech-to-Text API is used for speech recognition. The acquired text data is sent to a server. The server performs natural language processing on the received data and generates an appropriate response. Generative AI models such as OpenAI® GPT-3® and Dialogflow are used for this process. The generated response is converted back from text data to voice data and presented to the user through the terminal.

[0131] The server also analyzes user performance data and lifestyle log data to generate personalized training tasks and health advice. Machine learning algorithms are used for the analysis, enabling monitoring of the user's cognitive abilities and health status. A software component responsible for lifestyle management manages the user's schedule and activities and provides appropriate notifications based on this information.

[0132] For example, if a user asks the device, "What are my plans for tomorrow?", the system will retrieve the weather forecast for that location and respond with a voice message, "It will be sunny tomorrow. It's a good day to go out." An example of a prompt to the generative AI model corresponding to this prompt would be, "Generate weather advice: 'What's the weather like tomorrow?' and suggest recommended activities based on the user's state."

[0133] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0134] Step 1:

[0135] The device receives voice input from the user. This voice data is acquired as a digital audio signal.

[0136] Step 2:

[0137] The device converts the acquired audio data into text data. It uses the Google Speech-to-Text API for speech recognition. This process converts the audio data into corresponding text data.

[0138] Step 3:

[0139] The server receives text data sent from the terminal. Based on this text data, it performs natural language processing. Using OpenAI GPT-3 and Dialogflow, it analyzes the user's intent and generates an appropriate response. This response is generated as text data.

[0140] Step 4:

[0141] The server sends the generated response data to the terminal. This data contains the message to be conveyed to the user.

[0142] Step 5:

[0143] The device converts the received text data into audio data. Using speech synthesis technology, it outputs the audio in a format the user can understand. As a result, the user receives the device's response as audio.

[0144] Step 6:

[0145] The server checks for any additional instructions or questions from the user and generates training tasks and health advice for each individual user based on life log data and performance data. This information is provided to the user via the terminal as needed.

[0146] Step 7:

[0147] The user receives responses and advice from the device and, if necessary, provides further voice input or asks questions. This sequence of steps is repeated until the interaction is complete.

[0148] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0149] The system according to the present invention supports the prevention and early detection of dementia by providing functions related to the user's emotions, in addition to everyday conversation, cognitive function training, and management of daily routines, targeting elderly users and other users.

[0150] The system operates by linking a terminal and a server. A terminal equipped with an emotion engine acquires the user's voice, converts it into text data through speech recognition, and uses this voice input to recognize emotions based on the user's speaking style and content. The recognized emotion information is then sent to the server.

[0151] The server analyzes text data and uses natural language processing techniques to understand the user's intent, while also incorporating emotional information from an emotion engine. This generates a response that takes the user's emotions into consideration. This response is then presented to the user as audio on their device.

[0152] For example, if a user says to the device, "I'm a little tired," the device will transcribe this speech into text and recognize the emotion of "fatigue" from the user's tone of voice and word choice. The server will then consider this emotional information and generate a message such as, "You must be tired. Why don't you take a short break?" which the device will then deliver to the user verbally. This process provides flexible dialogue that also addresses emotional needs.

[0153] Furthermore, the server references the user's past performance data and uses an emotion engine to generate brain training tasks appropriate to the user's state. For example, the server selects a task to reduce the user's stress based on the emotion analysis results and presents it to the user on the terminal. After the task is completed, the user's emotional changes are re-evaluated and used to provide appropriate feedback and incorporate it into subsequent tasks.

[0154] Furthermore, based on life log data, emotional information is incorporated into the evaluation of the user's health status and notifications for improving lifestyle habits. For example, if a user is feeling down, a notification recommending aerobic exercise will be delivered from the device along with an encouraging message. By utilizing the emotional engine, the effectiveness of notifications can be enhanced, and user motivation can be increased.

[0155] Thus, through the system of the present invention, it becomes possible to engage in dialogue and provide support that takes into account the user's emotional changes, thereby maintaining and improving cognitive function and enhancing the user's quality of life.

[0156] The following describes the processing flow.

[0157] Step 1:

[0158] The device acquires the user's voice through the microphone. Since the voice data is processed in real time, the captured audio is digitized immediately.

[0159] Step 2:

[0160] The device converts the acquired audio into text data using a speech recognition engine. During this process, the tone, speed, and rhythm of the voice are also analyzed to extract the user's emotions from the audio data.

[0161] Step 3:

[0162] The device sends the extracted sentiment data to the server along with the converted text data. An encryption protocol is used to ensure security during data transmission.

[0163] Step 4:

[0164] The server analyzes the received text data using natural language processing techniques to understand the user's intent. Simultaneously, it incorporates the received emotional data into the analysis.

[0165] Step 5:

[0166] The server generates an appropriate response based on intent understanding and sentiment analysis. The response is emotionally sensitive; for example, if the user is expressing anxiety, it creates a reassuring message.

[0167] Step 6:

[0168] The server sends the generated response to the terminal. This response includes emotionally sensitive language and advice to encourage the next action.

[0169] Step 7:

[0170] The device converts the received response text into speech using a speech synthesis engine. This conversion process adjusts the intonation and emotional expression of the voice.

[0171] Step 8:

[0172] The device plays the converted voice response through its speaker and presents it to the user. The voice played is modified to be more approachable according to the user's emotional state.

[0173] Step 9:

[0174] The server uses user performance data to generate brain training tasks that take emotions into account. Emotional data influences task selection and difficulty adjustment.

[0175] Step 10:

[0176] The device presents the generated task to the user. The task is presented via voice or text, and the device waits for the user's response.

[0177] Step 11:

[0178] The user answers the presented task. The answer is entered into the device either as voice or text.

[0179] Step 12:

[0180] The device sends the response data to the server, which evaluates the accuracy of the response. The evaluation results, along with the user's emotional state, are recorded.

[0181] Step 13:

[0182] The server generates feedback for the user based on the evaluation results and sentiment data. This feedback includes advice for future tasks and comments to provide emotional support.

[0183] Step 14:

[0184] The device notifies the user of feedback. These notifications are delivered via audio or visual means and are tailored to the user's emotional state.

[0185] (Example 2)

[0186] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0187] In modern society, it is crucial for the elderly and users with specific needs to maintain and improve their cognitive function in their daily lives. However, conventional systems often struggle to provide dialogue that adequately considers emotional states and lack appropriate feedback tailored to the user's psychological state. As a result, users do not experience sufficient satisfaction or support from interacting with the system, and it is difficult to achieve adequate results in dementia prevention and mental health care.

[0188] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0189] In this invention, the server includes an input means for acquiring voice information and converting it into text information, an emotion recognition means for performing emotion analysis and acquiring emotion data, and a dialogue generation means for generating dialogue content based on the generated text information and emotion data. This enables the provision of rich dialogue that is sensitive to emotions and individualized support according to the user's cognitive function.

[0190] "Voice information" refers to data obtained from voice input provided by the user.

[0191] "Textual information" refers to data in text format that has been converted from audio information.

[0192] An "input means" is a means equipped with the function of acquiring audio information and converting it into text information.

[0193] "Emotional analysis" is a process used to determine a user's emotional state from acquired audio and text information.

[0194] "Emotional data" refers to data that shows emotional states and indicators obtained from emotional analysis.

[0195] An "emotion recognition tool" is a tool equipped with the function of performing emotion analysis and acquiring emotion data.

[0196] A "dialogue generation means" is a means equipped with the function of generating appropriate dialogue content for the user based on textual information and emotional data.

[0197] "Dialogue content" refers to the text or audio content of a dialogue created by a dialogue generation means and presented to the user.

[0198] This invention provides a system for elderly people and users with specific needs, offering cognitive enhancement and emotionally resonant dialogue.

[0199] The device first acquires voice information from the user using a high-performance microphone. The acquired voice information is then converted into text information by voice recognition software installed on the device. A general-purpose voice recognition API is used for this voice recognition, and a concrete example is the voice recognition service provided by a major technology company.

[0200] After the audio information is converted into text information, the device's emotion recognition system analyzes the user's voice tone and speaking style to generate emotion data. This emotion data quantifies the user's psychological state and can handle a variety of emotional states.

[0201] Next, the server receives text information and sentiment data transmitted from the terminal. The server utilizes natural language processing technology with generative AI models to deeply understand the user's intent. Using the generated sentiment data in conjunction, it generates dialogue content that is appropriate and emotionally sensitive to the user. This process employs advanced natural language generation models to provide the user with natural, human-like dialogue.

[0202] For example, if a user says "I'm a little tired" to the device, the device converts this voice into text, and the server, based on emotional data indicating fatigue, generates a dialogue such as "You must be tired. Why don't you take a short break?" This response is then spoken through the device's speech synthesis engine and presented to the user.

[0203] Thus, a system that enables emotion-based dialogue generation can conduct flexible dialogues that take into account the user's psychological state. An example of a prompt is, "Suggest relaxation methods according to the user's level of fatigue." By using this prompt, specific suggestions can be made that are tailored to the user's condition.

[0204] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0205] Step 1:

[0206] The device acquires the user's voice information through the microphone. The input is raw voice data. The device uses speech recognition software to convert this voice data into text. This process analyzes the voice waveform, performs phoneme analysis based on a language model, and generates the corresponding text. The output is the text information obtained by converting the acquired voice.

[0207] Step 2:

[0208] The device analyzes the user's voice tone, speed, and volume simultaneously with the converted text information to acquire emotional data. The input is voice attribute data. The device's emotion recognition engine performs analysis based on these voice attributes to estimate the emotional state. The output is the estimated emotional data. Specifically, data for emotional categories such as "fatigue" or "joy" is generated.

[0209] Step 3:

[0210] The server receives text information and sentiment data sent from the terminal. The input consists of text information and sentiment data from the previous step. The generative AI model installed on the server uses natural language processing to analyze the user's intent and generate an appropriate response. In this process, sentiment data is also referenced, and the sentiment of the response is adjusted accordingly. The output is a text-based response that takes sentiment into consideration.

[0211] Step 4:

[0212] The terminal receives the response sent from the server and converts it into speech using a speech synthesis engine. The input is the response in text format. The terminal's speech synthesis engine generates synthesized speech and adjusts the speech to make it easy for the user to understand. The output is the response in audio format presented to the user.

[0213] Step 5:

[0214] The server generates tasks that stimulate the user's cognitive functions based on the user's past performance and sentiment data. The inputs are user history data and sentiment data. The server creates suitable training tasks from the past data and sends them to the terminal. The output is the content of the training tasks directed at the user.

[0215] Step 6:

[0216] The server analyzes life log data, assesses the user's health status, and generates notifications for lifestyle improvements. The input is life log data. Based on the data analysis, the server creates a notification incorporating lifestyle advice and sends it to the device. The output is the message of the notification suggested to the user.

[0217] (Application Example 2)

[0218] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0219] The problem that this invention aims to solve is to realize a system that can provide effective dialogue and feedback while taking emotions into consideration, in order to maintain and improve the cognitive function of the elderly and improve their quality of daily life. Conventional systems cannot accurately grasp and reflect the emotional state of the user, which can result in insufficient prevention, early detection, and emotional support for dementia.

[0220] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0221] In this invention, the server includes an acquisition mechanism that acquires voice input from the user and converts it into text data, a dialogue generation mechanism that generates an appropriate response using natural language processing based on the generated text data, and an emotion response mechanism that provides dialogue and feedback that takes into account the user's mental state based on recognized emotion information. This enables flexible and high-quality support that responds to emotions in dialogue and task presentation aimed at maintaining the user's cognitive function.

[0222] The "acquisition mechanism" is a system that acquires voice input from the user and converts it into text data.

[0223] A "dialogue generation mechanism" is a system that generates appropriate responses using natural language processing based on generated text data.

[0224] A "response output mechanism" is a system that converts the generated response into speech and presents it to the user.

[0225] A "task generation mechanism" is a system that acquires user ability data and generates training tasks tailored to the user's cognitive function.

[0226] An "analysis mechanism" is a system that analyzes acquired lifestyle data and evaluates health status.

[0227] A "notification mechanism" is a system that generates and presents notifications based on analysis results to encourage habit improvement.

[0228] An "emotional response mechanism" is a system that provides dialogue and feedback that takes into account the user's mental state, based on recognized emotional information.

[0229] An "evaluation mechanism" is a system that monitors changes in cognitive function through data analysis and generates notifications prompting consultation with a medical professional when necessary.

[0230] This invention constitutes an emotionally sensitive cognitive support system for the elderly. The system is realized through the cooperation of a terminal and a server.

[0231] The device has a speech recognition function to obtain voice input from the user. This converts the voice into text data and sends it to the server. Common API services are used for speech recognition. For example, a speech API can be used for speech recognition, and a natural language processing API can be used for text conversion.

[0232] The server analyzes the received text data. Using natural language processing (NLP) techniques, it understands the user's intent and related content. Sentiment analysis is also incorporated, extracting the user's emotional information from the audio. Text analysis APIs and emotion recognition models can be used for sentiment analysis.

[0233] The server then uses a generative AI model, based on recognized emotional information and historical data, to generate appropriate responses that correspond to the user's emotions. These responses, processed through an emotion response mechanism, are gentle and considerate towards the user. For example, if a user inputs "I'm not feeling well today," the system will generate an empathetic response such as "Please take it easy and get some rest."

[0234] The generated responses are presented to the user using speech synthesis technology, allowing the user to intuitively engage in the conversation.

[0235] For example, if a user says, "I'm a little tired today," the system recognizes the user's fatigue level through emotion analysis, and the server generates encouraging words and messages suggesting rest, presenting them in a gentle voice as, "You've worked hard. Shall we take a short break?"

[0236] As an example of a prompt to the generating AI model, one possible sentence would be, "The user is feeling tired. Please create appropriate rest suggestions." This would allow the system to provide support tailored to the user's emotional state, thereby supporting the maintenance and improvement of the user's cognitive function and mental health.

[0237] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0238] Step 1:

[0239] The device acquires voice input from the user via the microphone. The acquired voice data is sent to a speech recognition API and converted into text data. In this process, the input is the user's voice and the output is text data.

[0240] Step 2:

[0241] The terminal sends the converted text data to the server. The server receives the text data and uses a natural language processing API to analyze the user's intent. The input is the text data, and the output is the analyzed intent and related data.

[0242] Step 3:

[0243] The server uses an emotion analysis model to extract user emotion information from text and past speech patterns. Input is text data, and output is emotion information. The extracted emotion information is used for subsequent response generation.

[0244] Step 4:

[0245] The server generates appropriate responses using a generative AI model based on the user's intent and emotional information. The generated responses are sensitive to the user's emotions. The input is the analyzed intent and emotional information, and the output is the generated voice response.

[0246] Step 5:

[0247] The server generates a response, which is then sent to the terminal. The terminal uses speech synthesis technology to convert the text response into speech. The input is a text response, and the output is a voice message.

[0248] Step 6:

[0249] The device presents the generated voice response to the user through its speaker. The user receives the system's response as sound. This output allows the user to continue the conversation.

[0250] Through this series of processing steps, the system can engage in dialogue that takes the user's emotions into account and provide cognitive support to the user.

[0251] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0252] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0253] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0254] [Second Embodiment]

[0255] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0256] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0257] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0258] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0259] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0260] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0261] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0262] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0263] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0264] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0265] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0266] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0267] The system according to the present invention primarily targets the elderly and supports the prevention and early detection of dementia by providing communication through daily conversation, cognitive function training, and management of daily routines.

[0268] This system is implemented by coordinating a terminal and a server. The terminal receives voice input from the user and converts this voice data into text. The server uses natural language processing techniques based on the text data to generate an appropriate response to the user's input. The terminal receives this response, converts it back into voice, and sends it back to the user.

[0269] For example, if a user says to the device, "I think I'll go for a walk today," the device converts this speech into text and sends it to the server. The server generates a response such as, "That's great! The weather's nice, so it'll be good exercise," which the device then delivers to the user as audio. This process helps maintain cognitive function through everyday conversation.

[0270] Furthermore, the server utilizes the user's past performance data to generate brain training tasks of appropriate difficulty. The terminal can then present these tasks to the user and receive their answers. For example, the server might generate a math problem and ask the user, "What is 3 + 5?" If the user answers "8," the server determines whether the answer is correct or incorrect and provides appropriate feedback. This allows for continuous monitoring of the user's cognitive abilities.

[0271] Furthermore, the user's health status and lifestyle habits are evaluated through the collection and analysis of life log data. For example, based on data such as steps taken and sleep duration, the server generates advice on a healthy lifestyle, and the device notifies the user. These functions support the user's daily life and contribute to maintaining their health.

[0272] This system can also generate notifications to encourage users and their families to consult a specialist if cognitive decline is suspected. This allows for early access to appropriate medical intervention.

[0273] With the above configuration, the system of the present invention comprehensively supports the maintenance and improvement of cognitive function in the elderly, and contributes to improving the quality of life for users.

[0274] The following describes the processing flow.

[0275] Step 1:

[0276] The terminal captures the user's voice with a microphone and converts the acquired voice into text data by means of a speech recognition engine. At this time, preprocessing is performed to remove noise and improve the clarity of the voice.

[0277] Step 2:

[0278] The terminal transmits the converted text data to the server. For this data transfer, error checking and data compression techniques are used to ensure the stability of communication.

[0279] Step 3:

[0280] The server analyzes the received text data and understands the user's intention using natural language processing techniques. In this process, algorithms for semantic analysis of text and improving the understanding of context are applied.

[0281] Step 4:

[0282] The server generates an appropriate response based on the analysis result. For this response generation, the conversation history and user profile are referenced to provide personalized content to the user.

[0283] Step 5:

[0284] The server transmits the generated response to the terminal. Here too, data compression is performed as necessary to improve the efficiency of data transfer.

[0285] Step 6:

[0286] The terminal converts the received response text into voice data using a text-to-speech engine. At this time, processing is performed to adjust the naturalness and clarity of the voice.

[0287] Step 7:

[0288] The device plays the converted voice response through its speaker and communicates it to the user. The volume and speed of the voice are also adjusted as needed to ensure the response is played correctly.

[0289] Step 8:

[0290] The server references the user's past performance data and generates training tasks appropriate for their cognitive abilities. The task generation process utilizes the user's score history and trend analysis.

[0291] Step 9:

[0292] The device presents the generated task to the user and prompts them to input their answer. The user's answer can be received in either audio or text format.

[0293] Step 10:

[0294] The terminal sends the user's answer to the server, which then determines whether the answer is correct or incorrect. The evaluation of the answer is performed by comparing it to pre-configured correct answer data.

[0295] Step 11:

[0296] The server generates feedback for the user based on the evaluation results of the answers. This feedback will be used to generate future assignments.

[0297] Step 12:

[0298] The device notifies the user of feedback and provides advice and future goals necessary for improving cognitive function. Notifications are made via voice or on-screen display.

[0299] Step 13:

[0300] The server analyzes the life log data sent by the user and performs a health status assessment. Based on the analysis results, it generates a notification encouraging lifestyle improvements and sends it to the device.

[0301] (Example 1)

[0302] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0303] There is a need for a method that comprehensively supports the maintenance of communication through daily conversations, training of cognitive functions, and monitoring of health conditions in elderly users. In addition, technologies for early detection of cognitive function decline and promotion of appropriate medical intervention are also necessary. However, there is currently no effective system that can consistently provide these.

[0304] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0305] In this invention, the server includes an acquisition means for acquiring information of a user who inputs voice and converting it into character data, a generation means for generating an appropriate response using language processing technology based on the generated character data, and an analysis means for analyzing life record data and evaluating the health condition. Thereby, it is possible to improve and maintain cognitive functions through the daily conversations of the user, and to monitor the health condition and promote early medical intervention.

[0306] "Information of a user who inputs voice" is information for recording voice data uttered by the user and analyzing its content.

[0307] "Acquisition means for converting into character data" is a technology or device for converting the input voice into character information one by one.

[0308] "Language processing technology" is a technology for generating a response in natural language expression based on an input from the user.

[0309] "Generation means" is a technology or device used when performing specific processing and generating necessary information and responses.

[0310] "Lifestyle record data" refers to data that records the user's behavior and health status, and is used for analysis to maintain a healthy lifestyle.

[0311] "Analysis means" refers to a technology or device that analyzes the user's current state and future trends based on collected data.

[0312] "Assessing health status" means diagnosing a user's physical condition and lifestyle habits based on data and providing advice for maintaining their health.

[0313] "Generating a response" means creating appropriate replies or information in response to user input.

[0314] This invention is a system for maintaining cognitive function and managing health through daily communication, targeting elderly individuals and others. This system is realized through the collaboration of a terminal and a server.

[0315] The terminal is a device for receiving voice input from the user. This device uses speech recognition technology to convert the user's voice into text. Here, the speech recognition technology uses general-purpose speech recognition software (e.g., industry-standard speech recognition tools). For example, if the user says, "I think I'll go for a walk today," that voice will be converted into text data.

[0316] The server generates an appropriate response based on the received text data using natural language processing techniques. Natural language processing utilizes generative AI models (e.g., widely used language models in the industry). An example of a prompt is, "Generate a response based on user utterance." The server uses this technique to generate a response such as, "That's great! The weather's nice, so it'll be good exercise."

[0317] The generated text response is returned to the terminal, which then uses speech synthesis technology to convert the text into speech. This speech synthesis technology uses common speech synthesis software (e.g., the synthesis tool used in the system). This provides the response to the user as speech.

[0318] Furthermore, the server has the ability to analyze the user's past performance data and life log data. This allows it to provide cognitive training tasks and health advice. For example, it can provide feedback such as, "Solve the following calculation: 7 + 6" or "You took 10,000 steps today. That's a great pace!"

[0319] In this way, the server and terminal work together to support the user's cognitive functions and create a system that can provide medical advice early on when necessary.

[0320] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0321] Step 1:

[0322] The device receives voice input from the user. Voice data is acquired via the built-in microphone. This input data is then used for subsequent speech recognition processing.

[0323] Step 2:

[0324] The terminal uses speech recognition technology to convert received audio data into text data. Industry-standard speech recognition software is used for processing, analyzing the speech patterns. This text data is then output and ready for transmission to the server.

[0325] Step 3:

[0326] The terminal sends the generated text data to the server. The server receives this text data as input and prepares to generate a response.

[0327] Step 4:

[0328] The server uses a generative AI model to generate an appropriate response based on the received text data. Here, natural language processing techniques are used to analyze the context within the text and select the optimal response to the user's statement. This response is then output as text data.

[0329] Step 5:

[0330] The server sends the generated response as text data to the terminal. The terminal receives this and prepares for the next speech conversion process.

[0331] Step 6:

[0332] The terminal converts the received text data into speech using speech synthesis technology. This process utilizes commonly available speech synthesis software. The resulting speech response data is then output.

[0333] Step 7:

[0334] The device provides the user with a response via voice. This allows the user to receive feedback from the system, completing the interaction.

[0335] Step 8:

[0336] The server analyzes the user's past performance data and generates new cognitive training tasks. This process takes user behavior data as input and outputs tasks of appropriate difficulty.

[0337] Step 9:

[0338] The server sends the generated task to the terminal. The terminal presents the task to the user and prepares to receive the answer.

[0339] Step 10:

[0340] The terminal receives the user's answer and sends it back to the server. The server receives the answer data as input, performs correctness checks, and generates feedback.

[0341] Step 11:

[0342] The server analyzes the life log data to assess the user's health status and generates appropriate advice and notifications. This output provides concrete suggestions for improving the user's lifestyle.

[0343] Step 12:

[0344] The server sends the analysis results to the terminal, and the terminal provides a notification to the user, completing the support for the entire service.

[0345] (Application Example 1)

[0346] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0347] With the aging of modern society, cognitive decline and health problems are becoming increasingly serious. There is a need for communication support to prevent social isolation among the elderly, as well as means for daily health management and maintaining and improving cognitive function. However, conventional methods struggle to provide comprehensive support tailored to individual users.

[0348] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0349] In this invention, the server includes input means for converting voice input into text data, dialogue generation means for generating appropriate responses using natural language processing, and task generation means for generating training tasks according to the user's cognitive function. This makes it possible to promote the user's daily communication, support health maintenance, and improve cognitive function.

[0350] "Voice input" is a technology that acquires voice signals emitted by a user as digital information.

[0351] "Text data" refers to data obtained by analyzing voice input and converting it into corresponding text information.

[0352] "Natural language processing" is a technology that enables computers to understand, generate, and manage human language.

[0353] A "dialogue generation means" is a function that generates appropriate responses or answers based on acquired text data.

[0354] A "training task" is a problem or activity provided to activate a user's cognitive functions.

[0355] "Lifestyle log data" refers to data related to a user's daily activities and health status.

[0356] "Analysis tools for evaluating health status" refers to a function that analyzes acquired lifestyle log data to evaluate the user's health.

[0357] "Notification methods" refer to technologies used to inform users of analysis results and instructions.

[0358] "Evaluation tools" are functions that monitor changes in the user's cognitive function and prompt action as needed.

[0359] "Lifestyle management tools" are technologies that manage users' schedules and activities and provide information related to maintaining their health.

[0360] This system is designed to support the daily lives of the elderly. The terminal receives voice input from the user and converts it into text data. The Google Speech-to-Text API is used for speech recognition. The acquired text data is sent to a server. The server performs natural language processing on the received data and generates an appropriate response. Generative AI models such as OpenAI GPT-3 and Dialogflow are used for this process. The generated response is converted back from text data to audio data and presented to the user through the terminal.

[0361] The server also analyzes user performance data and lifestyle log data to generate personalized training tasks and health advice. Machine learning algorithms are used for the analysis, enabling monitoring of the user's cognitive abilities and health status. A software component responsible for lifestyle management manages the user's schedule and activities and provides appropriate notifications based on this information.

[0362] For example, if a user asks the device, "What are my plans for tomorrow?", the system will retrieve the weather forecast for that location and respond with a voice message, "It will be sunny tomorrow. It's a good day to go out." An example of a prompt to the generative AI model corresponding to this prompt would be, "Generate weather advice: 'What's the weather like tomorrow?' and suggest recommended activities based on the user's state."

[0363] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0364] Step 1:

[0365] The device receives voice input from the user. This voice data is acquired as a digital audio signal.

[0366] Step 2:

[0367] The device converts the acquired audio data into text data. It uses the Google Speech-to-Text API for speech recognition. This process converts the audio data into corresponding text data.

[0368] Step 3:

[0369] The server receives text data sent from the terminal. Based on this text data, it performs natural language processing. Using OpenAI GPT-3 and Dialogflow, it analyzes the user's intent and generates an appropriate response. This response is generated as text data.

[0370] Step 4:

[0371] The server sends the generated response data to the terminal. This data contains the message to be conveyed to the user.

[0372] Step 5:

[0373] The device converts the received text data into audio data. Using speech synthesis technology, it outputs the audio in a format the user can understand. As a result, the user receives the device's response as audio.

[0374] Step 6:

[0375] The server checks for any additional instructions or questions from the user and generates training tasks and health advice for each individual user based on life log data and performance data. This information is provided to the user via the terminal as needed.

[0376] Step 7:

[0377] The user receives responses and advice from the device and, if necessary, provides further voice input or asks questions. This sequence of steps is repeated until the interaction is complete.

[0378] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0379] The system according to the present invention supports the prevention and early detection of dementia by providing functions related to the user's emotions, in addition to everyday conversation, cognitive function training, and management of daily routines, targeting elderly users and other users.

[0380] The system operates by linking a terminal and a server. A terminal equipped with an emotion engine acquires the user's voice, converts it into text data through speech recognition, and uses this voice input to recognize emotions based on the user's speaking style and content. The recognized emotion information is then sent to the server.

[0381] The server analyzes text data and uses natural language processing techniques to understand the user's intent, while also incorporating emotional information from an emotion engine. This generates a response that takes the user's emotions into consideration. This response is then presented to the user as audio on their device.

[0382] For example, if a user says to the device, "I'm a little tired," the device will transcribe this speech into text and recognize the emotion of "fatigue" from the user's tone of voice and word choice. The server will then consider this emotional information and generate a message such as, "You must be tired. Why don't you take a short break?" which the device will then deliver to the user verbally. This process provides flexible dialogue that also addresses emotional needs.

[0383] Furthermore, the server references the user's past performance data and uses an emotion engine to generate brain training tasks appropriate to the user's state. For example, the server selects a task to reduce the user's stress based on the emotion analysis results and presents it to the user on the terminal. After the task is completed, the user's emotional changes are re-evaluated and used to provide appropriate feedback and incorporate it into subsequent tasks.

[0384] Furthermore, based on life log data, emotional information is incorporated into the evaluation of the user's health status and notifications for improving lifestyle habits. For example, if a user is feeling down, a notification recommending aerobic exercise will be delivered from the device along with an encouraging message. By utilizing the emotional engine, the effectiveness of notifications can be enhanced, and user motivation can be increased.

[0385] Thus, through the system of the present invention, it becomes possible to engage in dialogue and provide support that takes into account the user's emotional changes, thereby maintaining and improving cognitive function and enhancing the user's quality of life.

[0386] The following describes the processing flow.

[0387] Step 1:

[0388] The device acquires the user's voice through the microphone. Since the voice data is processed in real time, the captured audio is digitized immediately.

[0389] Step 2:

[0390] The device converts the acquired audio into text data using a speech recognition engine. During this process, the tone, speed, and rhythm of the voice are also analyzed to extract the user's emotions from the audio data.

[0391] Step 3:

[0392] The device sends the extracted sentiment data to the server along with the converted text data. An encryption protocol is used to ensure security during data transmission.

[0393] Step 4:

[0394] The server analyzes the received text data using natural language processing techniques to understand the user's intent. Simultaneously, it incorporates the received emotional data into the analysis.

[0395] Step 5:

[0396] The server generates an appropriate response based on intent understanding and sentiment analysis. The response is emotionally sensitive; for example, if the user is expressing anxiety, it creates a reassuring message.

[0397] Step 6:

[0398] The server sends the generated response to the terminal. This response includes emotionally sensitive language and advice to encourage the next action.

[0399] Step 7:

[0400] The device converts the received response text into speech using a speech synthesis engine. This conversion process adjusts the intonation and emotional expression of the voice.

[0401] Step 8:

[0402] The device plays the converted voice response through its speaker and presents it to the user. The voice played is modified to be more approachable according to the user's emotional state.

[0403] Step 9:

[0404] The server uses user performance data to generate brain training tasks that take emotions into account. Emotional data influences task selection and difficulty adjustment.

[0405] Step 10:

[0406] The device presents the generated task to the user. The task is presented via voice or text, and the device waits for the user's response.

[0407] Step 11:

[0408] The user answers the presented task. The answer is entered into the device either as voice or text.

[0409] Step 12:

[0410] The device sends the response data to the server, which evaluates the accuracy of the response. The evaluation results, along with the user's emotional state, are recorded.

[0411] Step 13:

[0412] The server generates feedback for the user based on the evaluation results and sentiment data. This feedback includes advice for future tasks and comments to provide emotional support.

[0413] Step 14:

[0414] The device notifies the user of feedback. These notifications are delivered via audio or visual means and are tailored to the user's emotional state.

[0415] (Example 2)

[0416] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0417] In modern society, it is crucial for the elderly and users with specific needs to maintain and improve their cognitive function in their daily lives. However, conventional systems often struggle to provide dialogue that adequately considers emotional states and lack appropriate feedback tailored to the user's psychological state. As a result, users do not experience sufficient satisfaction or support from interacting with the system, and it is difficult to achieve adequate results in dementia prevention and mental health care.

[0418] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0419] In this invention, the server includes an input means for acquiring voice information and converting it into text information, an emotion recognition means for performing emotion analysis and acquiring emotion data, and a dialogue generation means for generating dialogue content based on the generated text information and emotion data. This enables the provision of rich dialogue that is sensitive to emotions and individualized support according to the user's cognitive function.

[0420] "Voice information" refers to data obtained from voice input provided by the user.

[0421] "Textual information" refers to data in text format that has been converted from audio information.

[0422] An "input means" is a means equipped with the function of acquiring audio information and converting it into text information.

[0423] "Emotional analysis" is a process used to determine a user's emotional state from acquired audio and text information.

[0424] "Emotional data" refers to data that shows emotional states and indicators obtained from emotional analysis.

[0425] An "emotion recognition tool" is a tool equipped with the function of performing emotion analysis and acquiring emotion data.

[0426] A "dialogue generation means" is a means equipped with the function of generating appropriate dialogue content for the user based on textual information and emotional data.

[0427] "Dialogue content" refers to the text or audio content of a dialogue created by a dialogue generation means and presented to the user.

[0428] This invention provides a system for elderly people and users with specific needs, offering cognitive enhancement and emotionally resonant dialogue.

[0429] The device first acquires voice information from the user using a high-performance microphone. The acquired voice information is then converted into text information by voice recognition software installed on the device. A general-purpose voice recognition API is used for this voice recognition, and a concrete example is the voice recognition service provided by a major technology company.

[0430] After the audio information is converted into text information, the device's emotion recognition system analyzes the user's voice tone and speaking style to generate emotion data. This emotion data quantifies the user's psychological state and can handle a variety of emotional states.

[0431] Next, the server receives text information and sentiment data transmitted from the terminal. The server utilizes natural language processing technology with generative AI models to deeply understand the user's intent. Using the generated sentiment data in conjunction, it generates dialogue content that is appropriate and emotionally sensitive to the user. This process employs advanced natural language generation models to provide the user with natural, human-like dialogue.

[0432] For example, if a user says "I'm a little tired" to the device, the device converts this voice into text, and the server, based on emotional data indicating fatigue, generates a dialogue such as "You must be tired. Why don't you take a short break?" This response is then spoken through the device's speech synthesis engine and presented to the user.

[0433] Thus, a system that enables emotion-based dialogue generation can conduct flexible dialogues that take into account the user's psychological state. An example of a prompt is, "Suggest relaxation methods according to the user's level of fatigue." By using this prompt, specific suggestions can be made that are tailored to the user's condition.

[0434] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0435] Step 1:

[0436] The device acquires the user's voice information through the microphone. The input is raw voice data. The device uses speech recognition software to convert this voice data into text. This process analyzes the voice waveform, performs phoneme analysis based on a language model, and generates the corresponding text. The output is the text information obtained by converting the acquired voice.

[0437] Step 2:

[0438] The device analyzes the user's voice tone, speed, and volume simultaneously with the converted text information to acquire emotional data. The input is voice attribute data. The device's emotion recognition engine performs analysis based on these voice attributes to estimate the emotional state. The output is the estimated emotional data. Specifically, data for emotional categories such as "fatigue" or "joy" is generated.

[0439] Step 3:

[0440] The server receives text information and sentiment data sent from the terminal. The input consists of text information and sentiment data from the previous step. The generative AI model installed on the server uses natural language processing to analyze the user's intent and generate an appropriate response. In this process, sentiment data is also referenced, and the sentiment of the response is adjusted accordingly. The output is a text-based response that takes sentiment into consideration.

[0441] Step 4:

[0442] The terminal receives the response sent from the server and converts it into speech using a speech synthesis engine. The input is the response in text format. The terminal's speech synthesis engine generates synthesized speech and adjusts the speech to make it easy for the user to understand. The output is the response in audio format presented to the user.

[0443] Step 5:

[0444] The server generates tasks that stimulate the user's cognitive functions based on the user's past performance and sentiment data. The inputs are user history data and sentiment data. The server creates suitable training tasks from the past data and sends them to the terminal. The output is the content of the training tasks directed at the user.

[0445] Step 6:

[0446] The server analyzes life log data, assesses the user's health status, and generates notifications for lifestyle improvements. The input is life log data. Based on the data analysis, the server creates a notification incorporating lifestyle advice and sends it to the device. The output is the message of the notification suggested to the user.

[0447] (Application Example 2)

[0448] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0449] The problem that this invention aims to solve is to realize a system that can provide effective dialogue and feedback while taking emotions into consideration, in order to maintain and improve the cognitive function of the elderly and improve their quality of daily life. Conventional systems cannot accurately grasp and reflect the emotional state of the user, which can result in insufficient prevention, early detection, and emotional support for dementia.

[0450] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0451] In this invention, the server includes an acquisition mechanism that acquires voice input from the user and converts it into text data, a dialogue generation mechanism that generates an appropriate response using natural language processing based on the generated text data, and an emotion response mechanism that provides dialogue and feedback that takes into account the user's mental state based on recognized emotion information. This enables flexible and high-quality support that responds to emotions in dialogue and task presentation aimed at maintaining the user's cognitive function.

[0452] The "acquisition mechanism" is a system that acquires voice input from the user and converts it into text data.

[0453] A "dialogue generation mechanism" is a system that generates appropriate responses using natural language processing based on generated text data.

[0454] A "response output mechanism" is a system that converts the generated response into speech and presents it to the user.

[0455] A "task generation mechanism" is a system that acquires user ability data and generates training tasks tailored to the user's cognitive function.

[0456] An "analysis mechanism" is a system that analyzes acquired lifestyle data and evaluates health status.

[0457] A "notification mechanism" is a system that generates and presents notifications based on analysis results to encourage habit improvement.

[0458] An "emotional response mechanism" is a system that provides dialogue and feedback that takes into account the user's mental state, based on recognized emotional information.

[0459] An "evaluation mechanism" is a system that monitors changes in cognitive function through data analysis and generates notifications prompting consultation with a medical professional when necessary.

[0460] This invention constitutes an emotionally sensitive cognitive support system for the elderly. The system is realized through the cooperation of a terminal and a server.

[0461] The device has a speech recognition function to obtain voice input from the user. This converts the voice into text data and sends it to the server. Common API services are used for speech recognition. For example, a speech API can be used for speech recognition, and a natural language processing API can be used for text conversion.

[0462] The server analyzes the received text data. Using natural language processing (NLP) techniques, it understands the user's intent and related content. Sentiment analysis is also incorporated, extracting the user's emotional information from the audio. Text analysis APIs and emotion recognition models can be used for sentiment analysis.

[0463] The server then uses a generative AI model, based on recognized emotional information and historical data, to generate appropriate responses that correspond to the user's emotions. These responses, processed through an emotion response mechanism, are gentle and considerate towards the user. For example, if a user inputs "I'm not feeling well today," the system will generate an empathetic response such as "Please take it easy and get some rest."

[0464] The generated responses are presented to the user using speech synthesis technology, allowing the user to intuitively engage in the conversation.

[0465] For example, if a user says, "I'm a little tired today," the system recognizes the user's fatigue level through emotion analysis, and the server generates encouraging words and messages suggesting rest, presenting them in a gentle voice as, "You've worked hard. Shall we take a short break?"

[0466] As an example of a prompt to the generating AI model, one possible sentence would be, "The user is feeling tired. Please create appropriate rest suggestions." This would allow the system to provide support tailored to the user's emotional state, thereby supporting the maintenance and improvement of the user's cognitive function and mental health.

[0467] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0468] Step 1:

[0469] The device acquires voice input from the user via the microphone. The acquired voice data is sent to a speech recognition API and converted into text data. In this process, the input is the user's voice and the output is text data.

[0470] Step 2:

[0471] The terminal sends the converted text data to the server. The server receives the text data and uses a natural language processing API to analyze the user's intent. The input is the text data, and the output is the analyzed intent and related data.

[0472] Step 3:

[0473] The server uses an emotion analysis model to extract user emotion information from text and past speech patterns. Input is text data, and output is emotion information. The extracted emotion information is used for subsequent response generation.

[0474] Step 4:

[0475] The server generates appropriate responses using a generative AI model based on the user's intent and emotional information. The generated responses are sensitive to the user's emotions. The input is the analyzed intent and emotional information, and the output is the generated voice response.

[0476] Step 5:

[0477] The server generates a response, which is then sent to the terminal. The terminal uses speech synthesis technology to convert the text response into speech. The input is a text response, and the output is a voice message.

[0478] Step 6:

[0479] The device presents the generated voice response to the user through its speaker. The user receives the system's response as sound. This output allows the user to continue the conversation.

[0480] Through this series of processing steps, the system can engage in dialogue that takes the user's emotions into account and provide cognitive support to the user.

[0481] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0482] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0483] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0484] [Third Embodiment]

[0485] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0486] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0487] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0488] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0489] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0490] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0491] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0492] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0493] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0494] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0495] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0496] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0497] The system according to the present invention primarily targets the elderly and supports the prevention and early detection of dementia by providing communication through daily conversation, cognitive function training, and management of daily routines.

[0498] This system is implemented by coordinating a terminal and a server. The terminal receives voice input from the user and converts this voice data into text. The server uses natural language processing techniques based on the text data to generate an appropriate response to the user's input. The terminal receives this response, converts it back into voice, and sends it back to the user.

[0499] For example, if a user says to the device, "I think I'll go for a walk today," the device converts this speech into text and sends it to the server. The server generates a response such as, "That's great! The weather's nice, so it'll be good exercise," which the device then delivers to the user as audio. This process helps maintain cognitive function through everyday conversation.

[0500] Furthermore, the server utilizes the user's past performance data to generate brain training tasks of appropriate difficulty. The terminal can then present these tasks to the user and receive their answers. For example, the server might generate a math problem and ask the user, "What is 3 + 5?" If the user answers "8," the server determines whether the answer is correct or incorrect and provides appropriate feedback. This allows for continuous monitoring of the user's cognitive abilities.

[0501] Furthermore, the user's health status and lifestyle habits are evaluated through the collection and analysis of life log data. For example, based on data such as steps taken and sleep duration, the server generates advice on a healthy lifestyle, and the device notifies the user. These functions support the user's daily life and contribute to maintaining their health.

[0502] This system can also generate notifications to encourage users and their families to consult a specialist if cognitive decline is suspected. This allows for early access to appropriate medical intervention.

[0503] With the above configuration, the system of the present invention comprehensively supports the maintenance and improvement of cognitive function in the elderly, and contributes to improving the quality of life for users.

[0504] The following describes the processing flow.

[0505] Step 1:

[0506] The device captures the user's voice using a microphone and converts the acquired audio into text data using a speech recognition engine. During this process, pre-processing is performed to reduce noise and improve speech clarity.

[0507] Step 2:

[0508] The terminal sends the converted text data to the server. Error checking and data compression techniques are used during this data transfer to ensure communication stability.

[0509] Step 3:

[0510] The server analyzes the received text data and uses natural language processing techniques to understand the user's intent. This process applies algorithms to improve semantic analysis and contextual understanding of the text.

[0511] Step 4:

[0512] The server generates an appropriate response based on the analysis results. This response generation references the conversation history and user profile to provide the user with personalized content.

[0513] Step 5:

[0514] The server sends the generated response to the terminal. Here too, data compression is performed as needed to improve the efficiency of data transfer.

[0515] Step 6:

[0516] The terminal converts the received response text into speech data using a speech synthesis engine. During this process, adjustments are made to improve the naturalness and clarity of the speech.

[0517] Step 7:

[0518] The device plays the converted voice response through its speaker and communicates it to the user. The volume and speed of the voice are also adjusted as needed to ensure the response is played correctly.

[0519] Step 8:

[0520] The server references the user's past performance data and generates training tasks appropriate for their cognitive abilities. The task generation process utilizes the user's score history and trend analysis.

[0521] Step 9:

[0522] The device presents the generated task to the user and prompts them to input their answer. The user's answer can be received in either audio or text format.

[0523] Step 10:

[0524] The terminal sends the user's answer to the server, which then determines whether the answer is correct or incorrect. The evaluation of the answer is performed by comparing it to pre-configured correct answer data.

[0525] Step 11:

[0526] The server generates feedback for the user based on the evaluation results of the answers. This feedback will be used to generate future assignments.

[0527] Step 12:

[0528] The device notifies the user of feedback and provides advice and future goals necessary for improving cognitive function. Notifications are made via voice or on-screen display.

[0529] Step 13:

[0530] The server analyzes the life log data sent by the user and performs a health status assessment. Based on the analysis results, it generates a notification encouraging lifestyle improvements and sends it to the device.

[0531] (Example 1)

[0532] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0533] For elderly users, there is a need for comprehensive support that includes maintaining communication through everyday conversation, training cognitive function, and monitoring their health status. Furthermore, technologies are needed to detect cognitive decline early and facilitate appropriate medical intervention. However, an effective system that can consistently provide all of these is currently lacking.

[0534] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0535] In this invention, the server includes an acquisition means that acquires information of the user who inputs voice and converts it into text data, a generation means that generates an appropriate response using language processing technology based on the generated text data, and an analysis means that analyzes lifestyle record data and evaluates the user's health status. This makes it possible to maintain and improve cognitive function through the user's daily conversations, as well as to monitor the user's health status and promote early medical intervention.

[0536] "Information about the user who inputs voice data" refers to information used to record and analyze the voice data emitted by the user.

[0537] "Means for acquiring data and converting it into text data" refers to a technology or device for converting input speech into text information word for word.

[0538] "Language processing technology" is a technology that generates responses in natural language based on input from users.

[0539] "Generation means" refers to technologies and devices used to perform specific processing and generate necessary information or responses.

[0540] "Lifestyle record data" refers to data that records the user's behavior and health status, and is used for analysis to maintain a healthy lifestyle.

[0541] "Analysis means" refers to a technology or device that analyzes the user's current state and future trends based on collected data.

[0542] "Assessing health status" means diagnosing a user's physical condition and lifestyle habits based on data and providing advice for maintaining their health.

[0543] "Generating a response" means creating appropriate replies or information in response to user input.

[0544] This invention is a system for maintaining cognitive function and managing health through daily communication, targeting elderly individuals and others. This system is realized through the collaboration of a terminal and a server.

[0545] The terminal is a device for receiving voice input from the user. This device uses speech recognition technology to convert the user's voice into text. Here, the speech recognition technology uses general-purpose speech recognition software (e.g., industry-standard speech recognition tools). For example, if the user says, "I think I'll go for a walk today," that voice will be converted into text data.

[0546] The server generates an appropriate response based on the received text data using natural language processing techniques. Natural language processing utilizes generative AI models (e.g., widely used language models in the industry). An example of a prompt is, "Generate a response based on user utterance." The server uses this technique to generate a response such as, "That's great! The weather's nice, so it'll be good exercise."

[0547] The generated text response is returned to the terminal, which then uses speech synthesis technology to convert the text into speech. This speech synthesis technology uses common speech synthesis software (e.g., the synthesis tool used in the system). This provides the response to the user as speech.

[0548] Furthermore, the server has the ability to analyze the user's past performance data and life log data. This allows it to provide cognitive training tasks and health advice. For example, it can provide feedback such as, "Solve the following calculation: 7 + 6" or "You took 10,000 steps today. That's a great pace!"

[0549] In this way, the server and terminal work together to support the user's cognitive functions and create a system that can provide medical advice early on when necessary.

[0550] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0551] Step 1:

[0552] The device receives voice input from the user. Voice data is acquired via the built-in microphone. This input data is then used for subsequent speech recognition processing.

[0553] Step 2:

[0554] The terminal uses speech recognition technology to convert received audio data into text data. Industry-standard speech recognition software is used for processing, analyzing the speech patterns. This text data is then output and ready for transmission to the server.

[0555] Step 3:

[0556] The terminal sends the generated text data to the server. The server receives this text data as input and prepares to generate a response.

[0557] Step 4:

[0558] The server uses a generative AI model to generate an appropriate response based on the received text data. Here, natural language processing techniques are used to analyze the context within the text and select the optimal response to the user's statement. This response is then output as text data.

[0559] Step 5:

[0560] The server sends the generated response as text data to the terminal. The terminal receives this and prepares for the next speech conversion process.

[0561] Step 6:

[0562] The terminal converts the received text data into speech using speech synthesis technology. This process utilizes commonly available speech synthesis software. The resulting speech response data is then output.

[0563] Step 7:

[0564] The device provides the user with a response via voice. This allows the user to receive feedback from the system, completing the interaction.

[0565] Step 8:

[0566] The server analyzes the user's past performance data and generates new cognitive training tasks. This process takes user behavior data as input and outputs tasks of appropriate difficulty.

[0567] Step 9:

[0568] The server sends the generated task to the terminal. The terminal presents the task to the user and prepares to receive the answer.

[0569] Step 10:

[0570] The terminal receives the user's answer and sends it back to the server. The server receives the answer data as input, performs correctness checks, and generates feedback.

[0571] Step 11:

[0572] The server analyzes the life log data to assess the user's health status and generates appropriate advice and notifications. This output provides concrete suggestions for improving the user's lifestyle.

[0573] Step 12:

[0574] The server sends the analysis results to the terminal, and the terminal provides a notification to the user, completing the support for the entire service.

[0575] (Application Example 1)

[0576] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0577] With the aging of modern society, cognitive decline and health problems are becoming increasingly serious. There is a need for communication support to prevent social isolation among the elderly, as well as means for daily health management and maintaining and improving cognitive function. However, conventional methods struggle to provide comprehensive support tailored to individual users.

[0578] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0579] In this invention, the server includes input means for converting voice input into text data, dialogue generation means for generating appropriate responses using natural language processing, and task generation means for generating training tasks according to the user's cognitive function. This makes it possible to promote the user's daily communication, support health maintenance, and improve cognitive function.

[0580] "Voice input" is a technology that acquires voice signals emitted by a user as digital information.

[0581] "Text data" refers to data obtained by analyzing voice input and converting it into corresponding text information.

[0582] "Natural language processing" is a technology that enables computers to understand, generate, and manage human language.

[0583] A "dialogue generation means" is a function that generates appropriate responses or answers based on acquired text data.

[0584] A "training task" is a problem or activity provided to activate a user's cognitive functions.

[0585] "Lifestyle log data" refers to data related to a user's daily activities and health status.

[0586] "Analysis tools for evaluating health status" refers to a function that analyzes acquired lifestyle log data to evaluate the user's health.

[0587] "Notification methods" refer to technologies used to inform users of analysis results and instructions.

[0588] "Evaluation tools" are functions that monitor changes in the user's cognitive function and prompt action as needed.

[0589] "Lifestyle management tools" are technologies that manage users' schedules and activities and provide information related to maintaining their health.

[0590] This system is designed to support the daily lives of the elderly. The terminal receives voice input from the user and converts it into text data. The Google Speech-to-Text API is used for speech recognition. The acquired text data is sent to a server. The server performs natural language processing on the received data and generates an appropriate response. Generative AI models such as OpenAI GPT-3 and Dialogflow are used for this process. The generated response is converted back from text data to audio data and presented to the user through the terminal.

[0591] The server also analyzes user performance data and lifestyle log data to generate personalized training tasks and health advice. Machine learning algorithms are used for the analysis, enabling monitoring of the user's cognitive abilities and health status. A software component responsible for lifestyle management manages the user's schedule and activities and provides appropriate notifications based on this information.

[0592] For example, if a user asks the device, "What are my plans for tomorrow?", the system will retrieve the weather forecast for that location and respond with a voice message, "It will be sunny tomorrow. It's a good day to go out." An example of a prompt to the generative AI model corresponding to this prompt would be, "Generate weather advice: 'What's the weather like tomorrow?' and suggest recommended activities based on the user's state."

[0593] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0594] Step 1:

[0595] The device receives voice input from the user. This voice data is acquired as a digital audio signal.

[0596] Step 2:

[0597] The device converts the acquired audio data into text data. It uses the Google Speech-to-Text API for speech recognition. This process converts the audio data into corresponding text data.

[0598] Step 3:

[0599] The server receives text data sent from the terminal. Based on this text data, it performs natural language processing. Using OpenAI GPT-3 and Dialogflow, it analyzes the user's intent and generates an appropriate response. This response is generated as text data.

[0600] Step 4:

[0601] The server sends the generated response data to the terminal. This data contains the message to be conveyed to the user.

[0602] Step 5:

[0603] The device converts the received text data into audio data. Using speech synthesis technology, it outputs the audio in a format the user can understand. As a result, the user receives the device's response as audio.

[0604] Step 6:

[0605] The server checks for any additional instructions or questions from the user and generates training tasks and health advice for each individual user based on life log data and performance data. This information is provided to the user via the terminal as needed.

[0606] Step 7:

[0607] The user receives responses and advice from the device and, if necessary, provides further voice input or asks questions. This sequence of steps is repeated until the interaction is complete.

[0608] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0609] The system according to the present invention supports the prevention and early detection of dementia by providing functions related to the user's emotions, in addition to everyday conversation, cognitive function training, and management of daily routines, targeting elderly users and other users.

[0610] The system operates by linking a terminal and a server. A terminal equipped with an emotion engine acquires the user's voice, converts it into text data through speech recognition, and uses this voice input to recognize emotions based on the user's speaking style and content. The recognized emotion information is then sent to the server.

[0611] The server analyzes text data and uses natural language processing techniques to understand the user's intent, while also incorporating emotional information from an emotion engine. This generates a response that takes the user's emotions into consideration. This response is then presented to the user as audio on their device.

[0612] For example, if a user says to the device, "I'm a little tired," the device will transcribe this speech into text and recognize the emotion of "fatigue" from the user's tone of voice and word choice. The server will then consider this emotional information and generate a message such as, "You must be tired. Why don't you take a short break?" which the device will then deliver to the user verbally. This process provides flexible dialogue that also addresses emotional needs.

[0613] Furthermore, the server references the user's past performance data and uses an emotion engine to generate brain training tasks appropriate to the user's state. For example, the server selects a task to reduce the user's stress based on the emotion analysis results and presents it to the user on the terminal. After the task is completed, the user's emotional changes are re-evaluated and used to provide appropriate feedback and incorporate it into subsequent tasks.

[0614] Furthermore, based on life log data, emotional information is incorporated into the evaluation of the user's health status and notifications for improving lifestyle habits. For example, if a user is feeling down, a notification recommending aerobic exercise will be delivered from the device along with an encouraging message. By utilizing the emotional engine, the effectiveness of notifications can be enhanced, and user motivation can be increased.

[0615] Thus, through the system of the present invention, it becomes possible to engage in dialogue and provide support that takes into account the user's emotional changes, thereby maintaining and improving cognitive function and enhancing the user's quality of life.

[0616] The following describes the processing flow.

[0617] Step 1:

[0618] The device acquires the user's voice through the microphone. Since the voice data is processed in real time, the captured audio is digitized immediately.

[0619] Step 2:

[0620] The device converts the acquired audio into text data using a speech recognition engine. During this process, the tone, speed, and rhythm of the voice are also analyzed to extract the user's emotions from the audio data.

[0621] Step 3:

[0622] The device sends the extracted sentiment data to the server along with the converted text data. An encryption protocol is used to ensure security during data transmission.

[0623] Step 4:

[0624] The server analyzes the received text data using natural language processing techniques to understand the user's intent. Simultaneously, it incorporates the received emotional data into the analysis.

[0625] Step 5:

[0626] The server generates an appropriate response based on intent understanding and sentiment analysis. The response is emotionally sensitive; for example, if the user is expressing anxiety, it creates a reassuring message.

[0627] Step 6:

[0628] The server sends the generated response to the terminal. This response includes emotionally sensitive language and advice to encourage the next action.

[0629] Step 7:

[0630] The device converts the received response text into speech using a speech synthesis engine. This conversion process adjusts the intonation and emotional expression of the voice.

[0631] Step 8:

[0632] The device plays the converted voice response through its speaker and presents it to the user. The voice played is modified to be more approachable according to the user's emotional state.

[0633] Step 9:

[0634] The server uses user performance data to generate brain training tasks that take emotions into account. Emotional data influences task selection and difficulty adjustment.

[0635] Step 10:

[0636] The device presents the generated task to the user. The task is presented via voice or text, and the device waits for the user's response.

[0637] Step 11:

[0638] The user answers the presented task. The answer is entered into the device either as voice or text.

[0639] Step 12:

[0640] The device sends the response data to the server, which evaluates the accuracy of the response. The evaluation results, along with the user's emotional state, are recorded.

[0641] Step 13:

[0642] The server generates feedback for the user based on the evaluation results and sentiment data. This feedback includes advice for future tasks and comments to provide emotional support.

[0643] Step 14:

[0644] The device notifies the user of feedback. These notifications are delivered via audio or visual means and are tailored to the user's emotional state.

[0645] (Example 2)

[0646] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0647] In modern society, it is crucial for the elderly and users with specific needs to maintain and improve their cognitive function in their daily lives. However, conventional systems often struggle to provide dialogue that adequately considers emotional states and lack appropriate feedback tailored to the user's psychological state. As a result, users do not experience sufficient satisfaction or support from interacting with the system, and it is difficult to achieve adequate results in dementia prevention and mental health care.

[0648] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0649] In this invention, the server includes an input means for acquiring voice information and converting it into text information, an emotion recognition means for performing emotion analysis and acquiring emotion data, and a dialogue generation means for generating dialogue content based on the generated text information and emotion data. This enables the provision of rich dialogue that is sensitive to emotions and individualized support according to the user's cognitive function.

[0650] "Voice information" refers to data obtained from voice input provided by the user.

[0651] "Textual information" refers to data in text format that has been converted from audio information.

[0652] An "input means" is a means equipped with the function of acquiring audio information and converting it into text information.

[0653] "Emotional analysis" is a process used to determine a user's emotional state from acquired audio and text information.

[0654] "Emotional data" refers to data that shows emotional states and indicators obtained from emotional analysis.

[0655] An "emotion recognition tool" is a tool equipped with the function of performing emotion analysis and acquiring emotion data.

[0656] A "dialogue generation means" is a means equipped with the function of generating appropriate dialogue content for the user based on textual information and emotional data.

[0657] "Dialogue content" refers to the text or audio content of a dialogue created by a dialogue generation means and presented to the user.

[0658] This invention provides a system for elderly people and users with specific needs, offering cognitive enhancement and emotionally resonant dialogue.

[0659] The device first acquires voice information from the user using a high-performance microphone. The acquired voice information is then converted into text information by voice recognition software installed on the device. A general-purpose voice recognition API is used for this voice recognition, and a concrete example is the voice recognition service provided by a major technology company.

[0660] After the audio information is converted into text information, the device's emotion recognition system analyzes the user's voice tone and speaking style to generate emotion data. This emotion data quantifies the user's psychological state and can handle a variety of emotional states.

[0661] Next, the server receives text information and sentiment data transmitted from the terminal. The server utilizes natural language processing technology with generative AI models to deeply understand the user's intent. Using the generated sentiment data in conjunction, it generates dialogue content that is appropriate and emotionally sensitive to the user. This process employs advanced natural language generation models to provide the user with natural, human-like dialogue.

[0662] For example, if a user says "I'm a little tired" to the device, the device converts this voice into text, and the server, based on emotional data indicating fatigue, generates a dialogue such as "You must be tired. Why don't you take a short break?" This response is then spoken through the device's speech synthesis engine and presented to the user.

[0663] Thus, a system that enables emotion-based dialogue generation can conduct flexible dialogues that take into account the user's psychological state. An example of a prompt is, "Suggest relaxation methods according to the user's level of fatigue." By using this prompt, specific suggestions can be made that are tailored to the user's condition.

[0664] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0665] Step 1:

[0666] The device acquires the user's voice information through the microphone. The input is raw voice data. The device uses speech recognition software to convert this voice data into text. This process analyzes the voice waveform, performs phoneme analysis based on a language model, and generates the corresponding text. The output is the text information obtained by converting the acquired voice.

[0667] Step 2:

[0668] The device analyzes the user's voice tone, speed, and volume simultaneously with the converted text information to acquire emotional data. The input is voice attribute data. The device's emotion recognition engine performs analysis based on these voice attributes to estimate the emotional state. The output is the estimated emotional data. Specifically, data for emotional categories such as "fatigue" or "joy" is generated.

[0669] Step 3:

[0670] The server receives text information and sentiment data sent from the terminal. The input consists of text information and sentiment data from the previous step. The generative AI model installed on the server uses natural language processing to analyze the user's intent and generate an appropriate response. In this process, sentiment data is also referenced, and the sentiment of the response is adjusted accordingly. The output is a text-based response that takes sentiment into consideration.

[0671] Step 4:

[0672] The terminal receives the response sent from the server and converts it into speech using a speech synthesis engine. The input is the response in text format. The terminal's speech synthesis engine generates synthesized speech and adjusts the speech to make it easy for the user to understand. The output is the response in audio format presented to the user.

[0673] Step 5:

[0674] The server generates tasks that stimulate the user's cognitive functions based on the user's past performance and sentiment data. The inputs are user history data and sentiment data. The server creates suitable training tasks from the past data and sends them to the terminal. The output is the content of the training tasks directed at the user.

[0675] Step 6:

[0676] The server analyzes life log data, assesses the user's health status, and generates notifications for lifestyle improvements. The input is life log data. Based on the data analysis, the server creates a notification incorporating lifestyle advice and sends it to the device. The output is the message of the notification suggested to the user.

[0677] (Application Example 2)

[0678] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0679] The problem that this invention aims to solve is to realize a system that can provide effective dialogue and feedback while taking emotions into consideration, in order to maintain and improve the cognitive function of the elderly and improve their quality of daily life. Conventional systems cannot accurately grasp and reflect the emotional state of the user, which can result in insufficient prevention, early detection, and emotional support for dementia.

[0680] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0681] In this invention, the server includes an acquisition mechanism that acquires voice input from the user and converts it into text data, a dialogue generation mechanism that generates an appropriate response using natural language processing based on the generated text data, and an emotion response mechanism that provides dialogue and feedback that takes into account the user's mental state based on recognized emotion information. This enables flexible and high-quality support that responds to emotions in dialogue and task presentation aimed at maintaining the user's cognitive function.

[0682] The "acquisition mechanism" is a system that acquires voice input from the user and converts it into text data.

[0683] A "dialogue generation mechanism" is a system that generates appropriate responses using natural language processing based on generated text data.

[0684] A "response output mechanism" is a system that converts the generated response into speech and presents it to the user.

[0685] A "task generation mechanism" is a system that acquires user ability data and generates training tasks tailored to the user's cognitive function.

[0686] An "analysis mechanism" is a system that analyzes acquired lifestyle data and evaluates health status.

[0687] A "notification mechanism" is a system that generates and presents notifications based on analysis results to encourage habit improvement.

[0688] An "emotional response mechanism" is a system that provides dialogue and feedback that takes into account the user's mental state, based on recognized emotional information.

[0689] An "evaluation mechanism" is a system that monitors changes in cognitive function through data analysis and generates notifications prompting consultation with a medical professional when necessary.

[0690] This invention constitutes an emotionally sensitive cognitive support system for the elderly. The system is realized through the cooperation of a terminal and a server.

[0691] The device has a speech recognition function to obtain voice input from the user. This converts the voice into text data and sends it to the server. Common API services are used for speech recognition. For example, a speech API can be used for speech recognition, and a natural language processing API can be used for text conversion.

[0692] The server analyzes the received text data. Using natural language processing (NLP) techniques, it understands the user's intent and related content. Sentiment analysis is also incorporated, extracting the user's emotional information from the audio. Text analysis APIs and emotion recognition models can be used for sentiment analysis.

[0693] The server then uses a generative AI model, based on recognized emotional information and historical data, to generate appropriate responses that correspond to the user's emotions. These responses, processed through an emotion response mechanism, are gentle and considerate towards the user. For example, if a user inputs "I'm not feeling well today," the system will generate an empathetic response such as "Please take it easy and get some rest."

[0694] The generated responses are presented to the user using speech synthesis technology, allowing the user to intuitively engage in the conversation.

[0695] For example, if a user says, "I'm a little tired today," the system recognizes the user's fatigue level through emotion analysis, and the server generates encouraging words and messages suggesting rest, presenting them in a gentle voice as, "You've worked hard. Shall we take a short break?"

[0696] As an example of a prompt to the generating AI model, one possible sentence would be, "The user is feeling tired. Please create appropriate rest suggestions." This would allow the system to provide support tailored to the user's emotional state, thereby supporting the maintenance and improvement of the user's cognitive function and mental health.

[0697] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0698] Step 1:

[0699] The device acquires voice input from the user via the microphone. The acquired voice data is sent to a speech recognition API and converted into text data. In this process, the input is the user's voice and the output is text data.

[0700] Step 2:

[0701] The terminal sends the converted text data to the server. The server receives the text data and uses a natural language processing API to analyze the user's intent. The input is the text data, and the output is the analyzed intent and related data.

[0702] Step 3:

[0703] The server uses an emotion analysis model to extract user emotion information from text and past speech patterns. Input is text data, and output is emotion information. The extracted emotion information is used for subsequent response generation.

[0704] Step 4:

[0705] The server generates appropriate responses using a generative AI model based on the user's intent and emotional information. The generated responses are sensitive to the user's emotions. The input is the analyzed intent and emotional information, and the output is the generated voice response.

[0706] Step 5:

[0707] The server generates a response, which is then sent to the terminal. The terminal uses speech synthesis technology to convert the text response into speech. The input is a text response, and the output is a voice message.

[0708] Step 6:

[0709] The device presents the generated voice response to the user through its speaker. The user receives the system's response as sound. This output allows the user to continue the conversation.

[0710] Through this series of processing steps, the system can engage in dialogue that takes the user's emotions into account and provide cognitive support to the user.

[0711] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0712] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0713] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0714] [Fourth Embodiment]

[0715] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0716] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0717] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0718] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0719] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0720] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0721] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0722] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0723] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0724] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0725] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0726] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0727] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0728] The system according to the present invention primarily targets the elderly and supports the prevention and early detection of dementia by providing communication through daily conversation, cognitive function training, and management of daily routines.

[0729] This system is implemented by coordinating a terminal and a server. The terminal receives voice input from the user and converts this voice data into text. The server uses natural language processing techniques based on the text data to generate an appropriate response to the user's input. The terminal receives this response, converts it back into voice, and sends it back to the user.

[0730] For example, if a user says to the device, "I think I'll go for a walk today," the device converts this speech into text and sends it to the server. The server generates a response such as, "That's great! The weather's nice, so it'll be good exercise," which the device then delivers to the user as audio. This process helps maintain cognitive function through everyday conversation.

[0731] Furthermore, the server utilizes the user's past performance data to generate brain training tasks of appropriate difficulty. The terminal can then present these tasks to the user and receive their answers. For example, the server might generate a math problem and ask the user, "What is 3 + 5?" If the user answers "8," the server determines whether the answer is correct or incorrect and provides appropriate feedback. This allows for continuous monitoring of the user's cognitive abilities.

[0732] Furthermore, the user's health status and lifestyle habits are evaluated through the collection and analysis of life log data. For example, based on data such as steps taken and sleep duration, the server generates advice on a healthy lifestyle, and the device notifies the user. These functions support the user's daily life and contribute to maintaining their health.

[0733] This system can also generate notifications to encourage users and their families to consult a specialist if cognitive decline is suspected. This allows for early access to appropriate medical intervention.

[0734] With the above configuration, the system of the present invention comprehensively supports the maintenance and improvement of cognitive function in the elderly, and contributes to improving the quality of life for users.

[0735] The following describes the processing flow.

[0736] Step 1:

[0737] The device captures the user's voice using a microphone and converts the acquired audio into text data using a speech recognition engine. During this process, pre-processing is performed to reduce noise and improve speech clarity.

[0738] Step 2:

[0739] The terminal sends the converted text data to the server. Error checking and data compression techniques are used during this data transfer to ensure communication stability.

[0740] Step 3:

[0741] The server analyzes the received text data and uses natural language processing techniques to understand the user's intent. This process applies algorithms to improve semantic analysis and contextual understanding of the text.

[0742] Step 4:

[0743] The server generates an appropriate response based on the analysis results. This response generation references the conversation history and user profile to provide the user with personalized content.

[0744] Step 5:

[0745] The server sends the generated response to the terminal. Here too, data compression is performed as needed to improve the efficiency of data transfer.

[0746] Step 6:

[0747] The terminal converts the received response text into speech data using a speech synthesis engine. During this process, adjustments are made to improve the naturalness and clarity of the speech.

[0748] Step 7:

[0749] The device plays the converted voice response through its speaker and communicates it to the user. The volume and speed of the voice are also adjusted as needed to ensure the response is played correctly.

[0750] Step 8:

[0751] The server references the user's past performance data and generates training tasks appropriate for their cognitive abilities. The task generation process utilizes the user's score history and trend analysis.

[0752] Step 9:

[0753] The device presents the generated task to the user and prompts them to input their answer. The user's answer can be received in either audio or text format.

[0754] Step 10:

[0755] The terminal sends the user's answer to the server, which then determines whether the answer is correct or incorrect. The evaluation of the answer is performed by comparing it to pre-configured correct answer data.

[0756] Step 11:

[0757] The server generates feedback for the user based on the evaluation results of the answers. This feedback will be used to generate future assignments.

[0758] Step 12:

[0759] The device notifies the user of feedback and provides advice and future goals necessary for improving cognitive function. Notifications are made via voice or on-screen display.

[0760] Step 13:

[0761] The server analyzes the life log data sent by the user and performs a health status assessment. Based on the analysis results, it generates a notification encouraging lifestyle improvements and sends it to the device.

[0762] (Example 1)

[0763] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0764] For elderly users, there is a need for comprehensive support that includes maintaining communication through everyday conversation, training cognitive function, and monitoring their health status. Furthermore, technologies are needed to detect cognitive decline early and facilitate appropriate medical intervention. However, an effective system that can consistently provide all of these is currently lacking.

[0765] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0766] In this invention, the server includes an acquisition means that acquires information of the user who inputs voice and converts it into text data, a generation means that generates an appropriate response using language processing technology based on the generated text data, and an analysis means that analyzes lifestyle record data and evaluates the user's health status. This makes it possible to maintain and improve cognitive function through the user's daily conversations, as well as to monitor the user's health status and promote early medical intervention.

[0767] "Information about the user who inputs voice data" refers to information used to record and analyze the voice data emitted by the user.

[0768] "Means for acquiring data and converting it into text data" refers to a technology or device for converting input speech into text information word for word.

[0769] "Language processing technology" is a technology that generates responses in natural language based on input from users.

[0770] "Generation means" refers to technologies and devices used to perform specific processing and generate necessary information or responses.

[0771] "Lifestyle record data" refers to data that records the user's behavior and health status, and is used for analysis to maintain a healthy lifestyle.

[0772] "Analysis means" refers to a technology or device that analyzes the user's current state and future trends based on collected data.

[0773] "Assessing health status" means diagnosing a user's physical condition and lifestyle habits based on data and providing advice for maintaining their health.

[0774] "Generating a response" means creating appropriate replies or information in response to user input.

[0775] This invention is a system for maintaining cognitive function and managing health through daily communication, targeting elderly individuals and others. This system is realized through the collaboration of a terminal and a server.

[0776] The terminal is a device for receiving voice input from the user. This device uses speech recognition technology to convert the user's voice into text. Here, the speech recognition technology uses general-purpose speech recognition software (e.g., industry-standard speech recognition tools). For example, if the user says, "I think I'll go for a walk today," that voice will be converted into text data.

[0777] The server generates an appropriate response based on the received text data using natural language processing techniques. Natural language processing utilizes generative AI models (e.g., widely used language models in the industry). An example of a prompt is, "Generate a response based on user utterance." The server uses this technique to generate a response such as, "That's great! The weather's nice, so it'll be good exercise."

[0778] The generated text response is returned to the terminal, which then uses speech synthesis technology to convert the text into speech. This speech synthesis technology uses common speech synthesis software (e.g., the synthesis tool used in the system). This provides the response to the user as speech.

[0779] Furthermore, the server has the ability to analyze the user's past performance data and life log data. This allows it to provide cognitive training tasks and health advice. For example, it can provide feedback such as, "Solve the following calculation: 7 + 6" or "You took 10,000 steps today. That's a great pace!"

[0780] In this way, the server and terminal work together to support the user's cognitive functions and create a system that can provide medical advice early on when necessary.

[0781] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0782] Step 1:

[0783] The device receives voice input from the user. Voice data is acquired via the built-in microphone. This input data is then used for subsequent speech recognition processing.

[0784] Step 2:

[0785] The terminal uses speech recognition technology to convert received audio data into text data. Industry-standard speech recognition software is used for processing, analyzing the speech patterns. This text data is then output and ready for transmission to the server.

[0786] Step 3:

[0787] The terminal sends the generated text data to the server. The server receives this text data as input and prepares to generate a response.

[0788] Step 4:

[0789] The server uses a generative AI model to generate an appropriate response based on the received text data. Here, natural language processing techniques are used to analyze the context within the text and select the optimal response to the user's statement. This response is then output as text data.

[0790] Step 5:

[0791] The server sends the generated response as text data to the terminal. The terminal receives this and prepares for the next speech conversion process.

[0792] Step 6:

[0793] The terminal converts the received text data into speech using speech synthesis technology. This process utilizes commonly available speech synthesis software. The resulting speech response data is then output.

[0794] Step 7:

[0795] The device provides the user with a response via voice. This allows the user to receive feedback from the system, completing the interaction.

[0796] Step 8:

[0797] The server analyzes the user's past performance data and generates new cognitive training tasks. This process takes user behavior data as input and outputs tasks of appropriate difficulty.

[0798] Step 9:

[0799] The server sends the generated task to the terminal. The terminal presents the task to the user and prepares to receive the answer.

[0800] Step 10:

[0801] The terminal receives the user's answer and sends it back to the server. The server receives the answer data as input, performs correctness checks, and generates feedback.

[0802] Step 11:

[0803] The server analyzes the life log data to assess the user's health status and generates appropriate advice and notifications. This output provides concrete suggestions for improving the user's lifestyle.

[0804] Step 12:

[0805] The server sends the analysis results to the terminal, and the terminal provides a notification to the user, completing the support for the entire service.

[0806] (Application Example 1)

[0807] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0808] With the aging of modern society, cognitive decline and health problems are becoming increasingly serious. There is a need for communication support to prevent social isolation among the elderly, as well as means for daily health management and maintaining and improving cognitive function. However, conventional methods struggle to provide comprehensive support tailored to individual users.

[0809] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0810] In this invention, the server includes input means for converting voice input into text data, dialogue generation means for generating appropriate responses using natural language processing, and task generation means for generating training tasks according to the user's cognitive function. This makes it possible to promote the user's daily communication, support health maintenance, and improve cognitive function.

[0811] "Voice input" is a technology that acquires voice signals emitted by a user as digital information.

[0812] "Text data" refers to data obtained by analyzing voice input and converting it into corresponding text information.

[0813] "Natural language processing" is a technology that enables computers to understand, generate, and manage human language.

[0814] A "dialogue generation means" is a function that generates appropriate responses or answers based on acquired text data.

[0815] A "training task" is a problem or activity provided to activate a user's cognitive functions.

[0816] "Lifestyle log data" refers to data related to a user's daily activities and health status.

[0817] "Analysis tools for evaluating health status" refers to a function that analyzes acquired lifestyle log data to evaluate the user's health.

[0818] "Notification methods" refer to technologies used to inform users of analysis results and instructions.

[0819] "Evaluation tools" are functions that monitor changes in the user's cognitive function and prompt action as needed.

[0820] "Lifestyle management tools" are technologies that manage users' schedules and activities and provide information related to maintaining their health.

[0821] This system is designed to support the daily lives of the elderly. The terminal receives voice input from the user and converts it into text data. The Google Speech-to-Text API is used for speech recognition. The acquired text data is sent to a server. The server performs natural language processing on the received data and generates an appropriate response. Generative AI models such as OpenAI GPT-3 and Dialogflow are used for this process. The generated response is converted back from text data to audio data and presented to the user through the terminal.

[0822] The server also analyzes user performance data and lifestyle log data to generate personalized training tasks and health advice. Machine learning algorithms are used for the analysis, enabling monitoring of the user's cognitive abilities and health status. A software component responsible for lifestyle management manages the user's schedule and activities and provides appropriate notifications based on this information.

[0823] For example, if a user asks the device, "What are my plans for tomorrow?", the system will retrieve the weather forecast for that location and respond with a voice message, "It will be sunny tomorrow. It's a good day to go out." An example of a prompt to the generative AI model corresponding to this prompt would be, "Generate weather advice: 'What's the weather like tomorrow?' and suggest recommended activities based on the user's state."

[0824] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0825] Step 1:

[0826] The device receives voice input from the user. This voice data is acquired as a digital audio signal.

[0827] Step 2:

[0828] The device converts the acquired audio data into text data. It uses the Google Speech-to-Text API for speech recognition. This process converts the audio data into corresponding text data.

[0829] Step 3:

[0830] The server receives text data sent from the terminal. Based on this text data, it performs natural language processing. Using OpenAI GPT-3 and Dialogflow, it analyzes the user's intent and generates an appropriate response. This response is generated as text data.

[0831] Step 4:

[0832] The server sends the generated response data to the terminal. This data contains the message to be conveyed to the user.

[0833] Step 5:

[0834] The device converts the received text data into audio data. Using speech synthesis technology, it outputs the audio in a format the user can understand. As a result, the user receives the device's response as audio.

[0835] Step 6:

[0836] The server checks for any additional instructions or questions from the user and generates training tasks and health advice for each individual user based on life log data and performance data. This information is provided to the user via the terminal as needed.

[0837] Step 7:

[0838] The user receives responses and advice from the device and, if necessary, provides further voice input or asks questions. This sequence of steps is repeated until the interaction is complete.

[0839] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0840] The system according to the present invention supports the prevention and early detection of dementia by providing functions related to the user's emotions, in addition to everyday conversation, cognitive function training, and management of daily routines, targeting elderly users and other users.

[0841] The system operates by linking a terminal and a server. A terminal equipped with an emotion engine acquires the user's voice, converts it into text data through speech recognition, and uses this voice input to recognize emotions based on the user's speaking style and content. The recognized emotion information is then sent to the server.

[0842] The server analyzes text data and uses natural language processing techniques to understand the user's intent, while also incorporating emotional information from an emotion engine. This generates a response that takes the user's emotions into consideration. This response is then presented to the user as audio on their device.

[0843] For example, if a user says to the device, "I'm a little tired," the device will transcribe this speech into text and recognize the emotion of "fatigue" from the user's tone of voice and word choice. The server will then consider this emotional information and generate a message such as, "You must be tired. Why don't you take a short break?" which the device will then deliver to the user verbally. This process provides flexible dialogue that also addresses emotional needs.

[0844] Furthermore, the server references the user's past performance data and uses an emotion engine to generate brain training tasks appropriate to the user's state. For example, the server selects a task to reduce the user's stress based on the emotion analysis results and presents it to the user on the terminal. After the task is completed, the user's emotional changes are re-evaluated and used to provide appropriate feedback and incorporate it into subsequent tasks.

[0845] Furthermore, based on life log data, emotional information is incorporated into the evaluation of the user's health status and notifications for improving lifestyle habits. For example, if a user is feeling down, a notification recommending aerobic exercise will be delivered from the device along with an encouraging message. By utilizing the emotional engine, the effectiveness of notifications can be enhanced, and user motivation can be increased.

[0846] Thus, through the system of the present invention, it becomes possible to engage in dialogue and provide support that takes into account the user's emotional changes, thereby maintaining and improving cognitive function and enhancing the user's quality of life.

[0847] The following describes the processing flow.

[0848] Step 1:

[0849] The device acquires the user's voice through the microphone. Since the voice data is processed in real time, the captured audio is digitized immediately.

[0850] Step 2:

[0851] The device converts the acquired audio into text data using a speech recognition engine. During this process, the tone, speed, and rhythm of the voice are also analyzed to extract the user's emotions from the audio data.

[0852] Step 3:

[0853] The device sends the extracted sentiment data to the server along with the converted text data. An encryption protocol is used to ensure security during data transmission.

[0854] Step 4:

[0855] The server analyzes the received text data using natural language processing techniques to understand the user's intent. Simultaneously, it incorporates the received emotional data into the analysis.

[0856] Step 5:

[0857] The server generates an appropriate response based on intent understanding and sentiment analysis. The response is emotionally sensitive; for example, if the user is expressing anxiety, it creates a reassuring message.

[0858] Step 6:

[0859] The server sends the generated response to the terminal. This response includes emotionally sensitive language and advice to encourage the next action.

[0860] Step 7:

[0861] The device converts the received response text into speech using a speech synthesis engine. This conversion process adjusts the intonation and emotional expression of the voice.

[0862] Step 8:

[0863] The device plays the converted voice response through its speaker and presents it to the user. The voice played is modified to be more approachable according to the user's emotional state.

[0864] Step 9:

[0865] The server uses user performance data to generate brain training tasks that take emotions into account. Emotional data influences task selection and difficulty adjustment.

[0866] Step 10:

[0867] The device presents the generated task to the user. The task is presented via voice or text, and the device waits for the user's response.

[0868] Step 11:

[0869] The user answers the presented task. The answer is entered into the device either as voice or text.

[0870] Step 12:

[0871] The device sends the response data to the server, which evaluates the accuracy of the response. The evaluation results, along with the user's emotional state, are recorded.

[0872] Step 13:

[0873] The server generates feedback for the user based on the evaluation results and sentiment data. This feedback includes advice for future tasks and comments to provide emotional support.

[0874] Step 14:

[0875] The device notifies the user of feedback. These notifications are delivered via audio or visual means and are tailored to the user's emotional state.

[0876] (Example 2)

[0877] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0878] In modern society, it is crucial for the elderly and users with specific needs to maintain and improve their cognitive function in their daily lives. However, conventional systems often struggle to provide dialogue that adequately considers emotional states and lack appropriate feedback tailored to the user's psychological state. As a result, users do not experience sufficient satisfaction or support from interacting with the system, and it is difficult to achieve adequate results in dementia prevention and mental health care.

[0879] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0880] In this invention, the server includes an input means for acquiring voice information and converting it into text information, an emotion recognition means for performing emotion analysis and acquiring emotion data, and a dialogue generation means for generating dialogue content based on the generated text information and emotion data. This enables the provision of rich dialogue that is sensitive to emotions and individualized support according to the user's cognitive function.

[0881] "Voice information" refers to data obtained from voice input provided by the user.

[0882] "Textual information" refers to data in text format that has been converted from audio information.

[0883] An "input means" is a means equipped with the function of acquiring audio information and converting it into text information.

[0884] "Emotional analysis" is a process used to determine a user's emotional state from acquired audio and text information.

[0885] "Emotional data" refers to data that shows emotional states and indicators obtained from emotional analysis.

[0886] An "emotion recognition tool" is a tool equipped with the function of performing emotion analysis and acquiring emotion data.

[0887] A "dialogue generation means" is a means equipped with the function of generating appropriate dialogue content for the user based on textual information and emotional data.

[0888] "Dialogue content" refers to the text or audio content of a dialogue created by a dialogue generation means and presented to the user.

[0889] This invention provides a system for elderly people and users with specific needs, offering cognitive enhancement and emotionally resonant dialogue.

[0890] The device first acquires voice information from the user using a high-performance microphone. The acquired voice information is then converted into text information by voice recognition software installed on the device. A general-purpose voice recognition API is used for this voice recognition, and a concrete example is the voice recognition service provided by a major technology company.

[0891] After the audio information is converted into text information, the device's emotion recognition system analyzes the user's voice tone and speaking style to generate emotion data. This emotion data quantifies the user's psychological state and can handle a variety of emotional states.

[0892] Next, the server receives text information and sentiment data transmitted from the terminal. The server utilizes natural language processing technology with generative AI models to deeply understand the user's intent. Using the generated sentiment data in conjunction, it generates dialogue content that is appropriate and emotionally sensitive to the user. This process employs advanced natural language generation models to provide the user with natural, human-like dialogue.

[0893] For example, if a user says "I'm a little tired" to the device, the device converts this voice into text, and the server, based on emotional data indicating fatigue, generates a dialogue such as "You must be tired. Why don't you take a short break?" This response is then spoken through the device's speech synthesis engine and presented to the user.

[0894] Thus, a system that enables emotion-based dialogue generation can conduct flexible dialogues that take into account the user's psychological state. An example of a prompt is, "Suggest relaxation methods according to the user's level of fatigue." By using this prompt, specific suggestions can be made that are tailored to the user's condition.

[0895] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0896] Step 1:

[0897] The device acquires the user's voice information through the microphone. The input is raw voice data. The device uses speech recognition software to convert this voice data into text. This process analyzes the voice waveform, performs phoneme analysis based on a language model, and generates the corresponding text. The output is the text information obtained by converting the acquired voice.

[0898] Step 2:

[0899] The device analyzes the user's voice tone, speed, and volume simultaneously with the converted text information to acquire emotional data. The input is voice attribute data. The device's emotion recognition engine performs analysis based on these voice attributes to estimate the emotional state. The output is the estimated emotional data. Specifically, data for emotional categories such as "fatigue" or "joy" is generated.

[0900] Step 3:

[0901] The server receives text information and sentiment data sent from the terminal. The input consists of text information and sentiment data from the previous step. The generative AI model installed on the server uses natural language processing to analyze the user's intent and generate an appropriate response. In this process, sentiment data is also referenced, and the sentiment of the response is adjusted accordingly. The output is a text-based response that takes sentiment into consideration.

[0902] Step 4:

[0903] The terminal receives the response sent from the server and converts it into speech using a speech synthesis engine. The input is the response in text format. The terminal's speech synthesis engine generates synthesized speech and adjusts the speech to make it easy for the user to understand. The output is the response in audio format presented to the user.

[0904] Step 5:

[0905] The server generates tasks that stimulate the user's cognitive functions based on the user's past performance and sentiment data. The inputs are user history data and sentiment data. The server creates suitable training tasks from the past data and sends them to the terminal. The output is the content of the training tasks directed at the user.

[0906] Step 6:

[0907] The server analyzes life log data, assesses the user's health status, and generates notifications for lifestyle improvements. The input is life log data. Based on the data analysis, the server creates a notification incorporating lifestyle advice and sends it to the device. The output is the message of the notification suggested to the user.

[0908] (Application Example 2)

[0909] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0910] The problem that this invention aims to solve is to realize a system that can provide effective dialogue and feedback while taking emotions into consideration, in order to maintain and improve the cognitive function of the elderly and improve their quality of daily life. Conventional systems cannot accurately grasp and reflect the emotional state of the user, which can result in insufficient prevention, early detection, and emotional support for dementia.

[0911] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0912] In this invention, the server includes an acquisition mechanism that acquires voice input from the user and converts it into text data, a dialogue generation mechanism that generates an appropriate response using natural language processing based on the generated text data, and an emotion response mechanism that provides dialogue and feedback that takes into account the user's mental state based on recognized emotion information. This enables flexible and high-quality support that responds to emotions in dialogue and task presentation aimed at maintaining the user's cognitive function.

[0913] The "acquisition mechanism" is a system that acquires voice input from the user and converts it into text data.

[0914] A "dialogue generation mechanism" is a system that generates appropriate responses using natural language processing based on generated text data.

[0915] A "response output mechanism" is a system that converts the generated response into speech and presents it to the user.

[0916] A "task generation mechanism" is a system that acquires user ability data and generates training tasks tailored to the user's cognitive function.

[0917] An "analysis mechanism" is a system that analyzes acquired lifestyle data and evaluates health status.

[0918] A "notification mechanism" is a system that generates and presents notifications based on analysis results to encourage habit improvement.

[0919] An "emotional response mechanism" is a system that provides dialogue and feedback that takes into account the user's mental state, based on recognized emotional information.

[0920] An "evaluation mechanism" is a system that monitors changes in cognitive function through data analysis and generates notifications prompting consultation with a medical professional when necessary.

[0921] This invention constitutes an emotionally sensitive cognitive support system for the elderly. The system is realized through the cooperation of a terminal and a server.

[0922] The device has a speech recognition function to obtain voice input from the user. This converts the voice into text data and sends it to the server. Common API services are used for speech recognition. For example, a speech API can be used for speech recognition, and a natural language processing API can be used for text conversion.

[0923] The server analyzes the received text data. Using natural language processing (NLP) techniques, it understands the user's intent and related content. Sentiment analysis is also incorporated, extracting the user's emotional information from the audio. Text analysis APIs and emotion recognition models can be used for sentiment analysis.

[0924] The server then uses a generative AI model, based on recognized emotional information and historical data, to generate appropriate responses that correspond to the user's emotions. These responses, processed through an emotion response mechanism, are gentle and considerate towards the user. For example, if a user inputs "I'm not feeling well today," the system will generate an empathetic response such as "Please take it easy and get some rest."

[0925] The generated responses are presented to the user using speech synthesis technology, allowing the user to intuitively engage in the conversation.

[0926] For example, if a user says, "I'm a little tired today," the system recognizes the user's fatigue level through emotion analysis, and the server generates encouraging words and messages suggesting rest, presenting them in a gentle voice as, "You've worked hard. Shall we take a short break?"

[0927] As an example of a prompt to the generating AI model, one possible sentence would be, "The user is feeling tired. Please create appropriate rest suggestions." This would allow the system to provide support tailored to the user's emotional state, thereby supporting the maintenance and improvement of the user's cognitive function and mental health.

[0928] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0929] Step 1:

[0930] The device acquires voice input from the user via the microphone. The acquired voice data is sent to a speech recognition API and converted into text data. In this process, the input is the user's voice and the output is text data.

[0931] Step 2:

[0932] The terminal sends the converted text data to the server. The server receives the text data and uses a natural language processing API to analyze the user's intent. The input is the text data, and the output is the analyzed intent and related data.

[0933] Step 3:

[0934] The server uses an emotion analysis model to extract user emotion information from text and past speech patterns. Input is text data, and output is emotion information. The extracted emotion information is used for subsequent response generation.

[0935] Step 4:

[0936] The server generates appropriate responses using a generative AI model based on the user's intent and emotional information. The generated responses are sensitive to the user's emotions. The input is the analyzed intent and emotional information, and the output is the generated voice response.

[0937] Step 5:

[0938] The server generates a response, which is then sent to the terminal. The terminal uses speech synthesis technology to convert the text response into speech. The input is a text response, and the output is a voice message.

[0939] Step 6:

[0940] The device presents the generated voice response to the user through its speaker. The user receives the system's response as sound. This output allows the user to continue the conversation.

[0941] Through this series of processing steps, the system can engage in dialogue that takes the user's emotions into account and provide cognitive support to the user.

[0942] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0943] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0944] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0945] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0946] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0947] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0948] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0949] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0950] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0951] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0952] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0953] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0954] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0955] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0956] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0957] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0958] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0959] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0960] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0961] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0962] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0963] The following is further disclosed regarding the embodiments described above.

[0964] (Claim 1)

[0965] An input means that acquires voice input from the user and converts it into text data,

[0966] A dialogue generation means that generates an appropriate response using natural language processing based on the generated text data,

[0967] A response output means that converts the generated response into speech and presents it to the user,

[0968] A task generation means that acquires user performance data and generates training tasks according to the user's cognitive function,

[0969] An analytical means for analyzing acquired life log data and evaluating health status,

[0970] A notification means that generates and presents notifications to encourage lifestyle improvements based on analysis results,

[0971] An evaluation method that monitors changes in cognitive function through data analysis and generates notifications prompting consultation with a medical professional when necessary,

[0972] A system that includes this.

[0973] (Claim 2)

[0974] The system according to claim 1, characterized in that it improves the quality of dialogue with the user by using natural language processing and accumulates data on the user's cognitive function.

[0975] (Claim 3)

[0976] The system according to claim 1, characterized in that it acquires user life log data from multiple sensors and provides appropriate feedback on the user's lifestyle habits based on that data.

[0977] "Example 1"

[0978] (Claim 1)

[0979] A means for acquiring information about a user who inputs voice and converting it into text data,

[0980] A generation means that generates an appropriate response using language processing technology based on the generated character data,

[0981] An output means that converts the generated response into speech and presents it to the user,

[0982] A generation means that acquires user behavior data and generates training tasks according to the user's cognitive abilities,

[0983] An analytical means for analyzing acquired lifestyle record data and evaluating health status,

[0984] A notification means that generates and presents notifications to encourage lifestyle improvements based on analysis results,

[0985] An evaluation method that monitors changes in cognitive function through data analysis and generates notifications prompting consultation with a medical professional when necessary,

[0986] A system that includes this.

[0987] (Claim 2)

[0988] The system according to claim 1, characterized in that it improves the quality of dialogue with the user by using language processing technology and accumulates data on the user's cognitive abilities.

[0989] (Claim 3)

[0990] The system according to claim 1, characterized in that it acquires user lifestyle record data from multiple sensors and provides appropriate feedback on the user's lifestyle habits based on that data.

[0991] "Application Example 1"

[0992] (Claim 1)

[0993] An input means that acquires voice input from the user and converts it into text data,

[0994] A dialogue generation means that generates an appropriate response using natural language processing based on the generated character data,

[0995] A response output means that converts the generated response into speech and presents it to the user,

[0996] A task generation means that acquires user ability data and generates training tasks according to the user's cognitive function,

[0997] An analytical method for analyzing acquired lifestyle log data and evaluating health status,

[0998] A notification means that generates and presents notifications to encourage lifestyle improvements based on analysis results,

[0999] An evaluation method that monitors changes in cognitive function through data analysis and generates notifications prompting consultation with a medical professional when necessary,

[1000] A lifestyle management tool that manages the user's schedule and activities and provides notifications related to health maintenance,

[1001] A system that includes this.

[1002] (Claim 2)

[1003] The system according to claim 1, characterized in that it improves the quality of dialogue with the user by using natural language processing and accumulates data on the user's cognitive function.

[1004] (Claim 3)

[1005] The system according to claim 1, characterized in that it acquires user lifestyle log data from multiple measuring devices and provides appropriate feedback on the user's lifestyle habits based on that data.

[1006] "Example 2 of combining an emotion engine"

[1007] (Claim 1)

[1008] An input means that acquires audio information and converts it into text information,

[1009] An emotion recognition means that performs emotion analysis and acquires emotion data,

[1010] A dialogue generation means that generates dialogue content based on generated text information and emotion data,

[1011] A response output means that converts the generated dialogue content into speech and presents it,

[1012] A task generation means that acquires individual performance data and generates tasks according to cognitive function,

[1013] An analytical method for analyzing lifestyle log data and evaluating health status,

[1014] A notification means that generates and presents notifications to encourage lifestyle improvements based on analysis results,

[1015] A means for adjusting the content of dialogue generated based on emotional data and improving its quality,

[1016] A system that includes this.

[1017] (Claim 2)

[1018] The system according to claim 1, characterized in that it improves the quality of interaction with the user through speech recognition and emotion analysis, and accumulates data on the user's cognitive function.

[1019] (Claim 3)

[1020] The system according to claim 1, characterized by providing appropriate feedback on the user's lifestyle habits based on diverse emotional information.

[1021] "Application example 2 when combining with an emotional engine"

[1022] (Claim 1)

[1023] An acquisition mechanism that obtains voice input from the user and converts it into text data,

[1024] A dialogue generation mechanism that generates appropriate responses using natural language processing based on generated text data,

[1025] A response output mechanism that converts the generated response into speech and presents it to the user,

[1026] A task generation mechanism that acquires user ability data and generates training tasks according to the user's cognitive function,

[1027] An analytical mechanism that analyzes acquired lifestyle record data and evaluates health status,

[1028] A notification mechanism that generates and presents notifications to encourage habit improvement based on analysis results,

[1029] An emotion response mechanism that provides dialogue and feedback that takes the user's mental state into consideration, based on recognized emotion information,

[1030] An evaluation mechanism that monitors changes in cognitive function through data analysis and generates notifications prompting consultation with a medical professional when necessary,

[1031] A system that includes this.

[1032] (Claim 2)

[1033] The system according to claim 1, characterized in that it improves the quality of dialogue with the user by using natural language processing and sentiment analysis, and accumulates data on the user's cognitive function and psychological state.

[1034] (Claim 3)

[1035] The system according to claim 1, characterized in that it acquires user lifestyle record data from multiple detectors and provides appropriate feedback on the user's lifestyle habits based on that data. [Explanation of Symbols]

[1036] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An input means that acquires voice input from the user and converts it into text data, A dialogue generation means that generates an appropriate response using natural language processing based on the generated text data, A response output means that converts the generated response into speech and presents it to the user, A task generation means that acquires user performance data and generates training tasks according to the user's cognitive function, An analytical means for analyzing acquired life log data and evaluating health status, A notification means that generates and presents notifications to encourage lifestyle improvements based on analysis results, An evaluation method that monitors changes in cognitive function through data analysis and generates notifications prompting consultation with a medical professional when necessary, A system that includes this.

2. The system according to claim 1, characterized in that it improves the quality of dialogue with the user by using natural language processing and accumulates data on the user's cognitive function.

3. The system according to claim 1, characterized in that it acquires user life log data from multiple sensors and provides appropriate feedback on the user's lifestyle habits based on that data.

Images