System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving a speech input of a senior citizen; means for converting the received speech into text data; means for transmitting the converted text data to a server; means for analyzing the text data at the server and generating an appropriate response; means for transmitting the generated response in text form from the server to a terminal; and means for converting the received response into speech and notifying the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In Japan's aging society, elderly people are increasingly facing issues such as loneliness, dementia, and health management. For elderly people who tend to be isolated from their families and society, daily conversation and health management are particularly important challenges. Elderly people also require significant support to live their daily lives independently, but there are currently no fully established methods for providing such support. Therefore, to solve these problems, a system is needed that can alleviate elderly loneliness, prevent dementia, and provide support for health management and daily life. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including: means for receiving voice input from an elderly person; means for converting the received voice into text data; means for transmitting the converted text data to a server; means for analyzing the text data at the server and generating an appropriate response; means for transmitting the generated response in text format from the server to a terminal; and means for converting the received response into voice and notifying the user. The system also includes means for collecting user health data, means for transmitting the collected health data to a server, means for analyzing the health data at the server and evaluating the user's health status, means for generating health advice based on the evaluation results, means for transmitting the generated advice from the server to a terminal, and means for converting the received advice into voice and notifying the user. Furthermore, the system provides support for elderly people's daily lives by including means for managing the user's daily schedule, means for receiving the user's voice instructions, means for transmitting voice instructions to a server to acquire schedule information, means for analyzing the acquired schedule information and generating a response for the user, means for transmitting the generated response from the server to a terminal, and means for converting the received response into voice and notifying the user.

[0006] The "means for receiving voice input" refers to a part that has the function of capturing the user's voice using a device such as a microphone and processing it as a digital signal.

[0007] The "means for converting into text data" refers to a part that has the function of converting the received voice signal into text format using natural language processing technology.

[0008] The "means for transmitting to the server" is a part that has the function of transmitting the converted text data and other necessary information to a remote server via a network.

[0009] The "means for analyzing text data on the server and generating an appropriate response" refers to the part of the server that analyzes the received text data using natural language processing and generates an appropriate response in response to the data.

[0010] The "means for transmitting from the server to the terminal in text format" is a part having a network function for transmitting the generated response to the terminal as text data.

[0011] The "means for converting into voice and notifying the user" refers to the part that has the function of converting the received text data into voice using voice synthesis technology and notifying the user through a speaker or the like.

[0012] The "means for collecting health data" is a part that has the function of collecting data from devices that measure the user's health status (e.g., smart scales, smart watches, etc.).

[0013] The "means for transmitting to a server" is a part that has the function of transmitting collected health data to a remote server via a network.

[0014] The "means for analyzing health data and evaluating the user's health condition" refers to the part of the server that evaluates the user's health condition based on the received health data and analyzes the results.

[0015] The "means for generating health advice" is a part that has the function of generating specific health advice for the user based on the results of health data analysis.

[0016] The "means for transmitting from the server to the terminal" is a part having a network function for transmitting the generated health advice from the server to the terminal.

[0017] The "means for managing daily schedules" is a part that has a function for recording and managing the user's daily activities and schedules.

[0018] The "means for receiving voice instructions" is a part that has a function for receiving voice commands from the user.

[0019] The "means for acquiring schedule information" is a part that has a network function for inquiring about and acquiring schedule information stored in a server.

[0020] The "means for generating a response to the user" is a part that has a function for generating an appropriate response to the user based on the acquired schedule information. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention provides a system for providing a conversation partner for elderly people, preventing dementia, managing health, and supporting daily life. Specific embodiments for implementing the present invention will be described below.

[0043] This system includes a terminal for inputting voice from the elderly user, a means for converting voice into text data, a server for analyzing the text data and generating an appropriate response, a means for converting the generated response back into voice and notifying the user, a means for collecting and analyzing the user's health data and providing appropriate health advice, and a means for managing and notifying the elderly user's daily schedule.

[0044] Conversation Partners / Dementia Prevention

[0045] 1. Voice input

[0046] User: Talks to a smart speaker at home, for example, asking, "What's the weather like today?"

[0047] Terminal: Receives the user's voice and processes it as voice data.

[0048] 2. Converting voice data to text

[0049] Terminal: Converts received voice data into text data using voice recognition technology.

[0050] Terminal: Sends the converted text data to the server.

[0051] 3. Conversation Analysis and Response Generation

[0052] Server: Analyzes the received text data using natural language processing techniques and generates an appropriate response. In this example, it queries a weather API to obtain weather information.

[0053] Server: Generate a response such as "It's a sunny day today. Why don't you go for a walk?"

[0054] 4. Audio Output

[0055] Server: Sends the generated response to the terminal as text data.

[0056] Terminal: Text data is converted into speech using speech synthesis technology and notified to the user.

[0057] health care

[0058] 1. Health Data Collection

[0059] User: Weighs himself / herself and measures his / her blood pressure, and records the data on a smart device (e.g., a smartwatch or a scale).

[0060] Terminal: Obtain health data (e.g., weight "68kg") from a smart device.

[0061] 2. Data submission and analysis

[0062] Terminal: Sends acquired health data to the server.

[0063] Server: Receives health data and compares and analyzes it with past data.

[0064] 3. Health advice generation

[0065] Server: Based on the analysis results, the server generates specific health advice for the user. For example, it generates advice such as, "We recommend that you exercise for about 30 minutes today."

[0066] Server: Sends the generated advice to the terminal as text data.

[0067] 4. Notification

[0068] Terminal: Converts text data into speech and notifies the user.

[0069] Daily life support

[0070] 1. Receiving voice instructions

[0071] User: Speaks to a smart speaker and says, "Tell me what's on my schedule for today."

[0072] Terminal: Receives the user's voice and converts the voice data into text.

[0073] 2. Obtaining schedule information

[0074] Terminal: Sends the converted text data to the server.

[0075] Server: Queries the user's schedule database and retrieves the schedule for the day.

[0076] 3. Response Generation

[0077] Server: Based on the obtained schedule information, it generates a response such as, "I have a hospital appointment at 10:00. I plan to go shopping at the supermarket at 3:00 p.m."

[0078] Server: Sends the generated response to the terminal as text data.

[0079] 4. Notification

[0080] Terminal: Converts text data into speech and notifies the user.

[0081] As described above, this system aims to provide the elderly with daily support from various angles and improve their quality of life. By combining technologies such as voice recognition, natural language processing, health data analysis, and smart device integration, this invention realizes a support system that is highly convenient for the elderly.

[0082] The processing flow will be explained below.

[0083] Conversation Partners / Dementia Prevention

[0084] Voice input and response generation

[0085] Step 1:

[0086] User: Asks their smart speaker at home, "What's the weather like today?"

[0087] Step 2:

[0088] Terminal: Receives the user's voice and processes it as voice data.

[0089] Step 3:

[0090] Terminal: Converts received voice data into text data using voice recognition technology.

[0091] Step 4:

[0092] Terminal: Sends the converted text data to the server.

[0093] Step 5:

[0094] Server: Analyzes the received text data using natural language processing technology. Based on the analysis results, queries the weather API to obtain weather information.

[0095] Step 6:

[0096] Server: Receives the response from the weather API and generates a response (e.g., "It's a sunny day today. Why don't you go for a walk?").

[0097] Step 7:

[0098] Server: Sends the generated response to the terminal as text data.

[0099] Step 8:

[0100] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[0101] health care

[0102] Step 1:

[0103] User: Weighs themselves and measures their blood pressure, and records the data on a smart device (e.g., a smartwatch or scale).

[0104] Step 2:

[0105] Terminal: Health data (e.g., "Weight: 68 kg") is acquired from the smart device via Bluetooth communication, etc.

[0106] Step 3:

[0107] Terminal: Sends acquired health data to the server.

[0108] Step 4:

[0109] Server: Analyzes the received health data, compares it with past data, and evaluates the user's health condition.

[0110] Step 5:

[0111] Server: Based on the evaluation results, generate specific health advice for the user (e.g., "We recommend that you exercise for about 30 minutes today").

[0112] Step 6:

[0113] Server: The generated health advice is sent to the terminal as text data.

[0114] Step 7:

[0115] Terminal: Converts received text data into speech and notifies the user.

[0116] Daily life support

[0117] Step 1:

[0118] User: Speaks to a smart speaker and says, "Tell me what's on my schedule for today."

[0119] Step 2:

[0120] Terminal: Receives the user's voice and processes it as voice data.

[0121] Step 3:

[0122] Terminal: Converts received voice data into text data using voice recognition technology.

[0123] Step 4:

[0124] Terminal: Sends the converted text data to the server.

[0125] Step 5:

[0126] Server: Analyzes the received text data and queries the user's schedule database to obtain the schedule for the day.

[0127] Step 6:

[0128] Server: Generates a response based on the obtained schedule information (e.g., "I have a doctor's appointment at 10:00. I plan to go grocery shopping at 3:00 PM.").

[0129] Step 7:

[0130] Server: Sends the generated response to the terminal as text data.

[0131] Step 8:

[0132] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[0133] Through the above processing steps, the present invention provides support to help elderly people live healthy and fulfilling lives without feeling lonely.

[0134] Example 1

[0135] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] Currently, support systems for elderly people to smoothly lead their daily lives are inadequate. There is a particular need for comprehensive support that provides a wide range of support, including simple voice-based operation, health management, and schedule management. Furthermore, there is a lack of systems that allow elderly people to understand their own health status and receive appropriate advice. To address these issues, the present invention aims to provide a comprehensive support system based on voice input by elderly people.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0138] In this invention, the server includes means for receiving a user's voice input, means for converting the received voice into text data, means for transmitting the converted text data to the processing device, means for analyzing the text data in the processing device and generating an appropriate response, means for transmitting the generated response in text format from the processing device to the display device, means for converting the received response into voice and notifying the user, means for collecting the user's health data, means for transmitting the collected health data to the processing device, means for analyzing the health data in the processing device and evaluating the user's health condition, means for generating health advice based on the evaluation results, means for transmitting the generated advice from the processing device to the display device, means for converting the received advice into voice and notifying the user, means for managing the user's daily schedule, means for receiving the user's voice instructions, means for transmitting the voice instructions to the processing device and acquiring schedule information, means for analyzing the acquired schedule information and generating a response to the user, means for transmitting the generated response from the processing device to the display device, and means for converting the received response into voice and notifying the user. This enables comprehensive support for the daily lives of elderly people in various aspects.

[0139] A "means for receiving voice input" is a device that recognizes the voice uttered by the user and captures it as digital data.

[0140] "Means for converting voice into text data" refers to an algorithm or system that uses voice recognition technology to convert received voice data into character string data.

[0141] The "means for transmitting text data to a processing device" is a function for transferring the converted text data to a device for performing other processing using a communication means such as a network.

[0142] "Means for analyzing text data using a processing device and generating an appropriate response" refers to a system that uses natural language processing technology to analyze the meaning and intent of received text data and generate an appropriate response to the user.

[0143] The "means for transmitting the generated response in text format from the processing device to the display device" is a function for transmitting the generated response in text data to another device for display.

[0144] The "means for converting the received response into voice and notifying the user" is a device that converts the response received as text data into voice using voice synthesis technology and transmits it to the user.

[0145] A "means for collecting health data" is a device that has the function of measuring and recording numerical health data such as a user's weight, blood pressure, and heart rate.

[0146] The "means for transmitting collected health data to a processing device" is a system that transmits collected health data to another processing device via a network.

[0147] "Means for analyzing health data using a processing device and assessing the user's health condition" refers to an algorithm or system that analyzes collected health data and understands and assesses the user's health condition.

[0148] The "means for generating health advice based on the evaluation results" is a system that generates specific advice for maintaining and improving the user's health based on the analysis results.

[0149] The "means for transmitting the generated advice from the processing device to the display device" is a function for transmitting the advice to the display device in order to notify the user of the generated health advice.

[0150] "Means for managing daily schedules" refers to a system that records and manages a user's daily plans and schedules.

[0151] The "means for receiving voice instructions" is a device that recognizes voice instructions from the user and captures them as digital data.

[0152] The "means for transmitting voice instructions to the processing device and acquiring schedule information" is a system for transmitting voice instructions to the processing device and acquiring the user's schedule information.

[0153] The "means for analyzing the acquired schedule information and generating a response to the user" refers to an algorithm or system that generates an appropriate response based on the schedule information.

[0154] The "means for transmitting the generated response from the processing device to the display device" is a function for transmitting the generated response to the display device and displaying it to the user.

[0155] The present invention provides a system for providing a conversation partner for elderly people, preventing dementia, managing health, and supporting daily life. Specific embodiments for implementing the present invention will be described below.

[0156] This system includes a terminal for inputting voice, a means for converting voice into text data, a server for analyzing the text data and generating an appropriate response, a means for converting the generated response into voice and notifying the user, a means for analyzing health data and providing appropriate health advice, and a means for managing and notifying the user of daily schedules.

[0157] The voice input in this system uses a smart speaker, such as Amazon Echo or Google Home, to capture the user's voice. The captured voice is then converted into text data using the Google Cloud Speech-to-Text API.

[0158] The text data is sent to a server, which then analyzes it using natural language processing technology, specifically a generative AI model such as GPT-4, to generate an appropriate response.

[0159] The generated response is then sent back to the device (such as a smart speaker or smartphone) and converted into audio using the Google Cloud Text-to-Speech API, which is then notified to the user.

[0160] Specific examples

[0161] Conversation Partners / Dementia Prevention

[0162] Prompt: "What's the weather like today?"

[0163] Response: "It's a beautiful sunny day today. Why don't you go for a walk?"

[0164] health care

[0165] Users' health data is collected using devices such as smartwatches and digital scales. The collected data is sent to a server via the device and analyzed there. Based on the analysis results, health advice for the user is generated, sent back to the device, and notified via voice.

[0166] Prompt: "I've weighed myself. What should I do next?"

[0167] Response: "According to our records, you weigh 140 lbs. I suggest you exercise for about 30 minutes today."

[0168] Daily life support

[0169] A smart speaker is used to manage daily schedules. When a user says, "Tell me what's on my schedule for today," the voice is converted into text data and sent to a server. The server retrieves the schedule using the Google Calendar API or similar, generates a response, and notifies the user by voice.

[0170] Prompt: "What's on your agenda for today?"

[0171] Response: "I have a doctor's appointment at 10:00. I'll be grocery shopping at 3:00."

[0172] As described above, this system aims to provide the elderly with daily support from various angles and improve their quality of life. By combining technologies such as voice recognition, natural language processing, health data analysis, and smart device integration, this invention realizes a support system that is highly convenient for the elderly.

[0173] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0174] Conversation Partners / Dementia Prevention

[0175] Step 1:

[0176] A user speaks to a smart speaker in their home, asking, for example, "What's the weather like today?"

[0177] Input: User's voice

[0178] Output: Audio data captured by the smart speaker

[0179] Step 2:

[0180] The device (smart speaker) converts the received audio into text data in real time using the Google Cloud Speech-to-Text API.

[0181] Input: Audio data

[0182] Output: Text data

[0183] Step 3:

[0184] The terminal transmits the converted text data to the server.

[0185] Input: Text data

[0186] Output: Text data sent to the server

[0187] Step 4:

[0188] The server analyzes the received text data using a natural language processing model (e.g., GPT-4). This analysis allows it to understand the user's intent. Next, the server queries an external weather API (e.g., OpenWeatherMap API) to obtain weather information.

[0189] Input: Text data

[0190] Output: Response text generated based on weather information

[0191] Step 5:

[0192] The server transmits the generated response to the terminal as text data.

[0193] Input: Response text

[0194] Output: Text data sent to the terminal

[0195] Step 6:

[0196] The device converts the received text data into audio using the Google Cloud Text-to-Speech API and notifies the user.

[0197] Input: Text data

[0198] Output: Audio data (notification audio)

[0199] health care

[0200] Step 1:

[0201] The user measures their weight and blood pressure and records the data on a smart device (e.g., a smart watch).

[0202] Input: Health data (weight, blood pressure, etc.)

[0203] Output: Data recorded on a smart device

[0204] Step 2:

[0205] The terminal (smartphone) acquires health data from the smart device via BLE (Bluetooth Low Energy).

[0206] Input: Data communication from smart devices

[0207] Output: Health data collected on the device

[0208] Step 3:

[0209] The terminal transmits the acquired health data to the server.

[0210] Input: Health data

[0211] Output: Health data sent to the server

[0212] Step 4:

[0213] The server compares and analyzes the received health data with past data to evaluate the user's health condition.

[0214] Input: Health data

[0215] Output: Analysis results (health assessment data)

[0216] Step 5:

[0217] Based on the analysis results, the server generates specific health advice for the user, such as "We recommend you exercise for about 30 minutes today."

[0218] Input: Health assessment data

[0219] Output: Generated health advice

[0220] Step 6:

[0221] The server transmits the generated advice to the terminal as text data.

[0222] Enter: Health Advice

[0223] Output: Text data sent to the terminal

[0224] Step 7:

[0225] The terminal converts the text data into voice and notifies the user.

[0226] Input: Text data

[0227] Output: Audio data (notification audio)

[0228] Daily life support

[0229] Step 1:

[0230] The user speaks to the smart speaker and says, "Tell me what's on my schedule for today."

[0231] Input: User's voice

[0232] Output: Audio data captured by the smart speaker

[0233] Step 2:

[0234] The device (smart speaker) converts the received voice data into text data using the Google Cloud Speech-to-Text API.

[0235] Input: Audio data

[0236] Output: Text data

[0237] Step 3:

[0238] The terminal transmits the text data to the server.

[0239] Input: Text data

[0240] Output: Text data sent to the server

[0241] Step 4:

[0242] The server analyzes the received text data and queries the user's schedule database (e.g., Google Calendar) to obtain the schedule for that day.

[0243] Input: Text data

[0244] Output: Schedule information

[0245] Step 5:

[0246] Based on the schedule information it has obtained, the server generates a response such as, "I have a hospital appointment at 10:00. I plan to go shopping at the supermarket at 3:00 p.m."

[0247] Input: Schedule information

[0248] Output: Generated response text data

[0249] Step 6:

[0250] The server transmits the generated response to the terminal as text data.

[0251] Input: Response text data

[0252] Output: Text data sent to the terminal

[0253] Step 7:

[0254] The terminal converts the text data into voice and notifies the user.

[0255] Input: Text data

[0256] Output: Audio data (notification audio)

[0257] (Application example 1)

[0258] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0259] Today's elderly people need support in many aspects of daily life. They may have difficulty managing their health and daily tasks, as well as having difficulty shopping and socializing. Shopping in brick-and-mortar stores can be particularly challenging, as they struggle to find the products they need and have difficulty accessing sales information. This often leads to a decline in the quality of life for elderly people and a sense of isolation. The present invention aims to address these challenges and improve the quality of life for elderly people.

[0260] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0261] In this invention, the server includes means for receiving voice input from the elderly person, means for converting the received voice into text data, means for transmitting the converted text data to the server, means for analyzing the text data in the server and generating an appropriate response, means for transmitting the generated response in text format from the server to the terminal, means for converting the received response into voice and notifying the user, and means for acquiring and providing sale information using a generative AI model, thereby enabling elderly people to shop more smoothly in physical stores.

[0262] definition statement

[0263] "Elderly people" generally refer to people aged 65 and over who often require special assistance and support in their daily lives.

[0264] A "means for receiving voice input" is a device that allows a user to issue voice commands and has the ability to record and process the voice as digital data.

[0265] The "means for converting voice into text data" is a system that uses voice recognition technology to convert input voice into text data in real time.

[0266] "Means for transmitting text data to a server" refers to a function for transmitting converted text data to a server located at a remote location via a network.

[0267] "Means for analyzing text data and generating appropriate responses" refers to a function that uses natural language processing technology to understand the input text and generate the information or response the user is looking for.

[0268] "Means for transmitting the generated response in text format from the server to the terminal" refers to the function of transmitting the analysis results from the server to the user's terminal as text data.

[0269] The "means for converting the received response into voice and notifying the user" is a system that has the function of converting text data back into voice and conveying that voice to the user.

[0270] A "generative AI model" is an artificial intelligence model that has been trained in advance using large amounts of data and is capable of automatically generating a variety of responses and suggestions.

[0271] The "means for acquiring and providing sale information" refers to a system that utilizes inventory management systems and marketing data from physical stores to acquire the latest sale information and provide it to users.

[0272] MODE FOR CARRYING OUT THE INVENTION

[0273] The present invention is a system that provides sales information based on voice input and supports shopping in brick-and-mortar stores. This system uses speech recognition, text analysis, natural language processing, and generative AI models to generate specific responses and provide them to users, thereby supporting shopping for elderly people and others. Specific embodiments of the present invention are described below.

[0274] System configuration

[0275] Hardware:

[0276] Smartphone: Receives voice input, sends the speech recognition results to the server, and converts the text response from the server into speech to notify the user.

[0277] Server: Converts voice data into text data, analyzes it, and generates a response.

[0278] software:

[0279] Speech recognition library (e.g., speech_recognition library): Converts voice input into text data.

[0280] Natural language processing model: Analyzes input text data and generates appropriate responses.

[0281] Generative AI model: Used to determine special offers.

[0282] Text-to-speech libraries (e.g. gTTS library): Convert text responses into speech.

[0283] Data processing and calculation

[0284] Receiving and converting voice input:

[0285] The user speaks into the smartphone, and a speech recognition library (e.g., speech_recognition library) converts this speech into text data and sends it to the server.

[0286] Analyzing text data:

[0287] The server then analyzes the converted text data using natural language processing technology, determining the user's intent from the analysis results and identifying requests such as sales information.

[0288] Get sale information and generate a response:

[0289] The server uses the generative AI model to retrieve sales information from the physical store's inventory management system and marketing database, and then generates a response containing the information and suggestions the user is looking for.

[0290] Response transcription and announcement:

[0291] The generated response is sent to the smartphone as text data and converted into speech using a speech synthesis library (e.g., gTTS library). The converted speech is then notified to the user through the smartphone's speaker.

[0292] Examples and prompts

[0293] Examples:

[0294] A user speaks to their smartphone, "What are the specials this week?" The smartphone converts this speech into text data and sends it to a server. The server analyzes the text data and uses a generative AI model to obtain the special sale information. Based on the obtained special sale information, it generates a response such as, "This week's special is apples, 100 yen. They're in the vegetable section." The smartphone converts this response into speech and notifies the user.

[0295] Example prompt sentence:

[0296] "What are the specials this week?"

[0297] "Which department should I go to?"

[0298] This system will enable seniors to easily obtain shopping information at brick-and-mortar stores, ensuring they don't miss out on special offers. It is also expected that the natural voice interaction will make shopping smoother and more enjoyable.

[0299] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0300] Program processing steps

[0301] Step 1:

[0302] User speaks:

[0303] The user speaks a question or request into the smartphone, for example, "What are the specials this week?" This is the initial input to the system.

[0304] Step 2:

[0305] Audio to text conversion:

[0306] The user's voice is received by the smartphone's microphone and converted into text data using the speech_recognition library. At this stage, the input voice is converted into string data.

[0307] Step 3:

[0308] Sending text data to the server:

[0309] The text data obtained by speech recognition is sent from the smartphone to the server, which then forwards the request to the server via the network.

[0310] Step 4:

[0311] Text data analysis and response generation:

[0312] The server analyzes the received text data using natural language processing technology. Based on the analyzed text data, it understands the user's request and uses a generative AI model to obtain special sale information. It then generates a response based on the special sale information and prepares it in text data format.

[0313] Step 5:

[0314] Sending the generated response from the server to the device:

[0315] The generated text response is then sent from the server to the smartphone via the network, and the analysis results are returned to the user's device.

[0316] Step 6:

[0317] Text response transcription:

[0318] The smartphone converts the received text data into speech using the gTTS library, and outputs synthesized speech data based on the input text.

[0319] Step 7:

[0320] Notify user of voice response:

[0321] The converted response is then broadcast to the user through the smartphone's speaker, allowing the user to hear the response to their question.

[0322] Specific processing operations

[0323] Step 1:

[0324] The user speaks to the smartphone, "What are the specials this week?" The voice is received by the smartphone.

[0325] Step 2:

[0326] The voice data received by the smartphone's microphone is converted into a string using the speech_recognition library, and the text data "What are the specials this week?" is generated.

[0327] Step 3:

[0328] The text data generated by the speech recognition is sent from the smartphone to the server, where data communication takes place via the network.

[0329] Step 4:

[0330] The server uses natural language processing technology to analyze the received text data and understand the user's request. It then uses a generative AI model to retrieve sales information from the physical store's inventory management system and generates a response such as, "This week's sales item is apples, priced at 100 yen. They're in the vegetable section."

[0331] Step 5:

[0332] The text response generated by the server is sent over the network to the smartphone, which receives the text response.

[0333] Step 6:

[0334] The smartphone converts the received text data into speech using the gTTS library, generating speech data such as, "This week's special offer is apples, priced at 100 yen. They're in the vegetable section."

[0335] Step 7:

[0336] The user is notified of the voice-converted response through the smartphone's speaker, and can listen to the voice response.

[0337] Through the above process, shopping support for the elderly is realized.

[0338] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0339] This invention is a system that supports the lives of the elderly, combining voice recognition, natural language processing, health data analysis, smart device integration, and an emotion engine. This system provides various functions to help the elderly relieve loneliness, prevent dementia, and support health management.

[0340] 1. Conversation partner / dementia prevention

[0341] Voice input and response generation

[0342] When a user asks a question or speaks to a smart speaker, the voice input is received by the device. The device converts the voice data into text data and sends it to the server. The server analyzes the text data and generates an appropriate response. This process also incorporates an emotion engine that recognizes emotions from the user's voice. The tone and content of the response are adjusted based on the recognized emotion. The generated response is sent from the server to the device in text format, where it is converted into voice and notified to the user.

[0343] Examples:

[0344] User: "What's the weather like today?"

[0345] Terminal: Receives audio, converts it into text, and sends it to the server.

[0346] Server: Analyzes text data and obtains weather information. The emotion engine recognizes the user's emotion (e.g., feeling down), and generates a response based on that, such as "It's sunny today. Maybe you should take a walk to change your mood."

[0347] Terminal: Converts the response from the server into voice and notifies the user.

[0348] 2. Health management

[0349] Health data collection and analysis

[0350] Users use smart devices to collect health data. For example, they measure their body temperature and blood pressure with a smartwatch. The collected health data is sent to a server via the device. The server analyzes the received data and evaluates the user's health status. An emotion engine is also incorporated into this evaluation, and health advice is generated taking the user's emotional state into account.

[0351] Examples:

[0352] User: Weighs himself and records the data on a smart scale.

[0353] Terminal: Receives data from the scale and sends it to the server.

[0354] Server: Analyzes the data and generates health advice such as, "You seem a little low today. I recommend some light exercise."

[0355] Terminal: Converts the advice into voice and notifies the user.

[0356] 3. Daily life support

[0357] Schedule management and coordination

[0358] When a user asks a smart speaker about their daily schedule, the device receives the voice, converts it into text, and sends it to a server. The server then checks the user's schedule and uses an emotion engine to generate a response that takes into account the user's current emotional state.

[0359] Examples:

[0360] User: "What's on my schedule for today?"

[0361] Terminal: Receives audio, converts it into text, and sends it to the server.

[0362] Server: Checks the schedule. Generates a response saying, "I have a doctor's appointment at 10:00. I'm going to the supermarket at 3:00 PM." The emotion engine determines that the user is feeling stressed, and gives additional advice, saying, "Try not to push yourself too hard, and take things slowly."

[0363] Terminal: Converts responses and advice into voice and notifies the user.

[0364] As described above, the present invention provides a multi-functional support system that helps elderly people avoid loneliness and lead healthy and fulfilling lives. By adding emotion recognition functionality, the system achieves more advanced and personalized support.

[0365] The processing flow will be explained below.

[0366] Conversation Partners / Dementia Prevention

[0367] Voice input and response generation

[0368] Step 1:

[0369] User: Asks their smart speaker at home, "What's the weather like today?"

[0370] Step 2:

[0371] Terminal: Receives the user's voice and processes it as voice data.

[0372] Step 3:

[0373] Terminal: Converts received voice data into text data using voice recognition technology.

[0374] Step 4:

[0375] Terminal: Sends the converted text data to the server.

[0376] Step 5:

[0377] Server: Analyzes the received text data using natural language processing technology. Based on the analysis results, queries the weather API to obtain weather information.

[0378] Step 6:

[0379] Server: Receives the response from the weather API and generates text data.

[0380] Step 7:

[0381] Server: Recognizes emotions from the user's voice using the emotion engine.

[0382] Step 8:

[0383] Server: Adjusts the content and tone of the response based on the recognized emotion. For example, if a user is feeling down, the server generates a response like, "It's a sunny day today. Why don't you take a walk to cheer yourself up?"

[0384] Step 9:

[0385] Server: Sends the generated response to the terminal as text data.

[0386] Step 10:

[0387] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[0388] health care

[0389] Step 1:

[0390] User: Records health data such as weight and blood pressure on a smart device (e.g., smartwatch or scale).

[0391] Step 2:

[0392] Terminal: Health data is acquired from smart devices using communication methods such as Bluetooth.

[0393] Step 3:

[0394] Terminal: Sends acquired health data to the server.

[0395] Step 4:

[0396] Server: Analyzes the received data and evaluates the user's health condition.

[0397] Step 5:

[0398] Server: Analyzes the user's emotional data simultaneously using an emotion engine.

[0399] Step 6:

[0400] Server: Evaluates the overall physical and mental health status and generates appropriate health advice. For example, it generates advice such as, "You're feeling good today, but you seem a little tired. I recommend some light exercise."

[0401] Step 7:

[0402] Server: The generated health advice is sent to the terminal as text data.

[0403] Step 8:

[0404] Terminal: Converts received text data into speech and notifies the user.

[0405] Daily life support

[0406] Step 1:

[0407] User: Speaks to a smart speaker and says, "Tell me what's on my schedule for today."

[0408] Step 2:

[0409] Terminal: Receives the user's voice and processes it as voice data.

[0410] Step 3:

[0411] Terminal: Converts received voice data into text data using voice recognition technology.

[0412] Step 4:

[0413] Terminal: Sends the converted text data to the server.

[0414] Step 5:

[0415] Server: Analyzes the received text data and queries the user's schedule database to obtain the schedule for the day.

[0416] Step 6:

[0417] Server: Analyzes the acquired schedule information and generates text data.

[0418] Step 7:

[0419] Server: Uses an emotion engine to recognize the user's current emotional state.

[0420] Step 8:

[0421] Server: Adjusts the tone and content of schedule notifications based on the recognized emotion. For example, if it determines that the user is feeling stressed, it generates a response such as, "You have a doctor's appointment at 10 o'clock today. Take it easy and take it easy."

[0422] Step 9:

[0423] Server: Sends the generated response to the terminal as text data.

[0424] Step 10:

[0425] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[0426] Through the above processing steps, the present invention provides a multi-functional support system that helps elderly people avoid loneliness and live healthy and fulfilling lives. The addition of emotion recognition functionality provides more advanced and personalized support.

[0427] Example 2

[0428] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0429] Elderly people often feel lonely in their daily lives and find health care a burden. Furthermore, there is a lack of support for dementia prevention and daily schedule management. These situations can reduce the quality of life for elderly people and lead to further health risks. Therefore, there is a need for the development of multifunctional and personalized systems to support the daily lives of elderly people.

[0430] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0431] In this invention, the server includes means for receiving voice input from the elderly person, means for converting the received voice into character data, means for transmitting the converted character data to a central control unit, means for analyzing the character data in the central control unit and generating an appropriate response (including an emotion engine), means for transmitting the generated response in character form from the central control unit to a terminal, and means for converting the received response into voice and notifying the elderly person. This reduces the elderly person's sense of loneliness, supports dementia prevention, and reduces the burden of health management.

[0432] "Means for receiving voice input" refers to devices or technology that detects voices made by the senior and processes the voice signals.

[0433] "Means for converting into character data" refers to technology that analyzes received voice signals, recognizes phonemes and words, and converts them into character strings.

[0434] "Means for transmitting to a server" refers to a device or technology that transmits the converted character data to a central control device (server) via a communications network.

[0435] "Central Controller" means a central control unit or system that analyzes received text data and generates an appropriate response.

[0436] "Means for generating an appropriate response" refers to technology that uses specific algorithms or emotion engines based on received data to generate an appropriate response to the user.

[0437] An "emotion engine" refers to an algorithm or system that analyzes emotions from received voice or data and generates a response based on those emotions.

[0438] "Means for sending to the terminal" refers to a device or technique for sending a response in the form of character data generated by the central control unit to the user's terminal.

[0439] "Means for converting the received response into voice" refers to a device or technology that synthesizes text data sent to the terminal into voice and notifies the user.

[0440] "Means for collecting health data" refers to devices and technologies for measuring health information such as a user's weight, body temperature, and blood pressure.

[0441] "Means for analyzing and evaluating health data" refers to algorithms or systems for analyzing collected health data and evaluating the user's health status.

[0442] The term "means for generating health advice" refers to a technology for generating health advice for a user based on the evaluation results of the health data.

[0443] "Means for managing daily schedules" refers to devices and technologies for managing a user's daily plans and tasks.

[0444] The term "means for receiving voice instructions" refers to a device or technology that detects voice instructions regarding the schedule given by the user and processes the voice signals.

[0445] "Means for obtaining schedule information" refers to devices and techniques for querying and obtaining the user's daily schedule information from the central control device.

[0446] The term "means for analyzing schedule information and generating a response" refers to a technique for analyzing acquired schedule information and generating an appropriate schedule-related response to the user.

[0447] This invention is a system for supporting the lives of the elderly, combining voice recognition, natural language processing, health data analysis, smart device integration, and an emotion engine. The system's main functions include serving as a conversation partner, health management, and daily schedule management for everyday use by the elderly.

[0448] The components of this system are:

[0449] 1. Conversation partner / dementia prevention

[0450] When a user asks a question or speaks to a smart speaker, the voice input is received by the device. The device converts the voice data into text data (using, for example, the Google Speech-to-Text API) and sends the converted data to a server. The server analyzes the text data using a natural language processing engine (for example, GPT-3) and generates an appropriate response. It also incorporates an emotion engine (for example, IBM Watson Tone Analyzer) that recognizes the user's emotions and adjusts the tone and content of the response. The generated response is sent from the server to the device in text format, where it is converted into speech and notified to the user (for example, using Amazon Polly).

[0451] Examples:

[0452] User: "What's the weather like today?"

[0453] Device: Receives audio, converts it to text using the Google Speech-to-Text API, and sends it to the server.

[0454] Server: Analyzes text data using GPT-3 and obtains weather information. The emotion engine recognizes the user's emotions and generates a response such as, "It's sunny today. It might be nice to take a walk to change your mood."

[0455] Terminal: The response from the server is converted into speech using Amazon Polly and notified to the user.

[0456] 2. Health management

[0457] Users collect their health data using a smart device. For example, they measure their body temperature and blood pressure with a smartwatch and send the data to the device. The device then sends the received data to a server, which analyzes it using a health data analysis algorithm (e.g., Google Fit API). An emotion engine is also incorporated into this evaluation, which generates health advice taking the user's emotional state into account. The advice is then sent from the server to the device, which converts it into audio and notifies the user.

[0458] Examples:

[0459] User: Weighs himself and records the data on a smart scale.

[0460] Terminal: Receives data from the scale and sends it to the server.

[0461] Server: Analyzes the data using the Google Fit API and generates health advice such as, "You seem a little low today. I recommend some light exercise."

[0462] Device: The advice is converted into speech using Amazon Polly and notified to the user.

[0463] 3. Daily life support

[0464] When a user speaks to a smart speaker to check their daily schedule, the device receives the voice, converts it into text data, and sends it to the server. The server then checks the user's daily schedule information and uses an emotion engine to generate a response that takes into account the user's current emotional state. The generated response is then sent from the server to the device, which converts it into voice and notifies the user.

[0465] Examples:

[0466] User: "What's on my schedule for today?"

[0467] Device: Receives audio, converts it to text using the Google Speech-to-Text API, and sends it to the server.

[0468] Server: Checks the schedule and generates a response saying, "I have a doctor's appointment at 10:00. I'm going to the supermarket at 3:00 PM." The emotion engine determines that the user is stressed and provides additional advice, saying, "Try not to push yourself too hard, and take things slowly."

[0469] Device: Responses and advice are converted into speech using Amazon Polly and communicated to the user.

[0470] Example prompts for generative AI models

[0471] "Please explain the function of an elderly life support system that uses voice recognition and an emotion engine to respond to user questions. Please also include specific processing procedures and examples when a user asks a question to a smart speaker."

[0472] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0473] 1. Conversation partner / dementia prevention

[0474] Step 1:

[0475] A user speaks to a smart speaker, such as "What's the weather like today?"

[0476] (Input) User's voice data

[0477] (Output) User's voice data

[0478] Step 2:

[0479] The device receives the voice data and converts it into text using voice recognition software (e.g., Google Speech-to-Text API).

[0480] (Input) User's voice data

[0481] (Output) The user's voice is converted into text data.

[0482] Step 3:

[0483] The terminal sends the converted text data to the server using a secure protocol such as HTTPS.

[0484] (Input) Character data

[0485] (Output) Character data sent to the server

[0486] Step 4:

[0487] The server uses a natural language processing engine (e.g., GPT-3) to analyze the text data, and at the same time, an emotion engine (e.g., IBM Watson Tone Analyzer) recognizes the user's emotions, which allows it to generate more appropriate responses.

[0488] (Input) Character data

[0489] (Data processing) Analyze emotions with an emotion engine and analyze data with a natural language processing engine

[0490] (Output) Analysis results and response data based on them

[0491] Step 5:

[0492] The server generates a response and sends it to the terminal in character data format.

[0493] (Input) Generated response data

[0494] (Output) Response data sent to the terminal

[0495] Step 6:

[0496] The response data received by the device is converted into speech using speech synthesis software (e.g., Amazon Polly) and notified to the user.

[0497] (Input) Response data

[0498] (Output) Response data converted to audio

[0499] 2. Health management

[0500] Step 1:

[0501] A user records health data using a health device (e.g., smart watch, smart scale, etc.).

[0502] (Input) User's health data (weight, body temperature, blood pressure, etc.)

[0503] (Output) Health data recorded on the health device

[0504] Step 2:

[0505] The terminal receives data from the health device using communication methods such as Bluetooth and Wi-Fi.

[0506] (Input) Data from health devices

[0507] (Output) Health data received by the device

[0508] Step 3:

[0509] The health data received by the device is sent to the server.

[0510] (Input) Health data

[0511] (Output) Health data sent to the server

[0512] Step 4:

[0513] The server uses a healthcare analysis algorithm (e.g., Google Fit API) to analyze the received health data, while also taking into account the user's emotional state using an emotion engine.

[0514] (Input) Health data

[0515] (Data processing) Health data analysis and emotional state assessment

[0516] (Output) Analysis results and health advice data

[0517] Step 5:

[0518] The server sends the generated health advice to the terminal.

[0519] (Input) Health advice data

[0520] (Output) Health advice data sent to the device

[0521] Step 6:

[0522] The health advice data received by the terminal is converted into voice using voice synthesis software and notified to the user.

[0523] (Input) Health advice data

[0524] (Output) Health advice converted into audio

[0525] 3. Daily life support

[0526] Step 1:

[0527] A user asks a smart speaker about their daily schedule, such as "Tell me what's on my schedule for today."

[0528] (Input) User's voice data

[0529] (Output) User's voice data

[0530] Step 2:

[0531] The device receives the audio and converts it into text data (e.g., Google Speech-to-Text API).

[0532] (Input) User's voice data

[0533] (Output) The user's voice is converted into text data.

[0534] Step 3:

[0535] The terminal sends character data to the server.

[0536] (Input) Character data

[0537] (Output) Character data sent to the server

[0538] Step 4:

[0539] The server analyzes the text data and queries the daily schedule database, while the emotion engine considers the user's current emotional state.

[0540] (Input) Character data, daily schedule database

[0541] (Data processing) Check schedule and evaluate emotional state

[0542] (Output) Schedule confirmation results and appropriate response data

[0543] Step 5:

[0544] The server sends the generated response data to the terminal.

[0545] (Input) Response data

[0546] (Output) Response data sent to the terminal

[0547] Step 6:

[0548] The response data received by the terminal is converted into voice using voice synthesis software and notified to the user.

[0549] (Input) Response data

[0550] (Output) The voiced response

[0551] (Application example 2)

[0552] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0553] In modern society, the safety and health management of elderly people has become an important issue. Elderly people living alone, in particular, need to respond quickly to emergencies and emergencies at home, and also need to alleviate feelings of loneliness and receive emotional care. However, current systems are unable to comprehensively meet these needs. Furthermore, technology for providing personalized support based on emotional state is immature. An advanced, integrated system to solve these issues is needed.

[0554] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice input from the elderly person, means for converting the received voice into text data, means for transmitting the converted text data to the server, means for analyzing the text data in the server and generating an appropriate response, means for transmitting the generated response in text format from the server to the terminal, means for converting the received response into voice and notifying the user, means for detecting abnormalities in the home and reporting if an abnormality is detected, means for making an emergency call based on the elderly person's voice instruction, and means for recognizing the user's emotional state and generating a response accordingly. This ensures the safety of the elderly person at home and enables rapid response in emergencies. Furthermore, personalized responses based on the elderly person's emotional state can alleviate feelings of loneliness and provide emotional care.

[0555] "Elderly" refers to individuals who are aging, especially those in an age group where health care and safety are of paramount importance.

[0556] "Audio input" refers to an audio signal collected using a microphone or other audio collection device.

[0557] "Text data" refers to data in the form of a string of characters generated from voice input or other digital data.

[0558] A "server" refers to a computer system that provides data processing and data storage over a network.

[0559] "Analysis" refers to the process of examining, interpreting, and evaluating data.

[0560] "Response" refers to information generated in response to a user input or request.

[0561] "Transmission" refers to the act of moving data from one point to another.

[0562] "Domestic anomalies" refers to abnormal or unexpected events or changes in the normal home environment.

[0563] "Reporting" refers to the act of communicating information about an anomaly or event to others.

[0564] "Emergency notification" refers to the prompt notification of pre-established contacts or organizations in an urgent situation.

[0565] "Emotional state" refers to an individual's psychological or emotional state at a particular point in time.

[0566] "Personalized responses" refer to responses that are customized based on the characteristics and circumstances of individual users.

[0567] This invention relates to a multifunctional system for supporting the lives of the elderly, specifically, a system that combines voice recognition, natural language processing, health data analysis, smart device integration, and an emotion engine to provide a variety of support functions.

[0568] 1. System Overview

[0569] The system consists of the following major components:

[0570] Voice input and text conversion methods

[0571] Text data analysis and response generation tools

[0572] Health data collection and analysis tools

[0573] In-home sensor monitor

[0574] Emotion Engine

[0575] Emergency notification means

[0576] 2. Program and Data Processing

[0577] The program of this system is built on Python and uses the following libraries and services.

[0578] Speech Recognition: Use the speech_recognition library to convert voice input into text data.

[0579] Speech synthesis: Uses the pyttsx3 library to convert text data into speech.

[0580] Emotion Recognition: Use your own emotion recognition model (e.g., a BERT-based model).

[0581] Emergency Calling: Call emergency contacts using APIs like Twilio.

[0582] Home sensor monitor: Collects data from various sensors installed in the home (e.g., door opening / closing sensors, motion detection sensors) and detects abnormalities.

[0583] 3. System processing overview

[0584] Speech Recognition and Emotion Analysis

[0585] The server receives voice input from the user and converts it to text using the speech_recognition library. The converted text is then analyzed through an emotion engine to evaluate the user's emotional state. Based on this evaluation, an appropriate response is generated.

[0586] Health data collection and analysis

[0587] A smart device (e.g., a smartwatch) worn by a user collects health data such as heart rate, body temperature, and blood pressure and sends it to a server. The server analyzes the received data and evaluates the user's health status. An emotion engine is also added to this evaluation, and health advice is generated taking the user's emotional state into account.

[0588] Detecting and reporting anomalies in the home

[0589] It monitors data from sensors installed in the home (e.g., door open / close sensors, motion detection sensors), and if an abnormality is detected, it notifies the user by voice. If the abnormality is serious, it automatically makes an emergency call.

[0590] emergency call

[0591] When a user utters keywords such as "help" or "emergency," the system determines this to be an emergency and notifies registered emergency contacts. The notification also includes location information, enabling a quick and accurate response.

[0592] 4. Description of use cases

[0593] Specific examples

[0594] When a user speaks to a smart device, asking, "What's the weather like today?", the server converts the speech into text data and analyzes it. The emotion engine evaluates the user's emotional state (e.g., feeling a little down), and generates a response such as, "It's sunny today. It might be nice to take a walk to change your mood." The generated response is synthesized into voice and notified to the user.

[0595] Furthermore, if the sensor that detects the user's movements detects an abnormality, the server will immediately notify them by voice, saying, "There is an abnormality. Please check." If the situation is serious, an emergency contact will be notified, allowing for a prompt response.

[0596] Prompt Sentence Examples

[0597] Generate a response to the question, "How's the weather today?" Consider an appropriate response if the user is elderly and indicates they are feeling a bit down.

[0598] This allows the system to provide personalized support appropriate to the elderly's emotional state, allowing them to live their daily lives with peace of mind.

[0599] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0600] Step 1:

[0601] The user speaks to the smart device, for example, "What's the weather like today?" This voice input is sent to the device via the microphone.

[0602] Input: User's voice

[0603] Output: Audio data

[0604] Step 2:

[0605] The device's microphone collects voice data and uses the speech_recognition library to convert this voice data into text data, which becomes "What's the weather like today?"

[0606] Input: Audio data

[0607] Output: Text data

[0608] Step 3:

[0609] The device then sends the converted text data to a server via the Internet.

[0610] Input: Text data

[0611] Output: Text data sent to the server

[0612] Step 4:

[0613] The server analyzes the received text data, using natural language processing (NLP) techniques to understand the user's intent and recognizes the user's emotional state through an emotion engine. For example, the server can determine from the user's tone of voice that they are "a little depressed."

[0614] Input: Text data

[0615] Output: Analysis results of text data and user's emotional state

[0616] Step 5:

[0617] The server generates an appropriate response based on the information it has obtained. It retrieves weather information from the weather API and generates a response that adds words of encouragement to a user who is feeling down, based on the results of the emotion engine. For example: "It's sunny today. Maybe it would be good to take a walk to change your mood."

[0618] Input: Text data analysis results, weather information, emotional state

[0619] Output: Response text data

[0620] Step 6:

[0621] The server then sends the generated response text data to the terminal. This communication also takes place over the Internet.

[0622] Input: Response text data

[0623] Output: Response text data sent from the server to the terminal

[0624] Step 7:

[0625] The device converts the received response text data into speech using a speech synthesis engine (pyttsx3 library). For example, it may say, "It's a sunny day today. It might be nice to take a walk to change your mood."

[0626] Input: Response text data

[0627] Output: Response audio data

[0628] Step 8:

[0629] Sensors in the home (door sensors, motion detection sensors, etc.) periodically send data to a server, which monitors for abnormalities and reports them via audio if any are detected.

[0630] Input: Sensor data

[0631] Output: Notification of whether anomalies exist

[0632] Step 9:

[0633] In an emergency, if the user utters "help" or "emergency," the device recognizes this and the server quickly notifies emergency contacts. The notification is made using Twilio's API, and location information is also included.

[0634] Input: Emergency voice command

[0635] Output: Emergency call notification

[0636] Step 10:

[0637] As an emotional care feature, a personalized response is generated according to the user's emotional state, and in cooperation with the emotion engine, advice is given to help the user relax if they are feeling stressed.

[0638] Input: Emotional state data

[0639] Output: Personalized response text and voice data

[0640] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0641] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0642] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0643] [Second embodiment]

[0644] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0645] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0646] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0647] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0648] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0649] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0650] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0651] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0652] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0653] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0654] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0655] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0656] The present invention provides a system for providing a conversation partner for elderly people, preventing dementia, managing health, and supporting daily life. Specific embodiments for implementing the present invention will be described below.

[0657] This system includes a terminal for inputting voice from the elderly user, a means for converting voice into text data, a server for analyzing the text data and generating an appropriate response, a means for converting the generated response back into voice and notifying the user, a means for collecting and analyzing the user's health data and providing appropriate health advice, and a means for managing and notifying the elderly user's daily schedule.

[0658] Conversation Partners / Dementia Prevention

[0659] 1. Voice input

[0660] User: Talks to a smart speaker at home, for example, asking, "What's the weather like today?"

[0661] Terminal: Receives the user's voice and processes it as voice data.

[0662] 2. Converting voice data to text

[0663] Terminal: Converts received voice data into text data using voice recognition technology.

[0664] Terminal: Sends the converted text data to the server.

[0665] 3. Conversation Analysis and Response Generation

[0666] Server: Analyzes the received text data using natural language processing techniques and generates an appropriate response. In this example, it queries a weather API to obtain weather information.

[0667] Server: Generate a response such as "It's a sunny day today. Why don't you go for a walk?"

[0668] 4. Audio Output

[0669] Server: Sends the generated response to the terminal as text data.

[0670] Terminal: Text data is converted into speech using speech synthesis technology and notified to the user.

[0671] health care

[0672] 1. Health Data Collection

[0673] User: Weighs himself / herself and measures his / her blood pressure, and records the data on a smart device (e.g., a smartwatch or a scale).

[0674] Terminal: Obtain health data (e.g., weight "68kg") from a smart device.

[0675] 2. Data submission and analysis

[0676] Terminal: Sends acquired health data to the server.

[0677] Server: Receives health data and compares and analyzes it with past data.

[0678] 3. Health advice generation

[0679] Server: Based on the analysis results, the server generates specific health advice for the user. For example, it generates advice such as, "We recommend that you exercise for about 30 minutes today."

[0680] Server: Sends the generated advice to the terminal as text data.

[0681] 4. Notification

[0682] Terminal: Converts text data into speech and notifies the user.

[0683] Daily life support

[0684] 1. Receiving voice instructions

[0685] User: Speaks to a smart speaker and says, "Tell me what's on my schedule for today."

[0686] Terminal: Receives the user's voice and converts the voice data into text.

[0687] 2. Obtaining schedule information

[0688] Terminal: Sends the converted text data to the server.

[0689] Server: Queries the user's schedule database and retrieves the schedule for the day.

[0690] 3. Response Generation

[0691] Server: Based on the obtained schedule information, it generates a response such as, "I have a hospital appointment at 10:00. I plan to go shopping at the supermarket at 3:00 p.m."

[0692] Server: Sends the generated response to the terminal as text data.

[0693] 4. Notification

[0694] Terminal: Converts text data into speech and notifies the user.

[0695] As described above, this system aims to provide the elderly with daily support from various angles and improve their quality of life. By combining technologies such as voice recognition, natural language processing, health data analysis, and smart device integration, this invention realizes a support system that is highly convenient for the elderly.

[0696] The processing flow will be explained below.

[0697] Conversation Partners / Dementia Prevention

[0698] Voice input and response generation

[0699] Step 1:

[0700] User: Asks their smart speaker at home, "What's the weather like today?"

[0701] Step 2:

[0702] Terminal: Receives the user's voice and processes it as voice data.

[0703] Step 3:

[0704] Terminal: Converts received voice data into text data using voice recognition technology.

[0705] Step 4:

[0706] Terminal: Sends the converted text data to the server.

[0707] Step 5:

[0708] Server: Analyzes the received text data using natural language processing technology. Based on the analysis results, queries the weather API to obtain weather information.

[0709] Step 6:

[0710] Server: Receives the response from the weather API and generates a response (e.g., "It's a sunny day today. Why don't you go for a walk?").

[0711] Step 7:

[0712] Server: Sends the generated response to the terminal as text data.

[0713] Step 8:

[0714] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[0715] health care

[0716] Step 1:

[0717] User: Weighs themselves and measures their blood pressure, and records the data on a smart device (e.g., a smartwatch or scale).

[0718] Step 2:

[0719] Terminal: Health data (e.g., "Weight: 68 kg") is acquired from the smart device via Bluetooth communication, etc.

[0720] Step 3:

[0721] Terminal: Sends acquired health data to the server.

[0722] Step 4:

[0723] Server: Analyzes the received health data, compares it with past data, and evaluates the user's health condition.

[0724] Step 5:

[0725] Server: Based on the evaluation results, generate specific health advice for the user (e.g., "We recommend that you exercise for about 30 minutes today").

[0726] Step 6:

[0727] Server: The generated health advice is sent to the terminal as text data.

[0728] Step 7:

[0729] Terminal: Converts received text data into speech and notifies the user.

[0730] Daily life support

[0731] Step 1:

[0732] User: Speaks to a smart speaker and says, "Tell me what's on my schedule for today."

[0733] Step 2:

[0734] Terminal: Receives the user's voice and processes it as voice data.

[0735] Step 3:

[0736] Terminal: Converts received voice data into text data using voice recognition technology.

[0737] Step 4:

[0738] Terminal: Sends the converted text data to the server.

[0739] Step 5:

[0740] Server: Analyzes the received text data and queries the user's schedule database to obtain the schedule for the day.

[0741] Step 6:

[0742] Server: Generates a response based on the obtained schedule information (e.g., "I have a doctor's appointment at 10:00. I plan to go grocery shopping at 3:00 PM.").

[0743] Step 7:

[0744] Server: Sends the generated response to the terminal as text data.

[0745] Step 8:

[0746] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[0747] Through the above processing steps, the present invention provides support to help elderly people live healthy and fulfilling lives without feeling lonely.

[0748] Example 1

[0749] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0750] Currently, support systems for elderly people to smoothly lead their daily lives are inadequate. There is a particular need for comprehensive support that provides a wide range of support, including simple voice-based operation, health management, and schedule management. Furthermore, there is a lack of systems that allow elderly people to understand their own health status and receive appropriate advice. To address these issues, the present invention aims to provide a comprehensive support system based on voice input by elderly people.

[0751] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0752] In this invention, the server includes means for receiving a user's voice input, means for converting the received voice into text data, means for transmitting the converted text data to the processing device, means for analyzing the text data in the processing device and generating an appropriate response, means for transmitting the generated response in text format from the processing device to the display device, means for converting the received response into voice and notifying the user, means for collecting the user's health data, means for transmitting the collected health data to the processing device, means for analyzing the health data in the processing device and evaluating the user's health condition, means for generating health advice based on the evaluation results, means for transmitting the generated advice from the processing device to the display device, means for converting the received advice into voice and notifying the user, means for managing the user's daily schedule, means for receiving the user's voice instructions, means for transmitting the voice instructions to the processing device and acquiring schedule information, means for analyzing the acquired schedule information and generating a response to the user, means for transmitting the generated response from the processing device to the display device, and means for converting the received response into voice and notifying the user. This enables comprehensive support for the daily lives of elderly people in various aspects.

[0753] A "means for receiving voice input" is a device that recognizes the voice uttered by the user and captures it as digital data.

[0754] "Means for converting voice into text data" refers to an algorithm or system that uses voice recognition technology to convert received voice data into character string data.

[0755] The "means for transmitting text data to a processing device" is a function for transferring the converted text data to a device for performing other processing using a communication means such as a network.

[0756] "Means for analyzing text data using a processing device and generating an appropriate response" refers to a system that uses natural language processing technology to analyze the meaning and intent of received text data and generate an appropriate response to the user.

[0757] The "means for transmitting the generated response in text format from the processing device to the display device" is a function for transmitting the generated response in text data to another device for display.

[0758] The "means for converting the received response into voice and notifying the user" is a device that converts the response received as text data into voice using voice synthesis technology and transmits it to the user.

[0759] A "means for collecting health data" is a device that has the function of measuring and recording numerical health data such as a user's weight, blood pressure, and heart rate.

[0760] The "means for transmitting collected health data to a processing device" is a system that transmits collected health data to another processing device via a network.

[0761] "Means for analyzing health data using a processing device and assessing the user's health condition" refers to an algorithm or system that analyzes collected health data and understands and assesses the user's health condition.

[0762] The "means for generating health advice based on the evaluation results" is a system that generates specific advice for maintaining and improving the user's health based on the analysis results.

[0763] The "means for transmitting the generated advice from the processing device to the display device" is a function for transmitting the advice to the display device in order to notify the user of the generated health advice.

[0764] "Means for managing daily schedules" refers to a system that records and manages a user's daily plans and schedules.

[0765] The "means for receiving voice instructions" is a device that recognizes voice instructions from the user and captures them as digital data.

[0766] The "means for transmitting voice instructions to the processing device and acquiring schedule information" is a system for transmitting voice instructions to the processing device and acquiring the user's schedule information.

[0767] The "means for analyzing the acquired schedule information and generating a response to the user" refers to an algorithm or system that generates an appropriate response based on the schedule information.

[0768] The "means for transmitting the generated response from the processing device to the display device" is a function for transmitting the generated response to the display device and displaying it to the user.

[0769] The present invention provides a system for providing a conversation partner for elderly people, preventing dementia, managing health, and supporting daily life. Specific embodiments for implementing the present invention will be described below.

[0770] This system includes a terminal for inputting voice, a means for converting voice into text data, a server for analyzing the text data and generating an appropriate response, a means for converting the generated response into voice and notifying the user, a means for analyzing health data and providing appropriate health advice, and a means for managing and notifying the user of daily schedules.

[0771] The voice input in this system uses a smart speaker, such as Amazon Echo or Google Home, to capture the user's voice. The captured voice is then converted into text data using the Google Cloud Speech-to-Text API.

[0772] The text data is sent to a server, which then analyzes it using natural language processing technology, specifically a generative AI model such as GPT-4, to generate an appropriate response.

[0773] The generated response is then sent back to the device (such as a smart speaker or smartphone) and converted into audio using the Google Cloud Text-to-Speech API, which is then notified to the user.

[0774] Specific examples

[0775] Conversation Partners / Dementia Prevention

[0776] Prompt: "What's the weather like today?"

[0777] Response: "It's a beautiful sunny day today. Why don't you go for a walk?"

[0778] health care

[0779] Users' health data is collected using devices such as smartwatches and digital scales. The collected data is sent to a server via the device and analyzed there. Based on the analysis results, health advice for the user is generated, sent back to the device, and notified via voice.

[0780] Prompt: "I've weighed myself. What should I do next?"

[0781] Response: "According to our records, you weigh 140 lbs. I suggest you exercise for about 30 minutes today."

[0782] Daily life support

[0783] A smart speaker is used to manage daily schedules. When a user says, "Tell me what's on my schedule for today," the voice is converted into text data and sent to a server. The server retrieves the schedule using the Google Calendar API or similar, generates a response, and notifies the user by voice.

[0784] Prompt: "What's on your agenda for today?"

[0785] Response: "I have a doctor's appointment at 10:00. I'll be grocery shopping at 3:00."

[0786] As described above, this system aims to provide the elderly with daily support from various angles and improve their quality of life. By combining technologies such as voice recognition, natural language processing, health data analysis, and smart device integration, this invention realizes a support system that is highly convenient for the elderly.

[0787] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0788] Conversation Partners / Dementia Prevention

[0789] Step 1:

[0790] A user speaks to a smart speaker in their home, asking, for example, "What's the weather like today?"

[0791] Input: User's voice

[0792] Output: Audio data captured by the smart speaker

[0793] Step 2:

[0794] The device (smart speaker) converts the received audio into text data in real time using the Google Cloud Speech-to-Text API.

[0795] Input: Audio data

[0796] Output: Text data

[0797] Step 3:

[0798] The terminal transmits the converted text data to the server.

[0799] Input: Text data

[0800] Output: Text data sent to the server

[0801] Step 4:

[0802] The server analyzes the received text data using a natural language processing model (e.g., GPT-4). This analysis allows it to understand the user's intent. Next, the server queries an external weather API (e.g., OpenWeatherMap API) to obtain weather information.

[0803] Input: Text data

[0804] Output: Response text generated based on weather information

[0805] Step 5:

[0806] The server transmits the generated response to the terminal as text data.

[0807] Input: Response text

[0808] Output: Text data sent to the terminal

[0809] Step 6:

[0810] The device converts the received text data into audio using the Google Cloud Text-to-Speech API and notifies the user.

[0811] Input: Text data

[0812] Output: Audio data (notification audio)

[0813] health care

[0814] Step 1:

[0815] The user measures their weight and blood pressure and records the data on a smart device (e.g., a smart watch).

[0816] Input: Health data (weight, blood pressure, etc.)

[0817] Output: Data recorded on a smart device

[0818] Step 2:

[0819] The terminal (smartphone) acquires health data from the smart device via BLE (Bluetooth Low Energy).

[0820] Input: Data communication from smart devices

[0821] Output: Health data collected on the device

[0822] Step 3:

[0823] The terminal transmits the acquired health data to the server.

[0824] Input: Health data

[0825] Output: Health data sent to the server

[0826] Step 4:

[0827] The server compares and analyzes the received health data with past data to evaluate the user's health condition.

[0828] Input: Health data

[0829] Output: Analysis results (health assessment data)

[0830] Step 5:

[0831] Based on the analysis results, the server generates specific health advice for the user, such as "We recommend you exercise for about 30 minutes today."

[0832] Input: Health assessment data

[0833] Output: Generated health advice

[0834] Step 6:

[0835] The server transmits the generated advice to the terminal as text data.

[0836] Enter: Health Advice

[0837] Output: Text data sent to the terminal

[0838] Step 7:

[0839] The terminal converts the text data into voice and notifies the user.

[0840] Input: Text data

[0841] Output: Audio data (notification audio)

[0842] Daily life support

[0843] Step 1:

[0844] The user speaks to the smart speaker and says, "Tell me what's on my schedule for today."

[0845] Input: User's voice

[0846] Output: Audio data captured by the smart speaker

[0847] Step 2:

[0848] The device (smart speaker) converts the received voice data into text data using the Google Cloud Speech-to-Text API.

[0849] Input: Audio data

[0850] Output: Text data

[0851] Step 3:

[0852] The terminal transmits the text data to the server.

[0853] Input: Text data

[0854] Output: Text data sent to the server

[0855] Step 4:

[0856] The server analyzes the received text data and queries the user's schedule database (e.g., Google Calendar) to obtain the schedule for that day.

[0857] Input: Text data

[0858] Output: Schedule information

[0859] Step 5:

[0860] Based on the schedule information it has obtained, the server generates a response such as, "I have a hospital appointment at 10:00. I plan to go shopping at the supermarket at 3:00 p.m."

[0861] Input: Schedule information

[0862] Output: Generated response text data

[0863] Step 6:

[0864] The server transmits the generated response to the terminal as text data.

[0865] Input: Response text data

[0866] Output: Text data sent to the terminal

[0867] Step 7:

[0868] The terminal converts the text data into voice and notifies the user.

[0869] Input: Text data

[0870] Output: Audio data (notification audio)

[0871] (Application example 1)

[0872] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0873] Today's elderly people need support in many aspects of daily life. They may have difficulty managing their health and daily tasks, as well as having difficulty shopping and socializing. Shopping in brick-and-mortar stores can be particularly challenging, as they struggle to find the products they need and have difficulty accessing sales information. This often leads to a decline in the quality of life for elderly people and a sense of isolation. The present invention aims to address these challenges and improve the quality of life for elderly people.

[0874] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0875] In this invention, the server includes means for receiving voice input from the elderly person, means for converting the received voice into text data, means for transmitting the converted text data to the server, means for analyzing the text data in the server and generating an appropriate response, means for transmitting the generated response in text format from the server to the terminal, means for converting the received response into voice and notifying the user, and means for acquiring and providing sale information using a generative AI model, thereby enabling elderly people to shop more smoothly in physical stores.

[0876] definition statement

[0877] "Elderly people" generally refer to people aged 65 and over who often require special assistance and support in their daily lives.

[0878] A "means for receiving voice input" is a device that allows a user to issue voice commands and has the ability to record and process the voice as digital data.

[0879] The "means for converting voice into text data" is a system that uses voice recognition technology to convert input voice into text data in real time.

[0880] "Means for transmitting text data to a server" refers to a function for transmitting converted text data to a server located at a remote location via a network.

[0881] "Means for analyzing text data and generating appropriate responses" refers to a function that uses natural language processing technology to understand the input text and generate the information or response the user is looking for.

[0882] "Means for transmitting the generated response in text format from the server to the terminal" refers to the function of transmitting the analysis results from the server to the user's terminal as text data.

[0883] The "means for converting the received response into voice and notifying the user" is a system that has the function of converting text data back into voice and conveying that voice to the user.

[0884] A "generative AI model" is an artificial intelligence model that has been trained in advance using large amounts of data and is capable of automatically generating a variety of responses and suggestions.

[0885] The "means for acquiring and providing sale information" refers to a system that utilizes inventory management systems and marketing data from physical stores to acquire the latest sale information and provide it to users.

[0886] MODE FOR CARRYING OUT THE INVENTION

[0887] The present invention is a system that provides sales information based on voice input and supports shopping in brick-and-mortar stores. This system uses speech recognition, text analysis, natural language processing, and generative AI models to generate specific responses and provide them to users, thereby supporting shopping for elderly people and others. Specific embodiments of the present invention are described below.

[0888] System configuration

[0889] Hardware:

[0890] Smartphone: Receives voice input, sends the speech recognition results to the server, and converts the text response from the server into speech to notify the user.

[0891] Server: Converts voice data into text data, analyzes it, and generates a response.

[0892] software:

[0893] Speech recognition library (e.g., speech_recognition library): Converts voice input into text data.

[0894] Natural language processing model: Analyzes input text data and generates appropriate responses.

[0895] Generative AI model: Used to determine special offers.

[0896] Text-to-speech libraries (e.g. gTTS library): Convert text responses into speech.

[0897] Data processing and calculation

[0898] Receiving and converting voice input:

[0899] The user speaks into the smartphone, and a speech recognition library (e.g., speech_recognition library) converts this speech into text data and sends it to the server.

[0900] Analyzing text data:

[0901] The server then analyzes the converted text data using natural language processing technology, determining the user's intent from the analysis results and identifying requests such as sales information.

[0902] Get sale information and generate a response:

[0903] The server uses the generative AI model to retrieve sales information from the physical store's inventory management system and marketing database, and then generates a response containing the information and suggestions the user is looking for.

[0904] Response transcription and announcement:

[0905] The generated response is sent to the smartphone as text data and converted into speech using a speech synthesis library (e.g., gTTS library). The converted speech is then notified to the user through the smartphone's speaker.

[0906] Examples and prompts

[0907] Examples:

[0908] A user speaks to their smartphone, "What are the specials this week?" The smartphone converts this speech into text data and sends it to a server. The server analyzes the text data and uses a generative AI model to obtain the special sale information. Based on the obtained special sale information, it generates a response such as, "This week's special is apples, 100 yen. They're in the vegetable section." The smartphone converts this response into speech and notifies the user.

[0909] Example prompt sentence:

[0910] "What are the specials this week?"

[0911] "Which department should I go to?"

[0912] This system will enable seniors to easily obtain shopping information at brick-and-mortar stores, ensuring they don't miss out on special offers. It is also expected that the natural voice interaction will make shopping smoother and more enjoyable.

[0913] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0914] Program processing steps

[0915] Step 1:

[0916] User speaks:

[0917] The user speaks a question or request into the smartphone, for example, "What are the specials this week?" This is the initial input to the system.

[0918] Step 2:

[0919] Audio to text conversion:

[0920] The user's voice is received by the smartphone's microphone and converted into text data using the speech_recognition library. At this stage, the input voice is converted into string data.

[0921] Step 3:

[0922] Sending text data to the server:

[0923] The text data obtained by speech recognition is sent from the smartphone to the server, which then forwards the request to the server via the network.

[0924] Step 4:

[0925] Text data analysis and response generation:

[0926] The server analyzes the received text data using natural language processing technology. Based on the analyzed text data, it understands the user's request and uses a generative AI model to obtain special sale information. It then generates a response based on the special sale information and prepares it in text data format.

[0927] Step 5:

[0928] Sending the generated response from the server to the device:

[0929] The generated text response is then sent from the server to the smartphone via the network, and the analysis results are returned to the user's device.

[0930] Step 6:

[0931] Text response transcription:

[0932] The smartphone converts the received text data into speech using the gTTS library, and outputs synthesized speech data based on the input text.

[0933] Step 7:

[0934] Notify user of voice response:

[0935] The converted response is then broadcast to the user through the smartphone's speaker, allowing the user to hear the response to their question.

[0936] Specific processing operations

[0937] Step 1:

[0938] The user speaks to the smartphone, "What are the specials this week?" The voice is received by the smartphone.

[0939] Step 2:

[0940] The voice data received by the smartphone's microphone is converted into a string using the speech_recognition library, and the text data "What are the specials this week?" is generated.

[0941] Step 3:

[0942] The text data generated by the speech recognition is sent from the smartphone to the server, where data communication takes place via the network.

[0943] Step 4:

[0944] The server uses natural language processing technology to analyze the received text data and understand the user's request. It then uses a generative AI model to retrieve sales information from the physical store's inventory management system and generates a response such as, "This week's sales item is apples, priced at 100 yen. They're in the vegetable section."

[0945] Step 5:

[0946] The text response generated by the server is sent over the network to the smartphone, which receives the text response.

[0947] Step 6:

[0948] The smartphone converts the received text data into speech using the gTTS library, generating speech data such as, "This week's special offer is apples, priced at 100 yen. They're in the vegetable section."

[0949] Step 7:

[0950] The user is notified of the voice-converted response through the smartphone's speaker, and can listen to the voice response.

[0951] Through the above process, shopping support for the elderly is realized.

[0952] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0953] This invention is a system that supports the lives of the elderly, combining voice recognition, natural language processing, health data analysis, smart device integration, and an emotion engine. This system provides various functions to help the elderly relieve loneliness, prevent dementia, and support health management.

[0954] 1. Conversation partner / dementia prevention

[0955] Voice input and response generation

[0956] When a user asks a question or speaks to a smart speaker, the voice input is received by the device. The device converts the voice data into text data and sends it to the server. The server analyzes the text data and generates an appropriate response. This process also incorporates an emotion engine that recognizes emotions from the user's voice. The tone and content of the response are adjusted based on the recognized emotion. The generated response is sent from the server to the device in text format, where it is converted into voice and notified to the user.

[0957] Examples:

[0958] User: "What's the weather like today?"

[0959] Terminal: Receives audio, converts it into text, and sends it to the server.

[0960] Server: Analyzes text data and obtains weather information. The emotion engine recognizes the user's emotion (e.g., feeling down), and generates a response based on that, such as "It's sunny today. Maybe you should take a walk to change your mood."

[0961] Terminal: Converts the response from the server into voice and notifies the user.

[0962] 2. Health management

[0963] Health data collection and analysis

[0964] Users use smart devices to collect health data. For example, they measure their body temperature and blood pressure with a smartwatch. The collected health data is sent to a server via the device. The server analyzes the received data and evaluates the user's health status. An emotion engine is also incorporated into this evaluation, and health advice is generated taking the user's emotional state into account.

[0965] Examples:

[0966] User: Weighs himself and records the data on a smart scale.

[0967] Terminal: Receives data from the scale and sends it to the server.

[0968] Server: Analyzes the data and generates health advice such as, "You seem a little low today. I recommend some light exercise."

[0969] Terminal: Converts the advice into voice and notifies the user.

[0970] 3. Daily life support

[0971] Schedule management and coordination

[0972] When a user asks a smart speaker about their daily schedule, the device receives the voice, converts it into text, and sends it to a server. The server then checks the user's schedule and uses an emotion engine to generate a response that takes into account the user's current emotional state.

[0973] Examples:

[0974] User: "What's on my schedule for today?"

[0975] Terminal: Receives audio, converts it into text, and sends it to the server.

[0976] Server: Checks the schedule. Generates a response saying, "I have a doctor's appointment at 10:00. I'm going to the supermarket at 3:00 PM." The emotion engine determines that the user is feeling stressed, and gives additional advice, saying, "Try not to push yourself too hard, and take things slowly."

[0977] Terminal: Converts responses and advice into voice and notifies the user.

[0978] As described above, the present invention provides a multi-functional support system that helps elderly people live healthy and fulfilling lives without feeling lonely. By adding emotion recognition functionality, the system achieves more advanced and personalized support.

[0979] The processing flow will be explained below.

[0980] Conversation Partners / Dementia Prevention

[0981] Voice input and response generation

[0982] Step 1:

[0983] User: Asks their smart speaker at home, "What's the weather like today?"

[0984] Step 2:

[0985] Terminal: Receives the user's voice and processes it as voice data.

[0986] Step 3:

[0987] Terminal: Converts received voice data into text data using voice recognition technology.

[0988] Step 4:

[0989] Terminal: Sends the converted text data to the server.

[0990] Step 5:

[0991] Server: Analyzes the received text data using natural language processing technology. Based on the analysis results, queries the weather API to obtain weather information.

[0992] Step 6:

[0993] Server: Receives the response from the weather API and generates text data.

[0994] Step 7:

[0995] Server: Recognizes emotions from the user's voice using the emotion engine.

[0996] Step 8:

[0997] Server: Adjusts the content and tone of the response based on the recognized emotion. For example, if a user is feeling down, the server generates a response like, "It's a sunny day today. Why don't you take a walk to cheer yourself up?"

[0998] Step 9:

[0999] Server: Sends the generated response to the terminal as text data.

[1000] Step 10:

[1001] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[1002] health care

[1003] Step 1:

[1004] User: Records health data such as weight and blood pressure on a smart device (e.g., smartwatch or scale).

[1005] Step 2:

[1006] Terminal: Health data is acquired from smart devices using communication methods such as Bluetooth.

[1007] Step 3:

[1008] Terminal: Sends acquired health data to the server.

[1009] Step 4:

[1010] Server: Analyzes the received data and evaluates the user's health condition.

[1011] Step 5:

[1012] Server: Analyzes the user's emotional data simultaneously using an emotion engine.

[1013] Step 6:

[1014] Server: Evaluates the overall physical and mental health status and generates appropriate health advice. For example, it generates advice such as, "You're feeling well today, but you seem a little tired. I recommend some light exercise."

[1015] Step 7:

[1016] Server: The generated health advice is sent to the terminal as text data.

[1017] Step 8:

[1018] Terminal: Converts received text data into speech and notifies the user.

[1019] Daily life support

[1020] Step 1:

[1021] User: Speaks to a smart speaker and says, "Tell me what's on my schedule for today."

[1022] Step 2:

[1023] Terminal: Receives the user's voice and processes it as voice data.

[1024] Step 3:

[1025] Terminal: Converts received voice data into text data using voice recognition technology.

[1026] Step 4:

[1027] Terminal: Sends the converted text data to the server.

[1028] Step 5:

[1029] Server: Analyzes the received text data and queries the user's schedule database to obtain the schedule for the day.

[1030] Step 6:

[1031] Server: Analyzes the acquired schedule information and generates text data.

[1032] Step 7:

[1033] Server: Uses an emotion engine to recognize the user's current emotional state.

[1034] Step 8:

[1035] Server: Adjusts the tone and content of schedule notifications based on the recognized emotion. For example, if it determines that the user is feeling stressed, it generates a response such as, "You have a doctor's appointment at 10 o'clock today. Take it easy and take it easy."

[1036] Step 9:

[1037] Server: Sends the generated response to the terminal as text data.

[1038] Step 10:

[1039] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[1040] Through the above processing steps, the present invention provides a multi-functional support system that helps elderly people avoid loneliness and live healthy and fulfilling lives. The addition of emotion recognition functionality provides more advanced and personalized support.

[1041] Example 2

[1042] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1043] Elderly people often feel lonely in their daily lives and find health care a burden. Furthermore, there is a lack of support for dementia prevention and daily schedule management. These situations can reduce the quality of life for elderly people and lead to further health risks. Therefore, there is a need for the development of multifunctional and personalized systems to support the daily lives of elderly people.

[1044] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1045] In this invention, the server includes means for receiving voice input from the elderly person, means for converting the received voice into character data, means for transmitting the converted character data to a central control unit, means for analyzing the character data in the central control unit and generating an appropriate response (including an emotion engine), means for transmitting the generated response in character form from the central control unit to a terminal, and means for converting the received response into voice and notifying the elderly person. This reduces the elderly person's sense of loneliness, supports dementia prevention, and reduces the burden of health management.

[1046] "Means for receiving voice input" refers to devices or technology that detects voices made by the senior and processes the voice signals.

[1047] "Means for converting into character data" refers to technology that analyzes received voice signals, recognizes phonemes and words, and converts them into character strings.

[1048] "Means for transmitting to a server" refers to a device or technology that transmits the converted character data to a central control device (server) via a communications network.

[1049] "Central Controller" means a central control unit or system that analyzes received text data and generates an appropriate response.

[1050] "Means for generating an appropriate response" refers to technology that uses specific algorithms or emotion engines based on received data to generate an appropriate response to the user.

[1051] An "emotion engine" refers to an algorithm or system that analyzes emotions from received voice or data and generates a response based on those emotions.

[1052] "Means for sending to the terminal" refers to a device or technique for sending a response in the form of character data generated by the central control unit to the user's terminal.

[1053] "Means for converting the received response into voice" refers to a device or technology that synthesizes text data sent to the terminal into voice and notifies the user.

[1054] "Means for collecting health data" refers to devices and technologies for measuring health information such as a user's weight, body temperature, and blood pressure.

[1055] "Means for analyzing and evaluating health data" refers to algorithms or systems for analyzing collected health data and evaluating the user's health status.

[1056] The term "means for generating health advice" refers to a technology for generating health advice for a user based on the evaluation results of the health data.

[1057] "Means for managing daily schedules" refers to devices and technologies for managing a user's daily plans and tasks.

[1058] The term "means for receiving voice instructions" refers to a device or technology that detects voice instructions regarding the schedule given by the user and processes the voice signals.

[1059] "Means for obtaining schedule information" refers to devices and techniques for querying and obtaining the user's daily schedule information from the central control device.

[1060] The term "means for analyzing schedule information and generating a response" refers to a technique for analyzing acquired schedule information and generating an appropriate schedule-related response to the user.

[1061] This invention is a system for supporting the lives of the elderly, combining voice recognition, natural language processing, health data analysis, smart device integration, and an emotion engine. The system's main functions include serving as a conversation partner, health management, and daily schedule management for everyday use by the elderly.

[1062] The components of this system are:

[1063] 1. Conversation partner / dementia prevention

[1064] When a user asks a question or speaks to a smart speaker, the voice input is received by the device. The device converts the voice data into text data (using, for example, the Google Speech-to-Text API) and sends the converted data to a server. The server analyzes the text data using a natural language processing engine (for example, GPT-3) and generates an appropriate response. It also incorporates an emotion engine (for example, IBM Watson Tone Analyzer) that recognizes the user's emotions and adjusts the tone and content of the response. The generated response is sent from the server to the device in text format, where it is converted into speech and notified to the user (for example, using Amazon Polly).

[1065] Examples:

[1066] User: "What's the weather like today?"

[1067] Device: Receives audio, converts it to text using the Google Speech-to-Text API, and sends it to the server.

[1068] Server: Analyzes text data using GPT-3 and obtains weather information. The emotion engine recognizes the user's emotions and generates a response such as, "It's sunny today. It might be nice to take a walk to change your mood."

[1069] Terminal: The response from the server is converted into speech using Amazon Polly and notified to the user.

[1070] 2. Health management

[1071] Users collect their health data using a smart device. For example, they measure their body temperature and blood pressure with a smartwatch and send the data to the device. The device then sends the received data to a server, which analyzes it using a health data analysis algorithm (e.g., Google Fit API). An emotion engine is also incorporated into this evaluation, which generates health advice taking the user's emotional state into account. The advice is then sent from the server to the device, which converts it into audio and notifies the user.

[1072] Examples:

[1073] User: Weighs himself and records the data on a smart scale.

[1074] Terminal: Receives data from the scale and sends it to the server.

[1075] Server: Analyzes the data using the Google Fit API and generates health advice such as, "You seem a little low today. I recommend some light exercise."

[1076] Device: The advice is converted into speech using Amazon Polly and notified to the user.

[1077] 3. Daily life support

[1078] When a user speaks to a smart speaker to check their daily schedule, the device receives the voice, converts it into text data, and sends it to the server. The server then checks the user's daily schedule information and uses an emotion engine to generate a response that takes into account the user's current emotional state. The generated response is then sent from the server to the device, which converts it into voice and notifies the user.

[1079] Examples:

[1080] User: "What's on my schedule for today?"

[1081] Device: Receives audio, converts it to text using the Google Speech-to-Text API, and sends it to the server.

[1082] Server: Checks the schedule and generates a response saying, "I have a doctor's appointment at 10:00. I'm going to the supermarket at 3:00 PM." The emotion engine determines that the user is stressed and provides additional advice, saying, "Try not to push yourself too hard, and take things slowly."

[1083] Device: Responses and advice are converted into speech using Amazon Polly and communicated to the user.

[1084] Example prompts for generative AI models

[1085] "Please explain the function of an elderly life support system that uses voice recognition and an emotion engine to respond to user questions. Please also include specific processing procedures and examples when a user asks a question to a smart speaker."

[1086] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1087] 1. Conversation partner / dementia prevention

[1088] Step 1:

[1089] A user speaks to a smart speaker, such as "What's the weather like today?"

[1090] (Input) User's voice data

[1091] (Output) User's voice data

[1092] Step 2:

[1093] The device receives the voice data and converts it into text using voice recognition software (e.g., Google Speech-to-Text API).

[1094] (Input) User's voice data

[1095] (Output) The user's voice is converted into text data.

[1096] Step 3:

[1097] The terminal sends the converted text data to the server using a secure protocol such as HTTPS.

[1098] (Input) Character data

[1099] (Output) Character data sent to the server

[1100] Step 4:

[1101] The server uses a natural language processing engine (e.g., GPT-3) to analyze the text data, and at the same time, an emotion engine (e.g., IBM Watson Tone Analyzer) recognizes the user's emotions, which allows it to generate more appropriate responses.

[1102] (Input) Character data

[1103] (Data processing) Analyze emotions with an emotion engine and analyze data with a natural language processing engine

[1104] (Output) Analysis results and response data based on them

[1105] Step 5:

[1106] The server generates a response and sends it to the terminal in character data format.

[1107] (Input) Generated response data

[1108] (Output) Response data sent to the terminal

[1109] Step 6:

[1110] The response data received by the device is converted into speech using speech synthesis software (e.g., Amazon Polly) and notified to the user.

[1111] (Input) Response data

[1112] (Output) Response data converted to audio

[1113] 2. Health management

[1114] Step 1:

[1115] A user records health data using a health device (e.g., smart watch, smart scale, etc.).

[1116] (Input) User's health data (weight, body temperature, blood pressure, etc.)

[1117] (Output) Health data recorded on the health device

[1118] Step 2:

[1119] The terminal receives data from the health device using communication methods such as Bluetooth and Wi-Fi.

[1120] (Input) Data from health devices

[1121] (Output) Health data received by the device

[1122] Step 3:

[1123] The health data received by the device is sent to the server.

[1124] (Input) Health data

[1125] (Output) Health data sent to the server

[1126] Step 4:

[1127] The server uses a healthcare analysis algorithm (e.g., Google Fit API) to analyze the received health data, while also taking into account the user's emotional state using an emotion engine.

[1128] (Input) Health data

[1129] (Data processing) Health data analysis and emotional state assessment

[1130] (Output) Analysis results and health advice data

[1131] Step 5:

[1132] The server sends the generated health advice to the terminal.

[1133] (Input) Health advice data

[1134] (Output) Health advice data sent to the device

[1135] Step 6:

[1136] The health advice data received by the terminal is converted into voice using voice synthesis software and notified to the user.

[1137] (Input) Health advice data

[1138] (Output) Health advice converted into audio

[1139] 3. Daily life support

[1140] Step 1:

[1141] A user asks a smart speaker about their daily schedule, such as "Tell me what's on my schedule for today."

[1142] (Input) User's voice data

[1143] (Output) User's voice data

[1144] Step 2:

[1145] The device receives the audio and converts it into text data (e.g., Google Speech-to-Text API).

[1146] (Input) User's voice data

[1147] (Output) The user's voice is converted into text data.

[1148] Step 3:

[1149] The terminal sends character data to the server.

[1150] (Input) Character data

[1151] (Output) Character data sent to the server

[1152] Step 4:

[1153] The server analyzes the text data and queries the daily schedule database, while the emotion engine considers the user's current emotional state.

[1154] (Input) Character data, daily schedule database

[1155] (Data processing) Check schedule and evaluate emotional state

[1156] (Output) Schedule confirmation results and appropriate response data

[1157] Step 5:

[1158] The server sends the generated response data to the terminal.

[1159] (Input) Response data

[1160] (Output) Response data sent to the terminal

[1161] Step 6:

[1162] The response data received by the terminal is converted into voice using voice synthesis software and notified to the user.

[1163] (Input) Response data

[1164] (Output) The voiced response

[1165] (Application example 2)

[1166] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1167] In modern society, the safety and health management of elderly people has become an important issue. Elderly people living alone, in particular, need to respond quickly to emergencies and emergencies at home, and also need to alleviate feelings of loneliness and receive emotional care. However, current systems are unable to comprehensively meet these needs. Furthermore, technology for providing personalized support based on emotional state is immature. An advanced, integrated system to solve these issues is needed.

[1168] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice input from the elderly person, means for converting the received voice into text data, means for transmitting the converted text data to the server, means for analyzing the text data in the server and generating an appropriate response, means for transmitting the generated response in text format from the server to the terminal, means for converting the received response into voice and notifying the user, means for detecting abnormalities in the home and reporting if an abnormality is detected, means for making an emergency call based on the elderly person's voice instruction, and means for recognizing the user's emotional state and generating a response accordingly. This ensures the safety of the elderly person at home and enables rapid response in emergencies. Furthermore, personalized responses based on the elderly person's emotional state can alleviate feelings of loneliness and provide emotional care.

[1169] "Elderly" refers to individuals who are aging, especially those in an age group where health care and safety are of paramount importance.

[1170] "Audio input" refers to an audio signal collected using a microphone or other audio collection device.

[1171] "Text data" refers to data in the form of a string of characters generated from voice input or other digital data.

[1172] A "server" refers to a computer system that provides data processing and data storage over a network.

[1173] "Analysis" refers to the process of examining, interpreting, and evaluating data.

[1174] "Response" refers to information generated in response to a user input or request.

[1175] "Transmission" refers to the act of moving data from one point to another.

[1176] "Domestic anomalies" refers to abnormal or unexpected events or changes in the normal home environment.

[1177] "Reporting" refers to the act of communicating information about an anomaly or event to others.

[1178] "Emergency notification" refers to the prompt notification of pre-established contacts or organizations in an urgent situation.

[1179] "Emotional state" refers to an individual's psychological or emotional state at a particular point in time.

[1180] "Personalized responses" refer to responses that are customized based on the characteristics and circumstances of individual users.

[1181] This invention relates to a multifunctional system for supporting the lives of the elderly, specifically, a system that combines voice recognition, natural language processing, health data analysis, smart device integration, and an emotion engine to provide a variety of support functions.

[1182] 1. System Overview

[1183] The system consists of the following major components:

[1184] Voice input and text conversion methods

[1185] Text data analysis and response generation tools

[1186] Health data collection and analysis tools

[1187] In-home sensor monitor

[1188] Emotion Engine

[1189] Emergency notification means

[1190] 2. Program and Data Processing

[1191] The program of this system is built on Python and uses the following libraries and services.

[1192] Speech Recognition: Use the speech_recognition library to convert voice input into text data.

[1193] Speech synthesis: Uses the pyttsx3 library to convert text data into speech.

[1194] Emotion Recognition: Use your own emotion recognition model (e.g., a BERT-based model).

[1195] Emergency Calling: Call emergency contacts using APIs like Twilio.

[1196] Home sensor monitor: Collects data from various sensors installed in the home (e.g., door opening / closing sensors, motion detection sensors) and detects abnormalities.

[1197] 3. System processing overview

[1198] Speech Recognition and Emotion Analysis

[1199] The server receives voice input from the user and converts it to text using the speech_recognition library. The converted text is then analyzed through an emotion engine to evaluate the user's emotional state. Based on this evaluation, an appropriate response is generated.

[1200] Health data collection and analysis

[1201] A smart device (e.g., a smartwatch) worn by a user collects health data such as heart rate, body temperature, and blood pressure and sends it to a server. The server analyzes the received data and evaluates the user's health status. An emotion engine is also added to this evaluation, and health advice is generated taking the user's emotional state into account.

[1202] Detecting and reporting anomalies in the home

[1203] It monitors data from sensors installed in the home (e.g., door open / close sensors, motion detection sensors), and if an abnormality is detected, it notifies the user by voice. If the abnormality is serious, it automatically makes an emergency call.

[1204] emergency call

[1205] When a user utters keywords such as "help" or "emergency," the system determines this to be an emergency and notifies registered emergency contacts. The notification also includes location information, enabling a quick and accurate response.

[1206] 4. Description of use cases

[1207] Specific examples

[1208] When a user speaks to a smart device, asking, "What's the weather like today?", the server converts the speech into text data and analyzes it. The emotion engine evaluates the user's emotional state (e.g., feeling a little down), and generates a response such as, "It's sunny today. It might be nice to take a walk to change your mood." The generated response is synthesized into voice and notified to the user.

[1209] Furthermore, if the sensor that detects the user's movements detects an abnormality, the server will immediately notify them by voice, saying, "There is an abnormality. Please check." If the situation is serious, an emergency contact will be notified, allowing for a prompt response.

[1210] Prompt Sentence Examples

[1211] Generate a response to the question, "How's the weather today?" Consider an appropriate response if the user is elderly and indicates they are feeling a bit down.

[1212] This allows the system to provide personalized support appropriate to the elderly's emotional state, allowing them to live their daily lives with peace of mind.

[1213] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1214] Step 1:

[1215] The user speaks to the smart device, for example, "What's the weather like today?" This voice input is sent to the device via the microphone.

[1216] Input: User's voice

[1217] Output: Audio data

[1218] Step 2:

[1219] The device's microphone collects voice data and uses the speech_recognition library to convert this voice data into text data, which becomes "What's the weather like today?"

[1220] Input: Audio data

[1221] Output: Text data

[1222] Step 3:

[1223] The device then sends the converted text data to a server via the Internet.

[1224] Input: Text data

[1225] Output: Text data sent to the server

[1226] Step 4:

[1227] The server analyzes the received text data, using natural language processing (NLP) techniques to understand the user's intent and recognizes the user's emotional state through an emotion engine. For example, the server can determine from the user's tone of voice that they are "a little depressed."

[1228] Input: Text data

[1229] Output: Analysis results of text data and user's emotional state

[1230] Step 5:

[1231] The server generates an appropriate response based on the information it has obtained. It retrieves weather information from the weather API and generates a response that adds words of encouragement to a user who is feeling down, based on the results of the emotion engine. For example: "It's sunny today. Maybe it would be good to take a walk to change your mood."

[1232] Input: Text data analysis results, weather information, emotional state

[1233] Output: Response text data

[1234] Step 6:

[1235] The server then sends the generated response text data to the terminal. This communication also takes place over the Internet.

[1236] Input: Response text data

[1237] Output: Response text data sent from the server to the terminal

[1238] Step 7:

[1239] The device converts the received response text data into speech using a speech synthesis engine (pyttsx3 library). For example, it may say, "It's a sunny day today. It might be nice to take a walk to change your mood."

[1240] Input: Response text data

[1241] Output: Response audio data

[1242] Step 8:

[1243] Sensors in the home (door sensors, motion detection sensors, etc.) periodically send data to a server, which monitors for abnormalities and reports them via audio if any are detected.

[1244] Input: Sensor data

[1245] Output: Notification of whether anomalies exist

[1246] Step 9:

[1247] In an emergency, if the user utters "help" or "emergency," the device recognizes this and the server quickly notifies emergency contacts. The notification is made using Twilio's API, and location information is also included.

[1248] Input: Emergency voice command

[1249] Output: Emergency call notification

[1250] Step 10:

[1251] As an emotional care feature, a personalized response is generated according to the user's emotional state, and in cooperation with the emotion engine, advice is given to help the user relax if they are feeling stressed.

[1252] Input: Emotional state data

[1253] Output: Personalized response text and voice data

[1254] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1255] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1256] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1257] [Third embodiment]

[1258] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1259] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1260] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1261] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1262] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1263] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1264] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1265] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1266] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1267] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1268] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1269] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1270] The present invention provides a system for providing a conversation partner for elderly people, preventing dementia, managing health, and supporting daily life. Specific embodiments for implementing the present invention will be described below.

[1271] This system includes a terminal for inputting voice from the elderly user, a means for converting voice into text data, a server for analyzing the text data and generating an appropriate response, a means for converting the generated response back into voice and notifying the user, a means for collecting and analyzing the user's health data and providing appropriate health advice, and a means for managing and notifying the elderly user's daily schedule.

[1272] Conversation Partners / Dementia Prevention

[1273] 1. Voice input

[1274] User: Talks to a smart speaker at home, for example, asking, "What's the weather like today?"

[1275] Terminal: Receives the user's voice and processes it as voice data.

[1276] 2. Converting voice data to text

[1277] Terminal: Converts received voice data into text data using voice recognition technology.

[1278] Terminal: Sends the converted text data to the server.

[1279] 3. Conversation Analysis and Response Generation

[1280] Server: Analyzes the received text data using natural language processing techniques and generates an appropriate response. In this example, it queries a weather API to obtain weather information.

[1281] Server: Generate a response such as "It's a sunny day today. Why don't you go for a walk?"

[1282] 4. Audio Output

[1283] Server: Sends the generated response to the terminal as text data.

[1284] Terminal: Text data is converted into speech using speech synthesis technology and notified to the user.

[1285] health care

[1286] 1. Health Data Collection

[1287] User: Weighs himself / herself and measures his / her blood pressure, and records the data on a smart device (e.g., a smartwatch or a scale).

[1288] Terminal: Obtain health data (e.g., weight "68kg") from a smart device.

[1289] 2. Data submission and analysis

[1290] Terminal: Sends acquired health data to the server.

[1291] Server: Receives health data and compares and analyzes it with past data.

[1292] 3. Health advice generation

[1293] Server: Based on the analysis results, the server generates specific health advice for the user. For example, it generates advice such as, "We recommend that you exercise for about 30 minutes today."

[1294] Server: Sends the generated advice to the terminal as text data.

[1295] 4. Notification

[1296] Terminal: Converts text data into speech and notifies the user.

[1297] Daily life support

[1298] 1. Receiving voice instructions

[1299] User: Speaks to a smart speaker and says, "Tell me what's on my schedule for today."

[1300] Terminal: Receives the user's voice and converts the voice data into text.

[1301] 2. Obtaining schedule information

[1302] Terminal: Sends the converted text data to the server.

[1303] Server: Queries the user's schedule database and retrieves the schedule for the day.

[1304] 3. Response Generation

[1305] Server: Based on the obtained schedule information, it generates a response such as, "I have a hospital appointment at 10:00. I plan to go shopping at the supermarket at 3:00 p.m."

[1306] Server: Sends the generated response to the terminal as text data.

[1307] 4. Notification

[1308] Terminal: Converts text data into speech and notifies the user.

[1309] As described above, this system aims to provide the elderly with daily support from various angles and improve their quality of life. By combining technologies such as voice recognition, natural language processing, health data analysis, and smart device integration, this invention realizes a support system that is highly convenient for the elderly.

[1310] The processing flow will be explained below.

[1311] Conversation Partners / Dementia Prevention

[1312] Voice input and response generation

[1313] Step 1:

[1314] User: Asks their smart speaker at home, "What's the weather like today?"

[1315] Step 2:

[1316] Terminal: Receives the user's voice and processes it as voice data.

[1317] Step 3:

[1318] Terminal: Converts received voice data into text data using voice recognition technology.

[1319] Step 4:

[1320] Terminal: Sends the converted text data to the server.

[1321] Step 5:

[1322] Server: Analyzes the received text data using natural language processing technology. Based on the analysis results, queries the weather API to obtain weather information.

[1323] Step 6:

[1324] Server: Receives the response from the weather API and generates a response (e.g., "It's a sunny day today. Why don't you go for a walk?").

[1325] Step 7:

[1326] Server: Sends the generated response to the terminal as text data.

[1327] Step 8:

[1328] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[1329] health care

[1330] Step 1:

[1331] User: Weighs themselves and measures their blood pressure, and records the data on a smart device (e.g., a smartwatch or scale).

[1332] Step 2:

[1333] Terminal: Health data (e.g., "Weight: 68 kg") is acquired from the smart device via Bluetooth communication, etc.

[1334] Step 3:

[1335] Terminal: Sends acquired health data to the server.

[1336] Step 4:

[1337] Server: Analyzes the received health data, compares it with past data, and evaluates the user's health condition.

[1338] Step 5:

[1339] Server: Based on the evaluation results, generate specific health advice for the user (e.g., "We recommend that you exercise for about 30 minutes today").

[1340] Step 6:

[1341] Server: The generated health advice is sent to the terminal as text data.

[1342] Step 7:

[1343] Terminal: Converts received text data into speech and notifies the user.

[1344] Daily life support

[1345] Step 1:

[1346] User: Speaks to a smart speaker and says, "Tell me what's on my schedule for today."

[1347] Step 2:

[1348] Terminal: Receives the user's voice and processes it as voice data.

[1349] Step 3:

[1350] Terminal: Converts received voice data into text data using voice recognition technology.

[1351] Step 4:

[1352] Terminal: Sends the converted text data to the server.

[1353] Step 5:

[1354] Server: Analyzes the received text data and queries the user's schedule database to obtain the schedule for the day.

[1355] Step 6:

[1356] Server: Generates a response based on the obtained schedule information (e.g., "I have a doctor's appointment at 10:00. I plan to go grocery shopping at 3:00 PM.").

[1357] Step 7:

[1358] Server: Sends the generated response to the terminal as text data.

[1359] Step 8:

[1360] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[1361] Through the above processing steps, the present invention provides support to help elderly people live healthy and fulfilling lives without feeling lonely.

[1362] Example 1

[1363] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1364] Currently, support systems for elderly people to smoothly lead their daily lives are inadequate. There is a particular need for comprehensive support that provides a wide range of support, including simple voice-based operation, health management, and schedule management. Furthermore, there is a lack of systems that allow elderly people to understand their own health status and receive appropriate advice. To address these issues, the present invention aims to provide a comprehensive support system based on voice input by elderly people.

[1365] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1366] In this invention, the server includes means for receiving a user's voice input, means for converting the received voice into text data, means for transmitting the converted text data to the processing device, means for analyzing the text data in the processing device and generating an appropriate response, means for transmitting the generated response in text format from the processing device to the display device, means for converting the received response into voice and notifying the user, means for collecting the user's health data, means for transmitting the collected health data to the processing device, means for analyzing the health data in the processing device and evaluating the user's health condition, means for generating health advice based on the evaluation results, means for transmitting the generated advice from the processing device to the display device, means for converting the received advice into voice and notifying the user, means for managing the user's daily schedule, means for receiving the user's voice instructions, means for transmitting the voice instructions to the processing device and acquiring schedule information, means for analyzing the acquired schedule information and generating a response to the user, means for transmitting the generated response from the processing device to the display device, and means for converting the received response into voice and notifying the user. This enables comprehensive support for the daily lives of elderly people in various aspects.

[1367] A "means for receiving voice input" is a device that recognizes the voice uttered by the user and captures it as digital data.

[1368] "Means for converting voice into text data" refers to an algorithm or system that uses voice recognition technology to convert received voice data into character string data.

[1369] The "means for transmitting text data to a processing device" is a function for transferring the converted text data to a device for performing other processing using a communication means such as a network.

[1370] "Means for analyzing text data using a processing device and generating an appropriate response" refers to a system that uses natural language processing technology to analyze the meaning and intent of received text data and generate an appropriate response to the user.

[1371] The "means for transmitting the generated response in text format from the processing device to the display device" is a function for transmitting the generated response in text data to another device for display.

[1372] The "means for converting the received response into voice and notifying the user" is a device that converts the response received as text data into voice using voice synthesis technology and transmits it to the user.

[1373] A "means for collecting health data" is a device that has the function of measuring and recording numerical health data such as a user's weight, blood pressure, and heart rate.

[1374] The "means for transmitting collected health data to a processing device" is a system that transmits collected health data to another processing device via a network.

[1375] "Means for analyzing health data using a processing device and assessing the user's health condition" refers to an algorithm or system that analyzes collected health data and understands and assesses the user's health condition.

[1376] The "means for generating health advice based on the evaluation results" is a system that generates specific advice for maintaining and improving the user's health based on the analysis results.

[1377] The "means for transmitting the generated advice from the processing device to the display device" is a function for transmitting the advice to the display device in order to notify the user of the generated health advice.

[1378] "Means for managing daily schedules" refers to a system that records and manages a user's daily plans and schedules.

[1379] The "means for receiving voice instructions" is a device that recognizes voice instructions from the user and captures them as digital data.

[1380] The "means for transmitting voice instructions to the processing device and acquiring schedule information" is a system for transmitting voice instructions to the processing device and acquiring the user's schedule information.

[1381] The "means for analyzing the acquired schedule information and generating a response to the user" refers to an algorithm or system that generates an appropriate response based on the schedule information.

[1382] The "means for transmitting the generated response from the processing device to the display device" is a function for transmitting the generated response to the display device and displaying it to the user.

[1383] The present invention provides a system for providing a conversation partner for elderly people, preventing dementia, managing health, and supporting daily life. Specific embodiments for implementing the present invention will be described below.

[1384] This system includes a terminal for inputting voice, a means for converting voice into text data, a server for analyzing the text data and generating an appropriate response, a means for converting the generated response into voice and notifying the user, a means for analyzing health data and providing appropriate health advice, and a means for managing and notifying the user of daily schedules.

[1385] The voice input in this system uses a smart speaker, such as Amazon Echo or Google Home, to capture the user's voice. The captured voice is then converted into text data using the Google Cloud Speech-to-Text API.

[1386] The text data is sent to a server, which then analyzes it using natural language processing technology, specifically a generative AI model such as GPT-4, to generate an appropriate response.

[1387] The generated response is then sent back to the device (such as a smart speaker or smartphone) and converted into audio using the Google Cloud Text-to-Speech API, which is then notified to the user.

[1388] Specific examples

[1389] Conversation Partners / Dementia Prevention

[1390] Prompt: "What's the weather like today?"

[1391] Response: "It's a beautiful sunny day today. Why don't you go for a walk?"

[1392] health care

[1393] Users' health data is collected using devices such as smartwatches and digital scales. The collected data is sent to a server via the device and analyzed there. Based on the analysis results, health advice for the user is generated, sent back to the device, and notified via voice.

[1394] Prompt: "I've weighed myself. What should I do next?"

[1395] Response: "According to our records, you weigh 140 lbs. I suggest you exercise for about 30 minutes today."

[1396] Daily life support

[1397] A smart speaker is used to manage daily schedules. When a user says, "Tell me what's on my schedule for today," the voice is converted into text data and sent to a server. The server retrieves the schedule using the Google Calendar API or similar, generates a response, and notifies the user by voice.

[1398] Prompt: "What's on your agenda for today?"

[1399] Response: "I have a doctor's appointment at 10:00. I'll be grocery shopping at 3:00."

[1400] As described above, this system aims to provide the elderly with daily support from various angles and improve their quality of life. By combining technologies such as voice recognition, natural language processing, health data analysis, and smart device integration, this invention realizes a support system that is highly convenient for the elderly.

[1401] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1402] Conversation Partners / Dementia Prevention

[1403] Step 1:

[1404] A user speaks to a smart speaker in their home, asking, for example, "What's the weather like today?"

[1405] Input: User's voice

[1406] Output: Audio data captured by the smart speaker

[1407] Step 2:

[1408] The device (smart speaker) converts the received audio into text data in real time using the Google Cloud Speech-to-Text API.

[1409] Input: Audio data

[1410] Output: Text data

[1411] Step 3:

[1412] The terminal transmits the converted text data to the server.

[1413] Input: Text data

[1414] Output: Text data sent to the server

[1415] Step 4:

[1416] The server analyzes the received text data using a natural language processing model (e.g., GPT-4). This analysis allows it to understand the user's intent. Next, the server queries an external weather API (e.g., OpenWeatherMap API) to obtain weather information.

[1417] Input: Text data

[1418] Output: Response text generated based on weather information

[1419] Step 5:

[1420] The server transmits the generated response to the terminal as text data.

[1421] Input: Response text

[1422] Output: Text data sent to the terminal

[1423] Step 6:

[1424] The device converts the received text data into audio using the Google Cloud Text-to-Speech API and notifies the user.

[1425] Input: Text data

[1426] Output: Audio data (notification audio)

[1427] health care

[1428] Step 1:

[1429] The user measures their weight and blood pressure and records the data on a smart device (e.g., a smart watch).

[1430] Input: Health data (weight, blood pressure, etc.)

[1431] Output: Data recorded on a smart device

[1432] Step 2:

[1433] The terminal (smartphone) acquires health data from the smart device via BLE (Bluetooth Low Energy).

[1434] Input: Data communication from smart devices

[1435] Output: Health data collected on the device

[1436] Step 3:

[1437] The terminal transmits the acquired health data to the server.

[1438] Input: Health data

[1439] Output: Health data sent to the server

[1440] Step 4:

[1441] The server compares and analyzes the received health data with past data to evaluate the user's health condition.

[1442] Input: Health data

[1443] Output: Analysis results (health assessment data)

[1444] Step 5:

[1445] Based on the analysis results, the server generates specific health advice for the user, such as "We recommend you exercise for about 30 minutes today."

[1446] Input: Health assessment data

[1447] Output: Generated health advice

[1448] Step 6:

[1449] The server transmits the generated advice to the terminal as text data.

[1450] Enter: Health Advice

[1451] Output: Text data sent to the terminal

[1452] Step 7:

[1453] The terminal converts the text data into voice and notifies the user.

[1454] Input: Text data

[1455] Output: Audio data (notification audio)

[1456] Daily life support

[1457] Step 1:

[1458] The user speaks to the smart speaker and says, "Tell me what's on my schedule for today."

[1459] Input: User's voice

[1460] Output: Audio data captured by the smart speaker

[1461] Step 2:

[1462] The device (smart speaker) converts the received voice data into text data using the Google Cloud Speech-to-Text API.

[1463] Input: Audio data

[1464] Output: Text data

[1465] Step 3:

[1466] The terminal transmits the text data to the server.

[1467] Input: Text data

[1468] Output: Text data sent to the server

[1469] Step 4:

[1470] The server analyzes the received text data and queries the user's schedule database (e.g., Google Calendar) to obtain the schedule for that day.

[1471] Input: Text data

[1472] Output: Schedule information

[1473] Step 5:

[1474] Based on the schedule information it has obtained, the server generates a response such as, "I have a hospital appointment at 10:00. I plan to go shopping at the supermarket at 3:00 p.m."

[1475] Input: Schedule information

[1476] Output: Generated response text data

[1477] Step 6:

[1478] The server transmits the generated response to the terminal as text data.

[1479] Input: Response text data

[1480] Output: Text data sent to the terminal

[1481] Step 7:

[1482] The terminal converts the text data into voice and notifies the user.

[1483] Input: Text data

[1484] Output: Audio data (notification audio)

[1485] (Application example 1)

[1486] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1487] Today's elderly people need support in many aspects of daily life. They may have difficulty managing their health and daily tasks, as well as having difficulty shopping and socializing. Shopping in brick-and-mortar stores can be particularly challenging, as they struggle to find the products they need and have difficulty accessing sales information. This often leads to a decline in the quality of life for elderly people and a sense of isolation. The present invention aims to address these challenges and improve the quality of life for elderly people.

[1488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1489] In this invention, the server includes means for receiving voice input from the elderly person, means for converting the received voice into text data, means for transmitting the converted text data to the server, means for analyzing the text data in the server and generating an appropriate response, means for transmitting the generated response in text format from the server to the terminal, means for converting the received response into voice and notifying the user, and means for acquiring and providing sale information using a generative AI model, thereby enabling elderly people to shop more smoothly in physical stores.

[1490] definition statement

[1491] "Elderly people" generally refer to people aged 65 and over who often require special assistance and support in their daily lives.

[1492] A "means for receiving voice input" is a device that allows a user to issue voice commands and has the ability to record and process the voice as digital data.

[1493] The "means for converting voice into text data" is a system that uses voice recognition technology to convert input voice into text data in real time.

[1494] "Means for transmitting text data to a server" refers to a function for transmitting converted text data to a server located at a remote location via a network.

[1495] "Means for analyzing text data and generating appropriate responses" refers to a function that uses natural language processing technology to understand the input text and generate the information or response the user is looking for.

[1496] "Means for transmitting the generated response in text format from the server to the terminal" refers to the function of transmitting the analysis results from the server to the user's terminal as text data.

[1497] The "means for converting the received response into voice and notifying the user" is a system that has the function of converting text data back into voice and conveying that voice to the user.

[1498] A "generative AI model" is an artificial intelligence model that has been trained in advance using large amounts of data and is capable of automatically generating a variety of responses and suggestions.

[1499] The "means for acquiring and providing sale information" refers to a system that utilizes inventory management systems and marketing data from physical stores to acquire the latest sale information and provide it to users.

[1500] MODE FOR CARRYING OUT THE INVENTION

[1501] The present invention is a system that provides sales information based on voice input and supports shopping in brick-and-mortar stores. This system uses speech recognition, text analysis, natural language processing, and generative AI models to generate specific responses and provide them to users, thereby supporting shopping for elderly people and others. Specific embodiments of the present invention are described below.

[1502] System configuration

[1503] Hardware:

[1504] Smartphone: Receives voice input, sends the speech recognition results to the server, and converts the text response from the server into speech to notify the user.

[1505] Server: Converts voice data into text data, analyzes it, and generates a response.

[1506] software:

[1507] Speech recognition library (e.g., speech_recognition library): Converts voice input into text data.

[1508] Natural language processing model: Analyzes input text data and generates appropriate responses.

[1509] Generative AI model: Used to determine special offers.

[1510] Text-to-speech libraries (e.g. gTTS library): Convert text responses into speech.

[1511] Data processing and calculation

[1512] Receiving and converting voice input:

[1513] The user speaks into the smartphone, and a speech recognition library (e.g., speech_recognition library) converts this speech into text data and sends it to the server.

[1514] Analyzing text data:

[1515] The server then analyzes the converted text data using natural language processing technology, determining the user's intent from the analysis results and identifying requests such as sales information.

[1516] Get sale information and generate a response:

[1517] The server uses the generative AI model to retrieve sales information from the physical store's inventory management system and marketing database, and then generates a response containing the information and suggestions the user is looking for.

[1518] Response transcription and announcement:

[1519] The generated response is sent to the smartphone as text data and converted into speech using a speech synthesis library (e.g., gTTS library). The converted speech is then notified to the user through the smartphone's speaker.

[1520] Examples and prompts

[1521] Examples:

[1522] A user speaks to their smartphone, "What are the specials this week?" The smartphone converts this speech into text data and sends it to a server. The server analyzes the text data and uses a generative AI model to obtain the special sale information. Based on the obtained special sale information, it generates a response such as, "This week's special is apples, 100 yen. They're in the vegetable section." The smartphone converts this response into speech and notifies the user.

[1523] Example prompt sentence:

[1524] "What are the specials this week?"

[1525] "Which department should I go to?"

[1526] This system will enable seniors to easily obtain shopping information at brick-and-mortar stores, ensuring they don't miss out on special offers. It is also expected that the natural voice interaction will make shopping smoother and more enjoyable.

[1527] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1528] Program processing steps

[1529] Step 1:

[1530] User speaks:

[1531] The user speaks a question or request into the smartphone, for example, "What are the specials this week?" This is the initial input to the system.

[1532] Step 2:

[1533] Audio to text conversion:

[1534] The user's voice is received by the smartphone's microphone and converted into text data using the speech_recognition library. At this stage, the input voice is converted into string data.

[1535] Step 3:

[1536] Sending text data to the server:

[1537] The text data obtained by speech recognition is sent from the smartphone to the server, which then forwards the request to the server via the network.

[1538] Step 4:

[1539] Text data analysis and response generation:

[1540] The server analyzes the received text data using natural language processing technology. Based on the analyzed text data, it understands the user's request and uses a generative AI model to obtain special sale information. It then generates a response based on the special sale information and prepares it in text data format.

[1541] Step 5:

[1542] Sending the generated response from the server to the device:

[1543] The generated text response is then sent from the server to the smartphone via the network, and the analysis results are returned to the user's device.

[1544] Step 6:

[1545] Text response transcription:

[1546] The smartphone converts the received text data into speech using the gTTS library, and outputs synthesized speech data based on the input text.

[1547] Step 7:

[1548] Notify user of voice response:

[1549] The converted response is then broadcast to the user through the smartphone's speaker, allowing the user to hear the response to their question.

[1550] Specific processing operations

[1551] Step 1:

[1552] The user speaks to the smartphone, "What are the specials this week?" The voice is received by the smartphone.

[1553] Step 2:

[1554] The voice data received by the smartphone's microphone is converted into a string using the speech_recognition library, and the text data "What are the specials this week?" is generated.

[1555] Step 3:

[1556] The text data generated by the speech recognition is sent from the smartphone to the server, where data communication takes place via the network.

[1557] Step 4:

[1558] The server uses natural language processing technology to analyze the received text data and understand the user's request. It then uses a generative AI model to retrieve sales information from the physical store's inventory management system and generates a response such as, "This week's sales item is apples, priced at 100 yen. They're in the vegetable section."

[1559] Step 5:

[1560] The text response generated by the server is sent over the network to the smartphone, which receives the text response.

[1561] Step 6:

[1562] The smartphone converts the received text data into speech using the gTTS library, generating speech data such as, "This week's special offer is apples, priced at 100 yen. They're in the vegetable section."

[1563] Step 7:

[1564] The user is notified of the voice-converted response through the smartphone's speaker, and can listen to the voice response.

[1565] Through the above process, shopping support for the elderly is realized.

[1566] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1567] This invention is a system that supports the lives of the elderly, combining voice recognition, natural language processing, health data analysis, smart device integration, and an emotion engine. This system provides various functions to help the elderly relieve loneliness, prevent dementia, and support health management.

[1568] 1. Conversation partner / dementia prevention

[1569] Voice input and response generation

[1570] When a user asks a question or speaks to a smart speaker, the voice input is received by the device. The device converts the voice data into text data and sends it to the server. The server analyzes the text data and generates an appropriate response. This process also incorporates an emotion engine that recognizes emotions from the user's voice. The tone and content of the response are adjusted based on the recognized emotion. The generated response is sent from the server to the device in text format, where it is converted into voice and notified to the user.

[1571] Examples:

[1572] User: "What's the weather like today?"

[1573] Terminal: Receives audio, converts it into text, and sends it to the server.

[1574] Server: Analyzes text data and obtains weather information. The emotion engine recognizes the user's emotion (e.g., feeling down), and generates a response based on that, such as "It's sunny today. Maybe you should take a walk to change your mood."

[1575] Terminal: Converts the response from the server into voice and notifies the user.

[1576] 2. Health management

[1577] Health data collection and analysis

[1578] Users use smart devices to collect health data. For example, they measure their body temperature and blood pressure with a smartwatch. The collected health data is sent to a server via the device. The server analyzes the received data and evaluates the user's health status. An emotion engine is also incorporated into this evaluation, and health advice is generated taking the user's emotional state into account.

[1579] Examples:

[1580] User: Weighs himself and records the data on a smart scale.

[1581] Terminal: Receives data from the scale and sends it to the server.

[1582] Server: Analyzes the data and generates health advice such as, "You seem a little low today. I recommend some light exercise."

[1583] Terminal: Converts the advice into voice and notifies the user.

[1584] 3. Daily life support

[1585] Schedule management and coordination

[1586] When a user asks a smart speaker about their daily schedule, the device receives the voice, converts it into text, and sends it to a server. The server then checks the user's schedule and uses an emotion engine to generate a response that takes into account the user's current emotional state.

[1587] Examples:

[1588] User: "What's on my schedule for today?"

[1589] Terminal: Receives audio, converts it into text, and sends it to the server.

[1590] Server: Checks the schedule. Generates a response saying, "I have a doctor's appointment at 10:00. I'm going to the supermarket at 3:00 PM." The emotion engine determines that the user is feeling stressed, and gives additional advice, saying, "Try not to push yourself too hard, and take things slowly."

[1591] Terminal: Converts responses and advice into voice and notifies the user.

[1592] As described above, the present invention provides a multi-functional support system that helps elderly people avoid loneliness and lead healthy and fulfilling lives. By adding emotion recognition functionality, the system achieves more advanced and personalized support.

[1593] The processing flow will be explained below.

[1594] Conversation Partners / Dementia Prevention

[1595] Voice input and response generation

[1596] Step 1:

[1597] User: Asks their smart speaker at home, "What's the weather like today?"

[1598] Step 2:

[1599] Terminal: Receives the user's voice and processes it as voice data.

[1600] Step 3:

[1601] Terminal: Converts received voice data into text data using voice recognition technology.

[1602] Step 4:

[1603] Terminal: Sends the converted text data to the server.

[1604] Step 5:

[1605] Server: Analyzes the received text data using natural language processing technology. Based on the analysis results, queries the weather API to obtain weather information.

[1606] Step 6:

[1607] Server: Receives the response from the weather API and generates text data.

[1608] Step 7:

[1609] Server: Recognizes emotions from the user's voice using the emotion engine.

[1610] Step 8:

[1611] Server: Adjusts the content and tone of the response based on the recognized emotion. For example, if a user is feeling down, the server generates a response like, "It's a sunny day today. Why don't you take a walk to cheer yourself up?"

[1612] Step 9:

[1613] Server: Sends the generated response to the terminal as text data.

[1614] Step 10:

[1615] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[1616] health care

[1617] Step 1:

[1618] User: Records health data such as weight and blood pressure on a smart device (e.g., smartwatch or scale).

[1619] Step 2:

[1620] Terminal: Health data is acquired from smart devices using communication methods such as Bluetooth.

[1621] Step 3:

[1622] Terminal: Sends acquired health data to the server.

[1623] Step 4:

[1624] Server: Analyzes the received data and evaluates the user's health condition.

[1625] Step 5:

[1626] Server: Analyzes the user's emotional data simultaneously using an emotion engine.

[1627] Step 6:

[1628] Server: Evaluates the overall physical and mental health status and generates appropriate health advice. For example, it generates advice such as, "You're feeling well today, but you seem a little tired. I recommend some light exercise."

[1629] Step 7:

[1630] Server: The generated health advice is sent to the terminal as text data.

[1631] Step 8:

[1632] Terminal: Converts received text data into speech and notifies the user.

[1633] Daily life support

[1634] Step 1:

[1635] User: Speaks to a smart speaker and says, "Tell me what's on my schedule for today."

[1636] Step 2:

[1637] Terminal: Receives the user's voice and processes it as voice data.

[1638] Step 3:

[1639] Terminal: Converts received voice data into text data using voice recognition technology.

[1640] Step 4:

[1641] Terminal: Sends the converted text data to the server.

[1642] Step 5:

[1643] Server: Analyzes the received text data and queries the user's schedule database to obtain the schedule for the day.

[1644] Step 6:

[1645] Server: Analyzes the acquired schedule information and generates text data.

[1646] Step 7:

[1647] Server: Uses an emotion engine to recognize the user's current emotional state.

[1648] Step 8:

[1649] Server: Adjust the tone and content of schedule notifications based on the recognized emotion. For example, if it determines that the user is feeling stressed, it generates a response such as, "You have a doctor's appointment at 10 o'clock today. Take it easy and take it easy."

[1650] Step 9:

[1651] Server: Sends the generated response to the terminal as text data.

[1652] Step 10:

[1653] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[1654] Through the above processing steps, the present invention provides a multi-functional support system that helps elderly people avoid loneliness and live healthy and fulfilling lives. The addition of emotion recognition functionality provides more advanced and personalized support.

[1655] Example 2

[1656] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1657] Elderly people often feel lonely in their daily lives and find health care a burden. Furthermore, there is a lack of support for dementia prevention and daily schedule management. These situations can reduce the quality of life for elderly people and lead to further health risks. Therefore, there is a need for the development of multifunctional and personalized systems to support the daily lives of elderly people.

[1658] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1659] In this invention, the server includes means for receiving voice input from the elderly person, means for converting the received voice into character data, means for transmitting the converted character data to a central control unit, means for analyzing the character data in the central control unit and generating an appropriate response (including an emotion engine), means for transmitting the generated response in character form from the central control unit to a terminal, and means for converting the received response into voice and notifying the elderly person. This reduces the elderly person's sense of loneliness, supports dementia prevention, and reduces the burden of health management.

[1660] "Means for receiving voice input" refers to devices or technology that detects voices made by the senior and processes the voice signals.

[1661] "Means for converting into character data" refers to technology that analyzes received voice signals, recognizes phonemes and words, and converts them into character strings.

[1662] "Means for transmitting to a server" refers to a device or technology that transmits the converted character data to a central control device (server) via a communications network.

[1663] "Central Controller" means a central control unit or system that analyzes received text data and generates an appropriate response.

[1664] "Means for generating an appropriate response" refers to technology that uses specific algorithms or emotion engines based on received data to generate an appropriate response to the user.

[1665] An "emotion engine" refers to an algorithm or system that analyzes emotions from received voice or data and generates a response based on those emotions.

[1666] "Means for sending to the terminal" refers to a device or technique for sending a response in the form of character data generated by the central control unit to the user's terminal.

[1667] "Means for converting the received response into voice" refers to a device or technology that synthesizes text data sent to the terminal into voice and notifies the user.

[1668] "Means for collecting health data" refers to devices and technologies for measuring health information such as a user's weight, body temperature, and blood pressure.

[1669] "Means for analyzing and evaluating health data" refers to algorithms or systems for analyzing collected health data and evaluating the user's health status.

[1670] The term "means for generating health advice" refers to a technology for generating health advice for a user based on the evaluation results of the health data.

[1671] "Means for managing daily schedules" refers to devices and technologies for managing a user's daily plans and tasks.

[1672] The term "means for receiving voice instructions" refers to a device or technology that detects voice instructions regarding the schedule given by the user and processes the voice signals.

[1673] "Means for obtaining schedule information" refers to devices and techniques for querying and obtaining the user's daily schedule information from the central control device.

[1674] The term "means for analyzing schedule information and generating a response" refers to a technique for analyzing acquired schedule information and generating an appropriate schedule-related response to the user.

[1675] This invention is a system for supporting the lives of the elderly, combining voice recognition, natural language processing, health data analysis, smart device integration, and an emotion engine. The system's main functions include serving as a conversation partner, health management, and daily schedule management for everyday use by the elderly.

[1676] The components of this system are:

[1677] 1. Conversation partner / dementia prevention

[1678] When a user asks a question or speaks to a smart speaker, the voice input is received by the device. The device converts the voice data into text data (using, for example, the Google Speech-to-Text API) and sends the converted data to a server. The server analyzes the text data using a natural language processing engine (for example, GPT-3) and generates an appropriate response. It also incorporates an emotion engine (for example, IBM Watson Tone Analyzer) that recognizes the user's emotions and adjusts the tone and content of the response. The generated response is sent from the server to the device in text format, where it is converted into speech and notified to the user (for example, using Amazon Polly).

[1679] Examples:

[1680] User: "What's the weather like today?"

[1681] Device: Receives audio, converts it to text using the Google Speech-to-Text API, and sends it to the server.

[1682] Server: Analyzes text data using GPT-3 and obtains weather information. The emotion engine recognizes the user's emotions and generates a response such as, "It's sunny today. It might be nice to take a walk to change your mood."

[1683] Terminal: The response from the server is converted into speech using Amazon Polly and notified to the user.

[1684] 2. Health management

[1685] Users collect their health data using a smart device. For example, they measure their body temperature and blood pressure with a smartwatch and send the data to the device. The device then sends the received data to a server, which analyzes it using a health data analysis algorithm (e.g., Google Fit API). An emotion engine is also incorporated into this evaluation, which generates health advice taking the user's emotional state into account. The advice is then sent from the server to the device, which converts it into audio and notifies the user.

[1686] Examples:

[1687] User: Weighs himself and records the data on a smart scale.

[1688] Terminal: Receives data from the scale and sends it to the server.

[1689] Server: Analyzes the data using the Google Fit API and generates health advice such as, "You seem a little low today. I recommend some light exercise."

[1690] Device: The advice is converted into speech using Amazon Polly and notified to the user.

[1691] 3. Daily life support

[1692] When a user speaks to a smart speaker to check their daily schedule, the device receives the voice, converts it into text data, and sends it to the server. The server then checks the user's daily schedule information and uses an emotion engine to generate a response that takes into account the user's current emotional state. The generated response is then sent from the server to the device, which converts it into voice and notifies the user.

[1693] Examples:

[1694] User: "What's on my schedule for today?"

[1695] Device: Receives audio, converts it to text using the Google Speech-to-Text API, and sends it to the server.

[1696] Server: Checks the schedule and generates a response saying, "I have a doctor's appointment at 10:00. I'm going to the supermarket at 3:00 PM." The emotion engine determines that the user is stressed and provides additional advice, saying, "Try not to push yourself too hard, and take things slowly."

[1697] Device: Responses and advice are converted into speech using Amazon Polly and communicated to the user.

[1698] Example prompts for generative AI models

[1699] "Please explain the function of an elderly life support system that uses voice recognition and an emotion engine to respond to user questions. Please also include specific processing procedures and examples when a user asks a question to a smart speaker."

[1700] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1701] 1. Conversation partner / dementia prevention

[1702] Step 1:

[1703] A user speaks to a smart speaker, such as "What's the weather like today?"

[1704] (Input) User's voice data

[1705] (Output) User's voice data

[1706] Step 2:

[1707] The device receives the voice data and converts it into text using voice recognition software (e.g., Google Speech-to-Text API).

[1708] (Input) User's voice data

[1709] (Output) The user's voice is converted into text data.

[1710] Step 3:

[1711] The terminal sends the converted text data to the server using a secure protocol such as HTTPS.

[1712] (Input) Character data

[1713] (Output) Character data sent to the server

[1714] Step 4:

[1715] The server uses a natural language processing engine (e.g., GPT-3) to analyze the text data, and at the same time, an emotion engine (e.g., IBM Watson Tone Analyzer) recognizes the user's emotions, which allows it to generate more appropriate responses.

[1716] (Input) Character data

[1717] (Data processing) Analyze emotions with an emotion engine and analyze data with a natural language processing engine

[1718] (Output) Analysis results and response data based on them

[1719] Step 5:

[1720] The server generates a response and sends it to the terminal in character data format.

[1721] (Input) Generated response data

[1722] (Output) Response data sent to the terminal

[1723] Step 6:

[1724] The response data received by the device is converted into speech using speech synthesis software (e.g., Amazon Polly) and notified to the user.

[1725] (Input) Response data

[1726] (Output) Response data converted to audio

[1727] 2. Health management

[1728] Step 1:

[1729] A user records health data using a health device (e.g., smart watch, smart scale, etc.).

[1730] (Input) User's health data (weight, body temperature, blood pressure, etc.)

[1731] (Output) Health data recorded on the health device

[1732] Step 2:

[1733] The terminal receives data from the health device using communication methods such as Bluetooth and Wi-Fi.

[1734] (Input) Data from health devices

[1735] (Output) Health data received by the device

[1736] Step 3:

[1737] The health data received by the device is sent to the server.

[1738] (Input) Health data

[1739] (Output) Health data sent to the server

[1740] Step 4:

[1741] The server uses a healthcare analysis algorithm (e.g., Google Fit API) to analyze the received health data, while also taking into account the user's emotional state using an emotion engine.

[1742] (Input) Health data

[1743] (Data processing) Health data analysis and emotional state assessment

[1744] (Output) Analysis results and health advice data

[1745] Step 5:

[1746] The server sends the generated health advice to the terminal.

[1747] (Input) Health advice data

[1748] (Output) Health advice data sent to the device

[1749] Step 6:

[1750] The health advice data received by the terminal is converted into voice using voice synthesis software and notified to the user.

[1751] (Input) Health advice data

[1752] (Output) Health advice converted into audio

[1753] 3. Daily life support

[1754] Step 1:

[1755] A user asks a smart speaker about their daily schedule, such as "Tell me what's on my schedule for today."

[1756] (Input) User's voice data

[1757] (Output) User's voice data

[1758] Step 2:

[1759] The device receives the audio and converts it into text data (e.g., Google Speech-to-Text API).

[1760] (Input) User's voice data

[1761] (Output) The user's voice is converted into text data.

[1762] Step 3:

[1763] The terminal sends character data to the server.

[1764] (Input) Character data

[1765] (Output) Character data sent to the server

[1766] Step 4:

[1767] The server analyzes the text data and queries the daily schedule database, while the emotion engine considers the user's current emotional state.

[1768] (Input) Character data, daily schedule database

[1769] (Data processing) Check schedule and evaluate emotional state

[1770] (Output) Schedule confirmation results and appropriate response data

[1771] Step 5:

[1772] The server sends the generated response data to the terminal.

[1773] (Input) Response data

[1774] (Output) Response data sent to the terminal

[1775] Step 6:

[1776] The response data received by the terminal is converted into voice using voice synthesis software and notified to the user.

[1777] (Input) Response data

[1778] (Output) The voiced response

[1779] (Application example 2)

[1780] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1781] In modern society, the safety and health management of elderly people has become an important issue. Elderly people living alone, in particular, need to respond quickly to emergencies and emergencies at home, and also need to alleviate feelings of loneliness and receive emotional care. However, current systems are unable to comprehensively meet these needs. Furthermore, technology for providing personalized support based on emotional state is immature. An advanced, integrated system to solve these issues is needed.

[1782] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice input from the elderly person, means for converting the received voice into text data, means for transmitting the converted text data to the server, means for analyzing the text data in the server and generating an appropriate response, means for transmitting the generated response in text format from the server to the terminal, means for converting the received response into voice and notifying the user, means for detecting abnormalities in the home and reporting if an abnormality is detected, means for making an emergency call based on the elderly person's voice instruction, and means for recognizing the user's emotional state and generating a response accordingly. This ensures the safety of the elderly person at home and enables rapid response in emergencies. Furthermore, personalized responses based on the elderly person's emotional state can alleviate feelings of loneliness and provide emotional care.

[1783] "Elderly" refers to individuals who are aging, especially those in an age group where health care and safety are of paramount importance.

[1784] "Audio input" refers to an audio signal collected using a microphone or other audio collection device.

[1785] "Text data" refers to data in the form of a string of characters generated from voice input or other digital data.

[1786] A "server" refers to a computer system that provides data processing and data storage over a network.

[1787] "Analysis" refers to the process of examining, interpreting, and evaluating data.

[1788] "Response" refers to information generated in response to a user input or request.

[1789] "Transmission" refers to the act of moving data from one point to another.

[1790] "Domestic anomalies" refers to abnormal or unexpected events or changes in the normal home environment.

[1791] "Reporting" refers to the act of communicating information about an anomaly or event to others.

[1792] "Emergency notification" refers to the prompt notification of pre-established contacts or organizations in an urgent situation.

[1793] "Emotional state" refers to an individual's psychological or emotional state at a particular point in time.

[1794] "Personalized responses" refer to responses that are customized based on the characteristics and circumstances of individual users.

[1795] This invention relates to a multifunctional system for supporting the lives of the elderly, specifically, a system that combines voice recognition, natural language processing, health data analysis, smart device integration, and an emotion engine to provide a variety of support functions.

[1796] 1. System Overview

[1797] The system consists of the following major components:

[1798] Voice input and text conversion methods

[1799] Text data analysis and response generation tools

[1800] Health data collection and analysis tools

[1801] In-home sensor monitor

[1802] Emotion Engine

[1803] Emergency notification means

[1804] 2. Program and Data Processing

[1805] The program of this system is built on Python and uses the following libraries and services.

[1806] Speech Recognition: Use the speech_recognition library to convert voice input into text data.

[1807] Speech synthesis: Uses the pyttsx3 library to convert text data into speech.

[1808] Emotion Recognition: Use your own emotion recognition model (e.g., a BERT-based model).

[1809] Emergency Calling: Call emergency contacts using APIs like Twilio.

[1810] Home sensor monitor: Collects data from various sensors installed in the home (e.g., door opening / closing sensors, motion detection sensors) and detects abnormalities.

[1811] 3. System processing overview

[1812] Speech Recognition and Emotion Analysis

[1813] The server receives voice input from the user and converts it to text using the speech_recognition library. The converted text is then analyzed through an emotion engine to evaluate the user's emotional state. Based on this evaluation, an appropriate response is generated.

[1814] Health data collection and analysis

[1815] A smart device (e.g., a smartwatch) worn by a user collects health data such as heart rate, body temperature, and blood pressure and sends it to a server. The server analyzes the received data and evaluates the user's health status. An emotion engine is also added to this evaluation, and health advice is generated taking the user's emotional state into account.

[1816] Detecting and reporting anomalies in the home

[1817] It monitors data from sensors installed in the home (e.g., door open / close sensors, motion detection sensors), and if an abnormality is detected, it notifies the user by voice. If the abnormality is serious, it automatically makes an emergency call.

[1818] emergency call

[1819] When a user utters keywords such as "help" or "emergency," the system determines this to be an emergency and notifies registered emergency contacts. The notification also includes location information, enabling a quick and accurate response.

[1820] 4. Description of use cases

[1821] Specific examples

[1822] When a user speaks to a smart device, asking, "What's the weather like today?", the server converts the speech into text data and analyzes it. The emotion engine evaluates the user's emotional state (e.g., feeling a little down), and generates a response such as, "It's sunny today. It might be nice to take a walk to change your mood." The generated response is synthesized into voice and notified to the user.

[1823] Furthermore, if the sensor that detects the user's movements detects an abnormality, the server will immediately notify them by voice, saying, "There is an abnormality. Please check." If the situation is serious, an emergency contact will be notified, allowing for a prompt response.

[1824] Prompt Sentence Examples

[1825] Generate a response to the question, "How's the weather today?" Consider an appropriate response if the user is elderly and indicates they are feeling a bit down.

[1826] This allows the system to provide personalized support appropriate to the elderly's emotional state, allowing them to live their daily lives with peace of mind.

[1827] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1828] Step 1:

[1829] The user speaks to the smart device, for example, "What's the weather like today?" This voice input is sent to the device via the microphone.

[1830] Input: User's voice

[1831] Output: Audio data

[1832] Step 2:

[1833] The device's microphone collects voice data and uses the speech_recognition library to convert this voice data into text data, which becomes "What's the weather like today?"

[1834] Input: Audio data

[1835] Output: Text data

[1836] Step 3:

[1837] The device then sends the converted text data to a server via the Internet.

[1838] Input: Text data

[1839] Output: Text data sent to the server

[1840] Step 4:

[1841] The server analyzes the received text data, using natural language processing (NLP) techniques to understand the user's intent and recognizes the user's emotional state through an emotion engine. For example, the server can determine from the user's tone of voice that they are "a little depressed."

[1842] Input: Text data

[1843] Output: Analysis results of text data and user's emotional state

[1844] Step 5:

[1845] The server generates an appropriate response based on the information it has obtained. It retrieves weather information from the weather API and generates a response that adds words of encouragement to a user who is feeling down, based on the results of the emotion engine. For example: "It's sunny today. Maybe it would be good to take a walk to change your mood."

[1846] Input: Text data analysis results, weather information, emotional state

[1847] Output: Response text data

[1848] Step 6:

[1849] The server then sends the generated response text data to the terminal. This communication also takes place over the Internet.

[1850] Input: Response text data

[1851] Output: Response text data sent from the server to the terminal

[1852] Step 7:

[1853] The device converts the received response text data into speech using a speech synthesis engine (pyttsx3 library). For example, it may say, "It's a sunny day today. It might be nice to take a walk to change your mood."

[1854] Input: Response text data

[1855] Output: Response audio data

[1856] Step 8:

[1857] Sensors in the home (door sensors, motion detection sensors, etc.) periodically send data to a server, which monitors for abnormalities and reports them via audio if any are detected.

[1858] Input: Sensor data

[1859] Output: Notification of whether anomalies exist

[1860] Step 9:

[1861] In an emergency, if the user utters "help" or "emergency," the device recognizes this and the server quickly notifies emergency contacts. The notification is made using Twilio's API, and location information is also included.

[1862] Input: Emergency voice command

[1863] Output: Emergency call notification

[1864] Step 10:

[1865] As an emotional care feature, a personalized response is generated according to the user's emotional state, and in cooperation with the emotion engine, advice is given to help the user relax if they are feeling stressed.

[1866] Input: Emotional state data

[1867] Output: Personalized response text and voice data

[1868] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1869] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1870] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1871] [Fourth embodiment]

[1872] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1873] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1874] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1875] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1876] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1877] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1878] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1879] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1880] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1881] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1882] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1883] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1884] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1885] The present invention provides a system for providing a conversation partner for elderly people, preventing dementia, managing health, and supporting daily life. Specific embodiments for implementing the present invention will be described below.

[1886] This system includes a terminal for inputting voice from the elderly user, a means for converting voice into text data, a server for analyzing the text data and generating an appropriate response, a means for converting the generated response back into voice and notifying the user, a means for collecting and analyzing the user's health data and providing appropriate health advice, and a means for managing and notifying the elderly user's daily schedule.

[1887] Conversation Partners / Dementia Prevention

[1888] 1. Voice input

[1889] User: Talks to a smart speaker at home, for example, asking, "What's the weather like today?"

[1890] Terminal: Receives the user's voice and processes it as voice data.

[1891] 2. Converting voice data to text

[1892] Terminal: Converts received voice data into text data using voice recognition technology.

[1893] Terminal: Sends the converted text data to the server.

[1894] 3. Conversation Analysis and Response Generation

[1895] Server: Analyzes the received text data using natural language processing techniques and generates an appropriate response. In this example, it queries a weather API to obtain weather information.

[1896] Server: Generate a response such as "It's a sunny day today. Why don't you go for a walk?"

[1897] 4. Audio Output

[1898] Server: Sends the generated response to the terminal as text data.

[1899] Terminal: Text data is converted into speech using speech synthesis technology and notified to the user.

[1900] health care

[1901] 1. Health Data Collection

[1902] User: Weighs himself / herself and measures his / her blood pressure, and records the data on a smart device (e.g., a smartwatch or a scale).

[1903] Terminal: Obtain health data (e.g., weight "68kg") from a smart device.

[1904] 2. Data submission and analysis

[1905] Terminal: Sends acquired health data to the server.

[1906] Server: Receives health data and compares and analyzes it with past data.

[1907] 3. Health advice generation

[1908] Server: Based on the analysis results, the server generates specific health advice for the user. For example, it generates advice such as, "We recommend that you exercise for about 30 minutes today."

[1909] Server: Sends the generated advice to the terminal as text data.

[1910] 4. Notification

[1911] Terminal: Converts text data into speech and notifies the user.

[1912] Daily life support

[1913] 1. Receiving voice instructions

[1914] User: Speaks to a smart speaker and says, "Tell me what's on my schedule for today."

[1915] Terminal: Receives the user's voice and converts the voice data into text.

[1916] 2. Obtaining schedule information

[1917] Terminal: Sends the converted text data to the server.

[1918] Server: Queries the user's schedule database and retrieves the schedule for the day.

[1919] 3. Response Generation

[1920] Server: Based on the obtained schedule information, it generates a response such as, "I have a hospital appointment at 10:00. I plan to go shopping at the supermarket at 3:00 p.m."

[1921] Server: Sends the generated response to the terminal as text data.

[1922] 4. Notification

[1923] Terminal: Converts text data into speech and notifies the user.

[1924] As described above, this system aims to provide the elderly with daily support from various angles and improve their quality of life. By combining technologies such as voice recognition, natural language processing, health data analysis, and smart device integration, this invention realizes a support system that is highly convenient for the elderly.

[1925] The processing flow will be explained below.

[1926] Conversation Partners / Dementia Prevention

[1927] Voice input and response generation

[1928] Step 1:

[1929] User: Asks their smart speaker at home, "What's the weather like today?"

[1930] Step 2:

[1931] Terminal: Receives the user's voice and processes it as voice data.

[1932] Step 3:

[1933] Terminal: Converts received voice data into text data using voice recognition technology.

[1934] Step 4:

[1935] Terminal: Sends the converted text data to the server.

[1936] Step 5:

[1937] Server: Analyzes the received text data using natural language processing technology. Based on the analysis results, queries the weather API to obtain weather information.

[1938] Step 6:

[1939] Server: Receives the response from the weather API and generates a response (e.g., "It's a sunny day today. Why don't you go for a walk?").

[1940] Step 7:

[1941] Server: Sends the generated response to the terminal as text data.

[1942] Step 8:

[1943] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[1944] health care

[1945] Step 1:

[1946] User: Weighs themselves and measures their blood pressure, and records the data on a smart device (e.g., a smartwatch or scale).

[1947] Step 2:

[1948] Terminal: Health data (e.g., "Weight: 68 kg") is acquired from the smart device via Bluetooth communication, etc.

[1949] Step 3:

[1950] Terminal: Sends acquired health data to the server.

[1951] Step 4:

[1952] Server: Analyzes the received health data, compares it with past data, and evaluates the user's health condition.

[1953] Step 5:

[1954] Server: Based on the evaluation results, generate specific health advice for the user (e.g., "We recommend that you exercise for about 30 minutes today").

[1955] Step 6:

[1956] Server: The generated health advice is sent to the terminal as text data.

[1957] Step 7:

[1958] Terminal: Converts received text data into speech and notifies the user.

[1959] Daily life support

[1960] Step 1:

[1961] User: Speaks to a smart speaker and says, "Tell me what's on my schedule for today."

[1962] Step 2:

[1963] Terminal: Receives the user's voice and processes it as voice data.

[1964] Step 3:

[1965] Terminal: Converts received voice data into text data using voice recognition technology.

[1966] Step 4:

[1967] Terminal: Sends the converted text data to the server.

[1968] Step 5:

[1969] Server: Analyzes the received text data and queries the user's schedule database to obtain the schedule for the day.

[1970] Step 6:

[1971] Server: Generates a response based on the obtained schedule information (e.g., "I have a doctor's appointment at 10:00. I plan to go grocery shopping at 3:00 PM.").

[1972] Step 7:

[1973] Server: Sends the generated response to the terminal as text data.

[1974] Step 8:

[1975] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[1976] Through the above processing steps, the present invention provides support to help elderly people live healthy and fulfilling lives without feeling lonely.

[1977] Example 1

[1978] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1979] Currently, support systems for elderly people to smoothly lead their daily lives are inadequate. There is a particular need for comprehensive support that provides a wide range of support, including simple voice-based operation, health management, and schedule management. Furthermore, there is a lack of systems that allow elderly people to understand their own health status and receive appropriate advice. To address these issues, the present invention aims to provide a comprehensive support system based on voice input by elderly people.

[1980] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1981] In this invention, the server includes means for receiving a user's voice input, means for converting the received voice into text data, means for transmitting the converted text data to the processing device, means for analyzing the text data in the processing device and generating an appropriate response, means for transmitting the generated response in text format from the processing device to the display device, means for converting the received response into voice and notifying the user, means for collecting the user's health data, means for transmitting the collected health data to the processing device, means for analyzing the health data in the processing device and evaluating the user's health condition, means for generating health advice based on the evaluation results, means for transmitting the generated advice from the processing device to the display device, means for converting the received advice into voice and notifying the user, means for managing the user's daily schedule, means for receiving the user's voice instructions, means for transmitting the voice instructions to the processing device and acquiring schedule information, means for analyzing the acquired schedule information and generating a response to the user, means for transmitting the generated response from the processing device to the display device, and means for converting the received response into voice and notifying the user. This enables comprehensive support for the daily lives of elderly people in various aspects.

[1982] A "means for receiving voice input" is a device that recognizes the voice uttered by the user and captures it as digital data.

[1983] "Means for converting voice into text data" refers to an algorithm or system that uses voice recognition technology to convert received voice data into character string data.

[1984] The "means for transmitting text data to a processing device" is a function for transferring the converted text data to a device for performing other processing using a communication means such as a network.

[1985] "Means for analyzing text data using a processing device and generating an appropriate response" refers to a system that uses natural language processing technology to analyze the meaning and intent of received text data and generate an appropriate response to the user.

[1986] The "means for transmitting the generated response in text format from the processing device to the display device" is a function for transmitting the generated response in text data to another device for display.

[1987] The "means for converting the received response into voice and notifying the user" is a device that converts the response received as text data into voice using voice synthesis technology and transmits it to the user.

[1988] A "means for collecting health data" is a device that has the function of measuring and recording numerical health data such as a user's weight, blood pressure, and heart rate.

[1989] The "means for transmitting collected health data to a processing device" is a system that transmits collected health data to another processing device via a network.

[1990] "Means for analyzing health data using a processing device and assessing the user's health condition" refers to an algorithm or system that analyzes collected health data and understands and assesses the user's health condition.

[1991] The "means for generating health advice based on the evaluation results" is a system that generates specific advice for maintaining and improving the user's health based on the analysis results.

[1992] The "means for transmitting the generated advice from the processing device to the display device" is a function for transmitting the advice to the display device in order to notify the user of the generated health advice.

[1993] "Means for managing daily schedules" refers to a system that records and manages a user's daily plans and schedules.

[1994] The "means for receiving voice instructions" is a device that recognizes voice instructions from the user and captures them as digital data.

[1995] The "means for transmitting voice instructions to the processing device and acquiring schedule information" is a system for transmitting voice instructions to the processing device and acquiring the user's schedule information.

[1996] The "means for analyzing the acquired schedule information and generating a response to the user" refers to an algorithm or system that generates an appropriate response based on the schedule information.

[1997] The "means for transmitting the generated response from the processing device to the display device" is a function for transmitting the generated response to the display device and displaying it to the user.

[1998] The present invention provides a system for providing a conversation partner for elderly people, preventing dementia, managing health, and supporting daily life. Specific embodiments for implementing the present invention will be described below.

[1999] This system includes a terminal for inputting voice, a means for converting voice into text data, a server for analyzing the text data and generating an appropriate response, a means for converting the generated response into voice and notifying the user, a means for analyzing health data and providing appropriate health advice, and a means for managing and notifying the user of daily schedules.

[2000] The voice input in this system uses a smart speaker, such as Amazon Echo or Google Home, to capture the user's voice. The captured voice is then converted into text data using the Google Cloud Speech-to-Text API.

[2001] The text data is sent to a server, which then analyzes it using natural language processing technology, specifically a generative AI model such as GPT-4, to generate an appropriate response.

[2002] The generated response is then sent back to the device (such as a smart speaker or smartphone) and converted into audio using the Google Cloud Text-to-Speech API, which is then notified to the user.

[2003] Specific examples

[2004] Conversation Partners / Dementia Prevention

[2005] Prompt: "What's the weather like today?"

[2006] Response: "It's a beautiful sunny day today. Why don't you go for a walk?"

[2007] health care

[2008] Users' health data is collected using devices such as smartwatches and digital scales. The collected data is sent to a server via the device and analyzed there. Based on the analysis results, health advice for the user is generated, sent back to the device, and notified via voice.

[2009] Prompt: "I've weighed myself. What should I do next?"

[2010] Response: "According to our records, you weigh 140 lbs. I suggest you exercise for about 30 minutes today."

[2011] Daily life support

[2012] A smart speaker is used to manage daily schedules. When a user says, "Tell me what's on my schedule for today," the voice is converted into text data and sent to a server. The server retrieves the schedule using the Google Calendar API or similar, generates a response, and notifies the user by voice.

[2013] Prompt: "What's on your agenda for today?"

[2014] Response: "I have a doctor's appointment at 10:00. I'll be grocery shopping at 3:00."

[2015] As described above, this system aims to provide the elderly with daily support from various angles and improve their quality of life. By combining technologies such as voice recognition, natural language processing, health data analysis, and smart device integration, this invention realizes a support system that is highly convenient for the elderly.

[2016] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2017] Conversation Partners / Dementia Prevention

[2018] Step 1:

[2019] A user speaks to a smart speaker in their home, asking, for example, "What's the weather like today?"

[2020] Input: User's voice

[2021] Output: Audio data captured by the smart speaker

[2022] Step 2:

[2023] The device (smart speaker) converts the received audio into text data in real time using the Google Cloud Speech-to-Text API.

[2024] Input: Audio data

[2025] Output: Text data

[2026] Step 3:

[2027] The terminal transmits the converted text data to the server.

[2028] Input: Text data

[2029] Output: Text data sent to the server

[2030] Step 4:

[2031] The server analyzes the received text data using a natural language processing model (e.g., GPT-4). This analysis allows it to understand the user's intent. Next, the server queries an external weather API (e.g., OpenWeatherMap API) to obtain weather information.

[2032] Input: Text data

[2033] Output: Response text generated based on weather information

[2034] Step 5:

[2035] The server transmits the generated response to the terminal as text data.

[2036] Input: Response text

[2037] Output: Text data sent to the terminal

[2038] Step 6:

[2039] The device converts the received text data into audio using the Google Cloud Text-to-Speech API and notifies the user.

[2040] Input: Text data

[2041] Output: Audio data (notification audio)

[2042] health care

[2043] Step 1:

[2044] The user measures their weight and blood pressure and records the data on a smart device (e.g., a smart watch).

[2045] Input: Health data (weight, blood pressure, etc.)

[2046] Output: Data recorded on a smart device

[2047] Step 2:

[2048] The terminal (smartphone) acquires health data from the smart device via BLE (Bluetooth Low Energy).

[2049] Input: Data communication from smart devices

[2050] Output: Health data collected on the device

[2051] Step 3:

[2052] The terminal transmits the acquired health data to the server.

[2053] Input: Health data

[2054] Output: Health data sent to the server

[2055] Step 4:

[2056] The server compares and analyzes the received health data with past data to evaluate the user's health condition.

[2057] Input: Health data

[2058] Output: Analysis results (health assessment data)

[2059] Step 5:

[2060] Based on the analysis results, the server generates specific health advice for the user, such as "We recommend you exercise for about 30 minutes today."

[2061] Input: Health assessment data

[2062] Output: Generated health advice

[2063] Step 6:

[2064] The server transmits the generated advice to the terminal as text data.

[2065] Enter: Health Advice

[2066] Output: Text data sent to the terminal

[2067] Step 7:

[2068] The terminal converts the text data into voice and notifies the user.

[2069] Input: Text data

[2070] Output: Audio data (notification audio)

[2071] Daily life support

[2072] Step 1:

[2073] The user speaks to the smart speaker and says, "Tell me what's on my schedule for today."

[2074] Input: User's voice

[2075] Output: Audio data captured by the smart speaker

[2076] Step 2:

[2077] The device (smart speaker) converts the received voice data into text data using the Google Cloud Speech-to-Text API.

[2078] Input: Audio data

[2079] Output: Text data

[2080] Step 3:

[2081] The terminal transmits the text data to the server.

[2082] Input: Text data

[2083] Output: Text data sent to the server

[2084] Step 4:

[2085] The server analyzes the received text data and queries the user's schedule database (e.g., Google Calendar) to obtain the schedule for that day.

[2086] Input: Text data

[2087] Output: Schedule information

[2088] Step 5:

[2089] Based on the schedule information it has obtained, the server generates a response such as, "I have a hospital appointment at 10:00. I plan to go shopping at the supermarket at 3:00 p.m."

[2090] Input: Schedule information

[2091] Output: Generated response text data

[2092] Step 6:

[2093] The server transmits the generated response to the terminal as text data.

[2094] Input: Response text data

[2095] Output: Text data sent to the terminal

[2096] Step 7:

[2097] The terminal converts the text data into voice and notifies the user.

[2098] Input: Text data

[2099] Output: Audio data (notification audio)

[2100] (Application example 1)

[2101] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2102] Today's elderly people need support in many aspects of daily life. They may have difficulty managing their health and daily tasks, as well as having difficulty shopping and socializing. Shopping in brick-and-mortar stores can be particularly challenging, as they struggle to find the products they need and have difficulty accessing sales information. This often leads to a decline in the quality of life for elderly people and a sense of isolation. The present invention aims to address these challenges and improve the quality of life for elderly people.

[2103] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2104] In this invention, the server includes means for receiving voice input from the elderly person, means for converting the received voice into text data, means for transmitting the converted text data to the server, means for analyzing the text data in the server and generating an appropriate response, means for transmitting the generated response in text format from the server to the terminal, means for converting the received response into voice and notifying the user, and means for acquiring and providing sale information using a generative AI model, thereby enabling elderly people to shop more smoothly in physical stores.

[2105] definition statement

[2106] "Elderly people" generally refer to people aged 65 and over who often require special assistance and support in their daily lives.

[2107] A "means for receiving voice input" is a device that allows a user to issue voice commands and has the ability to record and process the voice as digital data.

[2108] The "means for converting voice into text data" is a system that uses voice recognition technology to convert input voice into text data in real time.

[2109] "Means for transmitting text data to a server" refers to a function for transmitting converted text data to a server located at a remote location via a network.

[2110] "Means for analyzing text data and generating appropriate responses" refers to a function that uses natural language processing technology to understand the input text and generate the information or response the user is looking for.

[2111] "Means for transmitting the generated response in text format from the server to the terminal" refers to the function of transmitting the analysis results from the server to the user's terminal as text data.

[2112] The "means for converting the received response into voice and notifying the user" is a system that has the function of converting text data back into voice and conveying that voice to the user.

[2113] A "generative AI model" is an artificial intelligence model that has been trained in advance using large amounts of data and is capable of automatically generating a variety of responses and suggestions.

[2114] The "means for acquiring and providing sale information" refers to a system that utilizes inventory management systems and marketing data from physical stores to acquire the latest sale information and provide it to users.

[2115] MODE FOR CARRYING OUT THE INVENTION

[2116] The present invention is a system that provides sales information based on voice input and supports shopping in brick-and-mortar stores. This system uses speech recognition, text analysis, natural language processing, and generative AI models to generate specific responses and provide them to users, thereby supporting shopping for elderly people and others. Specific embodiments of the present invention are described below.

[2117] System configuration

[2118] Hardware:

[2119] Smartphone: Receives voice input, sends the speech recognition results to the server, and converts the text response from the server into speech to notify the user.

[2120] Server: Converts voice data into text data, analyzes it, and generates a response.

[2121] software:

[2122] Speech recognition library (e.g., speech_recognition library): Converts voice input into text data.

[2123] Natural language processing model: Analyzes input text data and generates appropriate responses.

[2124] Generative AI model: Used to determine special offers.

[2125] Text-to-speech libraries (e.g. gTTS library): Convert text responses into speech.

[2126] Data processing and calculation

[2127] Receiving and converting voice input:

[2128] The user speaks into the smartphone, and a speech recognition library (e.g., speech_recognition library) converts this speech into text data and sends it to the server.

[2129] Analyzing text data:

[2130] The server then analyzes the converted text data using natural language processing technology, determining the user's intent from the analysis results and identifying requests such as sales information.

[2131] Get sale information and generate a response:

[2132] The server uses the generative AI model to retrieve sales information from the physical store's inventory management system and marketing database, and generates a response based on the sales information, including the information and suggestions the user is looking for.

[2133] Response transcription and announcement:

[2134] The generated response is sent to the smartphone as text data and converted into speech using a speech synthesis library (e.g., gTTS library). The converted speech is then notified to the user through the smartphone's speaker.

[2135] Examples and prompts

[2136] Examples:

[2137] A user speaks to their smartphone, "What are the specials this week?" The smartphone converts this speech into text data and sends it to a server. The server analyzes the text data and uses a generative AI model to obtain the special sale information. Based on the obtained special sale information, it generates a response such as, "This week's special is apples, 100 yen. They're in the vegetable section." The smartphone converts this response into speech and notifies the user.

[2138] Example prompt sentence:

[2139] "What are the specials this week?"

[2140] "Which department should I go to?"

[2141] This system will enable seniors to easily obtain shopping information at brick-and-mortar stores, ensuring they don't miss out on special offers. It is also expected that the natural voice interaction will make shopping smoother and more enjoyable.

[2142] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2143] Program processing steps

[2144] Step 1:

[2145] User speaks:

[2146] The user speaks a question or request into the smartphone, for example, "What are the specials this week?" This is the initial input to the system.

[2147] Step 2:

[2148] Audio to text conversion:

[2149] The user's voice is received by the smartphone's microphone and converted into text data using the speech_recognition library. At this stage, the input voice is converted into string data.

[2150] Step 3:

[2151] Sending text data to the server:

[2152] The text data obtained by speech recognition is sent from the smartphone to the server, which then forwards the request to the server via the network.

[2153] Step 4:

[2154] Text data analysis and response generation:

[2155] The server analyzes the received text data using natural language processing technology. Based on the analyzed text data, it understands the user's request and uses a generative AI model to obtain special sale information. It then generates a response based on the special sale information and prepares it in text data format.

[2156] Step 5:

[2157] Sending the generated response from the server to the device:

[2158] The generated text response is then sent from the server to the smartphone via the network, and the analysis results are returned to the user's device.

[2159] Step 6:

[2160] Text response transcription:

[2161] The smartphone converts the received text data into speech using the gTTS library, and outputs synthesized speech data based on the input text.

[2162] Step 7:

[2163] Notify user of voice response:

[2164] The converted response is then broadcast to the user through the smartphone's speaker, allowing the user to hear the response to their question.

[2165] Specific processing operations

[2166] Step 1:

[2167] The user speaks to the smartphone, "What are the specials this week?" The voice is received by the smartphone.

[2168] Step 2:

[2169] The voice data received by the smartphone's microphone is converted into a string using the speech_recognition library, and the text data "What are the specials this week?" is generated.

[2170] Step 3:

[2171] The text data generated by the speech recognition is sent from the smartphone to the server, where data communication takes place via the network.

[2172] Step 4:

[2173] The server uses natural language processing technology to analyze the received text data and understand the user's request. It then uses a generative AI model to retrieve sales information from the physical store's inventory management system and generates a response such as, "This week's sales item is apples, priced at 100 yen. They're in the vegetable section."

[2174] Step 5:

[2175] The text response generated by the server is sent over the network to the smartphone, which receives the text response.

[2176] Step 6:

[2177] The smartphone converts the received text data into speech using the gTTS library, generating speech data such as, "This week's special offer is apples, priced at 100 yen. They're in the vegetable section."

[2178] Step 7:

[2179] The user is notified of the voice-converted response through the smartphone's speaker, and can listen to the voice response.

[2180] Through the above process, shopping support for the elderly is realized.

[2181] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2182] This invention is a system that supports the lives of the elderly, combining voice recognition, natural language processing, health data analysis, smart device integration, and an emotion engine. This system provides various functions to help the elderly relieve loneliness, prevent dementia, and support health management.

[2183] 1. Conversation partner / dementia prevention

[2184] Voice input and response generation

[2185] When a user asks a question or speaks to a smart speaker, the voice input is received by the device. The device converts the voice data into text data and sends it to the server. The server analyzes the text data and generates an appropriate response. This process also incorporates an emotion engine that recognizes emotions from the user's voice. The tone and content of the response are adjusted based on the recognized emotion. The generated response is sent from the server in text format to the device, which then converts it into voice and notifies the user.

[2186] Examples:

[2187] User: "What's the weather like today?"

[2188] Terminal: Receives audio, converts it into text, and sends it to the server.

[2189] Server: Analyzes text data and obtains weather information. The emotion engine recognizes the user's emotion (e.g., feeling down), and generates a response based on that, such as "It's sunny today. Maybe you should take a walk to change your mood."

[2190] Terminal: Converts the response from the server into voice and notifies the user.

[2191] 2. Health management

[2192] Health data collection and analysis

[2193] Users use smart devices to collect health data. For example, they measure their body temperature and blood pressure with a smartwatch. The collected health data is sent to a server via the device. The server analyzes the received data and evaluates the user's health status. An emotion engine is also incorporated into this evaluation, and health advice is generated taking the user's emotional state into account.

[2194] Examples:

[2195] User: Weighs himself and records the data on a smart scale.

[2196] Terminal: Receives data from the scale and sends it to the server.

[2197] Server: Analyzes the data and generates health advice such as, "You seem a little low today. I recommend some light exercise."

[2198] Terminal: Converts the advice into voice and notifies the user.

[2199] 3. Daily life support

[2200] Schedule management and coordination

[2201] When a user asks a smart speaker about their daily schedule, the device receives the voice, converts it into text, and sends it to a server. The server then checks the user's schedule and uses an emotion engine to generate a response that takes into account the user's current emotional state.

[2202] Examples:

[2203] User: "What's on my schedule for today?"

[2204] Terminal: Receives audio, converts it into text, and sends it to the server.

[2205] Server: Checks the schedule. Generates a response saying, "I have a doctor's appointment at 10:00. I'm going to the supermarket at 3:00 PM." The emotion engine determines that the user is feeling stressed, and gives additional advice, saying, "Try not to push yourself too hard, and take things slowly."

[2206] Terminal: Converts responses and advice into voice and notifies the user.

[2207] As described above, the present invention provides a multi-functional support system that helps elderly people live healthy and fulfilling lives without feeling lonely. By adding emotion recognition functionality, the system achieves more advanced and personalized support.

[2208] The processing flow will be explained below.

[2209] Conversation Partners / Dementia Prevention

[2210] Voice input and response generation

[2211] Step 1:

[2212] User: Asks their smart speaker at home, "What's the weather like today?"

[2213] Step 2:

[2214] Terminal: Receives the user's voice and processes it as voice data.

[2215] Step 3:

[2216] Terminal: Converts received voice data into text data using voice recognition technology.

[2217] Step 4:

[2218] Terminal: Sends the converted text data to the server.

[2219] Step 5:

[2220] Server: Analyzes the received text data using natural language processing technology. Based on the analysis results, queries the weather API to obtain weather information.

[2221] Step 6:

[2222] Server: Receives the response from the weather API and generates text data.

[2223] Step 7:

[2224] Server: Recognizes emotions from the user's voice using the emotion engine.

[2225] Step 8:

[2226] Server: Adjusts the content and tone of the response based on the recognized emotion. For example, if a user is feeling down, the server generates a response like, "It's a sunny day today. Why don't you take a walk to cheer yourself up?"

[2227] Step 9:

[2228] Server: Sends the generated response to the terminal as text data.

[2229] Step 10:

[2230] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[2231] health care

[2232] Step 1:

[2233] User: Records health data such as weight and blood pressure on a smart device (e.g., smartwatch or scale).

[2234] Step 2:

[2235] Terminal: Health data is acquired from smart devices using communication methods such as Bluetooth.

[2236] Step 3:

[2237] Terminal: Sends acquired health data to the server.

[2238] Step 4:

[2239] Server: Analyzes the received data and evaluates the user's health condition.

[2240] Step 5:

[2241] Server: Analyzes the user's emotional data simultaneously using an emotion engine.

[2242] Step 6:

[2243] Server: Evaluates the overall physical and mental health status and generates appropriate health advice. For example, it generates advice such as, "You're feeling good today, but you seem a little tired. I recommend some light exercise."

[2244] Step 7:

[2245] Server: The generated health advice is sent to the terminal as text data.

[2246] Step 8:

[2247] Terminal: Converts received text data into speech and notifies the user.

[2248] Daily life support

[2249] Step 1:

[2250] User: Speaks to a smart speaker and says, "Tell me what's on my schedule for today."

[2251] Step 2:

[2252] Terminal: Receives the user's voice and processes it as voice data.

[2253] Step 3:

[2254] Terminal: Converts received voice data into text data using voice recognition technology.

[2255] Step 4:

[2256] Terminal: Sends the converted text data to the server.

[2257] Step 5:

[2258] Server: Analyzes the received text data and queries the user's schedule database to obtain the schedule for the day.

[2259] Step 6:

[2260] Server: Analyzes the acquired schedule information and generates text data.

[2261] Step 7:

[2262] Server: Uses an emotion engine to recognize the user's current emotional state.

[2263] Step 8:

[2264] Server: Adjust the tone and content of schedule notifications based on the recognized emotion. For example, if it determines that the user is feeling stressed, it generates a response such as, "You have a doctor's appointment at 10 o'clock today. Take it easy and take it easy."

[2265] Step 9:

[2266] Server: Sends the generated response to the terminal as text data.

[2267] Step 10:

[2268] Terminal: Converts received text data into speech using speech synthesis technology and notifies the user.

[2269] Through the above processing steps, the present invention provides a multi-functional support system that helps elderly people avoid loneliness and live healthy and fulfilling lives. The addition of emotion recognition functionality provides more advanced and personalized support.

[2270] Example 2

[2271] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2272] Elderly people often feel lonely in their daily lives and find health care a burden. Furthermore, there is a lack of support for dementia prevention and daily schedule management. These situations can reduce the quality of life for elderly people and lead to further health risks. Therefore, there is a need for the development of multifunctional and personalized systems to support the daily lives of elderly people.

[2273] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2274] In this invention, the server includes means for receiving voice input from the elderly person, means for converting the received voice into character data, means for transmitting the converted character data to a central control unit, means for analyzing the character data in the central control unit and generating an appropriate response (including an emotion engine), means for transmitting the generated response in character form from the central control unit to a terminal, and means for converting the received response into voice and notifying the elderly person. This reduces the elderly person's sense of loneliness, supports dementia prevention, and reduces the burden of health management.

[2275] "Means for receiving voice input" refers to devices or technology that detects voices made by the senior and processes the voice signals.

[2276] "Means for converting into character data" refers to technology that analyzes received voice signals, recognizes phonemes and words, and converts them into character strings.

[2277] "Means for transmitting to a server" refers to a device or technology that transmits the converted character data to a central control device (server) via a communications network.

[2278] "Central Controller" means a central control unit or system that analyzes received text data and generates an appropriate response.

[2279] "Means for generating an appropriate response" refers to technology that uses specific algorithms or emotion engines based on received data to generate an appropriate response to the user.

[2280] An "emotion engine" refers to an algorithm or system that analyzes emotions from received voice or data and generates a response based on those emotions.

[2281] "Means for sending to the terminal" refers to a device or technique for sending a response in the form of character data generated by the central control unit to the user's terminal.

[2282] "Means for converting the received response into voice" refers to a device or technology that synthesizes text data sent to the terminal into voice and notifies the user.

[2283] "Means for collecting health data" refers to devices and technologies for measuring health information such as a user's weight, body temperature, and blood pressure.

[2284] "Means for analyzing and evaluating health data" refers to algorithms or systems for analyzing collected health data and evaluating the user's health status.

[2285] The term "means for generating health advice" refers to a technology for generating health advice for a user based on the evaluation results of the health data.

[2286] "Means for managing daily schedules" refers to devices and technologies for managing a user's daily plans and tasks.

[2287] The term "means for receiving voice instructions" refers to a device or technology that detects voice instructions regarding the schedule given by the user and processes the voice signals.

[2288] "Means for obtaining schedule information" refers to devices and techniques for querying and obtaining the user's daily schedule information from the central control device.

[2289] The term "means for analyzing schedule information and generating a response" refers to a technique for analyzing acquired schedule information and generating an appropriate schedule-related response to the user.

[2290] This invention is a system for supporting the lives of the elderly, combining voice recognition, natural language processing, health data analysis, smart device integration, and an emotion engine. The system's main functions include serving as a conversation partner, health management, and daily schedule management for everyday use by the elderly.

[2291] The components of this system are:

[2292] 1. Conversation partner / dementia prevention

[2293] When a user asks a question or speaks to a smart speaker, the voice input is received by the device. The device converts the voice data into text data (using, for example, the Google Speech-to-Text API) and sends the converted data to a server. The server analyzes the text data using a natural language processing engine (for example, GPT-3) and generates an appropriate response. It also incorporates an emotion engine (for example, IBM Watson Tone Analyzer) that recognizes the user's emotions and adjusts the tone and content of the response. The generated response is sent from the server to the device in text format, where it is converted into speech and notified to the user (for example, using Amazon Polly).

[2294] Examples:

[2295] User: "What's the weather like today?"

[2296] Device: Receives audio, converts it to text using the Google Speech-to-Text API, and sends it to the server.

[2297] Server: Analyzes text data using GPT-3 and obtains weather information. The emotion engine recognizes the user's emotions and generates a response such as, "It's sunny today. It might be nice to take a walk to change your mood."

[2298] Terminal: The response from the server is converted into speech using Amazon Polly and notified to the user.

[2299] 2. Health management

[2300] Users collect their health data using a smart device. For example, they measure their body temperature and blood pressure with a smartwatch and send the data to the device. The device then sends the received data to a server, which analyzes it using a health data analysis algorithm (e.g., Google Fit API). An emotion engine is also incorporated into this evaluation, which generates health advice taking the user's emotional state into account. The advice is then sent from the server to the device, which converts it into audio and notifies the user.

[2301] Examples:

[2302] User: Weighs himself and records the data on a smart scale.

[2303] Terminal: Receives data from the scale and sends it to the server.

[2304] Server: Analyzes the data using the Google Fit API and generates health advice such as, "You seem a little low today. I recommend some light exercise."

[2305] Device: The advice is converted into speech using Amazon Polly and notified to the user.

[2306] 3. Daily life support

[2307] When a user speaks to a smart speaker to check their daily schedule, the device receives the voice, converts it into text data, and sends it to the server. The server then checks the user's daily schedule information and uses an emotion engine to generate a response that takes into account the user's current emotional state. The generated response is then sent from the server to the device, which converts it into voice and notifies the user.

[2308] Examples:

[2309] User: "What's on my schedule for today?"

[2310] Device: Receives audio, converts it to text using the Google Speech-to-Text API, and sends it to the server.

[2311] Server: Checks the schedule and generates a response saying, "I have a doctor's appointment at 10:00. I'm going to the supermarket at 3:00 PM." The emotion engine determines that the user is stressed and provides additional advice, saying, "Try not to push yourself too hard, and take things slowly."

[2312] Device: Responses and advice are converted into speech using Amazon Polly and communicated to the user.

[2313] Example prompts for generative AI models

[2314] "Please explain the function of an elderly life support system that uses voice recognition and an emotion engine to respond to user questions. Please also include specific processing procedures and examples when a user asks a question to a smart speaker."

[2315] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2316] 1. Conversation partner / dementia prevention

[2317] Step 1:

[2318] A user speaks to a smart speaker, such as "What's the weather like today?"

[2319] (Input) User's voice data

[2320] (Output) User's voice data

[2321] Step 2:

[2322] The device receives the voice data and converts it into text using voice recognition software (e.g., Google Speech-to-Text API).

[2323] (Input) User's voice data

[2324] (Output) The user's voice is converted into text data.

[2325] Step 3:

[2326] The terminal sends the converted text data to the server using a secure protocol such as HTTPS.

[2327] (Input) Character data

[2328] (Output) Character data sent to the server

[2329] Step 4:

[2330] The server uses a natural language processing engine (e.g., GPT-3) to analyze the text data, and at the same time, an emotion engine (e.g., IBM Watson Tone Analyzer) recognizes the user's emotions, which allows it to generate more appropriate responses.

[2331] (Input) Character data

[2332] (Data processing) Analyze emotions with an emotion engine and analyze data with a natural language processing engine

[2333] (Output) Analysis results and response data based on them

[2334] Step 5:

[2335] The server generates a response and sends it to the terminal in character data format.

[2336] (Input) Generated response data

[2337] (Output) Response data sent to the terminal

[2338] Step 6:

[2339] The response data received by the device is converted into speech using speech synthesis software (e.g., Amazon Polly) and notified to the user.

[2340] (Input) Response data

[2341] (Output) Response data converted to audio

[2342] 2. Health management

[2343] Step 1:

[2344] A user records health data using a health device (e.g., smart watch, smart scale, etc.).

[2345] (Input) User's health data (weight, body temperature, blood pressure, etc.)

[2346] (Output) Health data recorded on the health device

[2347] Step 2:

[2348] The terminal receives data from the health device using communication methods such as Bluetooth and Wi-Fi.

[2349] (Input) Data from health devices

[2350] (Output) Health data received by the device

[2351] Step 3:

[2352] The health data received by the device is sent to the server.

[2353] (Input) Health data

[2354] (Output) Health data sent to the server

[2355] Step 4:

[2356] The server uses a healthcare analysis algorithm (e.g., Google Fit API) to analyze the received health data, while also taking into account the user's emotional state using an emotion engine.

[2357] (Input) Health data

[2358] (Data processing) Health data analysis and emotional state assessment

[2359] (Output) Analysis results and health advice data

[2360] Step 5:

[2361] The server sends the generated health advice to the terminal.

[2362] (Input) Health advice data

[2363] (Output) Health advice data sent to the device

[2364] Step 6:

[2365] The health advice data received by the terminal is converted into voice using voice synthesis software and notified to the user.

[2366] (Input) Health advice data

[2367] (Output) Health advice converted into audio

[2368] 3. Daily life support

[2369] Step 1:

[2370] A user asks a smart speaker about their daily schedule, such as "Tell me what's on my schedule for today."

[2371] (Input) User's voice data

[2372] (Output) User's voice data

[2373] Step 2:

[2374] The device receives the audio and converts it into text data (e.g., Google Speech-to-Text API).

[2375] (Input) User's voice data

[2376] (Output) The user's voice is converted into text data.

[2377] Step 3:

[2378] The terminal sends character data to the server.

[2379] (Input) Character data

[2380] (Output) Character data sent to the server

[2381] Step 4:

[2382] The server analyzes the text data and queries the daily schedule database, while the emotion engine considers the user's current emotional state.

[2383] (Input) Character data, daily schedule database

[2384] (Data processing) Check schedule and evaluate emotional state

[2385] (Output) Schedule confirmation results and appropriate response data

[2386] Step 5:

[2387] The server sends the generated response data to the terminal.

[2388] (Input) Response data

[2389] (Output) Response data sent to the terminal

[2390] Step 6:

[2391] The response data received by the terminal is converted into voice using voice synthesis software and notified to the user.

[2392] (Input) Response data

[2393] (Output) The voiced response

[2394] (Application example 2)

[2395] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2396] In modern society, the safety and health management of elderly people has become an important issue. Elderly people living alone, in particular, need to respond quickly to emergencies and emergencies at home, and also need to alleviate feelings of loneliness and receive emotional care. However, current systems are unable to comprehensively meet these needs. Furthermore, technology for providing personalized support based on emotional state is immature. An advanced, integrated system to solve these issues is needed.

[2397] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice input from the elderly person, means for converting the received voice into text data, means for transmitting the converted text data to the server, means for analyzing the text data in the server and generating an appropriate response, means for transmitting the generated response in text format from the server to the terminal, means for converting the received response into voice and notifying the user, means for detecting abnormalities in the home and reporting if an abnormality is detected, means for making an emergency call based on the elderly person's voice instruction, and means for recognizing the user's emotional state and generating a response accordingly. This ensures the safety of the elderly person at home and enables rapid response in emergencies. Furthermore, personalized responses based on the elderly person's emotional state can alleviate feelings of loneliness and provide emotional care.

[2398] "Elderly" refers to individuals who are aging, especially those in an age group where health care and safety are of paramount importance.

[2399] "Audio input" refers to an audio signal collected using a microphone or other audio collection device.

[2400] "Text data" refers to data in the form of a string of characters generated from voice input or other digital data.

[2401] A "server" refers to a computer system that provides data processing and data storage over a network.

[2402] "Analysis" refers to the process of examining, interpreting, and evaluating data.

[2403] "Response" refers to information generated in response to a user input or request.

[2404] "Transmission" refers to the act of moving data from one point to another.

[2405] "Domestic anomalies" refers to abnormal or unexpected events or changes in the normal home environment.

[2406] "Reporting" refers to the act of communicating information about an anomaly or event to others.

[2407] "Emergency notification" refers to the prompt notification of pre-established contacts or organizations in an urgent situation.

[2408] "Emotional state" refers to an individual's psychological or emotional state at a particular point in time.

[2409] "Personalized responses" refer to responses that are customized based on the characteristics and circumstances of individual users.

[2410] This invention relates to a multifunctional system for supporting the lives of the elderly, specifically, a system that combines voice recognition, natural language processing, health data analysis, smart device integration, and an emotion engine to provide a variety of support functions.

[2411] 1. System Overview

[2412] The system consists of the following major components:

[2413] Voice input and text conversion methods

[2414] Text data analysis and response generation tools

[2415] Health data collection and analysis tools

[2416] In-home sensor monitor

[2417] Emotion Engine

[2418] Emergency notification means

[2419] 2. Program and Data Processing

[2420] The program of this system is built on Python and uses the following libraries and services.

[2421] Speech Recognition: Use the speech_recognition library to convert voice input into text data.

[2422] Speech synthesis: Uses the pyttsx3 library to convert text data into speech.

[2423] Emotion Recognition: Use your own emotion recognition model (e.g., a BERT-based model).

[2424] Emergency Calling: Call emergency contacts using APIs like Twilio.

[2425] Home sensor monitor: Collects data from various sensors installed in the home (e.g., door opening / closing sensors, motion detection sensors) and detects abnormalities.

[2426] 3. System processing overview

[2427] Speech Recognition and Emotion Analysis

[2428] The server receives voice input from the user and converts it to text using the speech_recognition library. The converted text is then analyzed through an emotion engine to evaluate the user's emotional state. Based on this evaluation, an appropriate response is generated.

[2429] Health data collection and analysis

[2430] A smart device (e.g., a smartwatch) worn by a user collects health data such as heart rate, body temperature, and blood pressure and sends it to a server. The server analyzes the received data and evaluates the user's health status. An emotion engine is also added to this evaluation, and health advice is generated taking the user's emotional state into account.

[2431] Detecting and reporting anomalies in the home

[2432] It monitors data from sensors installed in the home (e.g., door open / close sensors, motion detection sensors), and if an abnormality is detected, it notifies the user by voice. If the abnormality is serious, it automatically makes an emergency call.

[2433] emergency call

[2434] When a user utters keywords such as "help" or "emergency," the system determines this to be an emergency and notifies registered emergency contacts. The notification also includes location information, enabling a quick and accurate response.

[2435] 4. Description of use cases

[2436] Specific examples

[2437] When a user speaks to a smart device, asking, "What's the weather like today?", the server converts the speech into text data and analyzes it. The emotion engine evaluates the user's emotional state (e.g., feeling a little down), and generates a response such as, "It's sunny today. It might be nice to take a walk to change your mood." The generated response is synthesized into voice and notified to the user.

[2438] Furthermore, if the sensor that detects the user's movements detects an abnormality, the server will immediately notify them by voice, saying, "There is an abnormality. Please check." If the situation is serious, an emergency contact will be notified, allowing for a prompt response.

[2439] Prompt Sentence Examples

[2440] Generate a response to the question, "How's the weather today?" Consider an appropriate response if the user is elderly and indicates they are feeling a bit down.

[2441] This allows the system to provide personalized support appropriate to the elderly's emotional state, allowing them to live their daily lives with peace of mind.

[2442] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2443] Step 1:

[2444] The user speaks to the smart device, for example, "What's the weather like today?" This voice input is sent to the device via the microphone.

[2445] Input: User's voice

[2446] Output: Audio data

[2447] Step 2:

[2448] The device's microphone collects voice data and uses the speech_recognition library to convert this voice data into text data, which becomes "What's the weather like today?"

[2449] Input: Audio data

[2450] Output: Text data

[2451] Step 3:

[2452] The device then sends the converted text data to a server via the Internet.

[2453] Input: Text data

[2454] Output: Text data sent to the server

[2455] Step 4:

[2456] The server analyzes the received text data, using natural language processing (NLP) techniques to understand the user's intent and recognizes the user's emotional state through an emotion engine. For example, the server can determine from the user's tone of voice that they are "a little depressed."

[2457] Input: Text data

[2458] Output: Analysis results of text data and user's emotional state

[2459] Step 5:

[2460] The server generates an appropriate response based on the information it has obtained. It retrieves weather information from the weather API and generates a response that adds words of encouragement to a user who is feeling down, based on the results of the emotion engine. For example: "It's sunny today. Maybe it would be good to take a walk to change your mood."

[2461] Input: Text data analysis results, weather information, emotional state

[2462] Output: Response text data

[2463] Step 6:

[2464] The server then sends the generated response text data to the terminal. This communication also takes place over the Internet.

[2465] Input: Response text data

[2466] Output: Response text data sent from the server to the terminal

[2467] Step 7:

[2468] The device converts the received response text data into speech using a speech synthesis engine (pyttsx3 library). For example, it may say, "It's a sunny day today. It might be nice to take a walk to change your mood."

[2469] Input: Response text data

[2470] Output: Response audio data

[2471] Step 8:

[2472] Sensors in the home (door sensors, motion detection sensors, etc.) periodically send data to a server, which monitors for abnormalities and reports them via audio if any are detected.

[2473] Input: Sensor data

[2474] Output: Notification of whether anomalies exist

[2475] Step 9:

[2476] In an emergency, if the user utters "help" or "emergency," the device recognizes this and the server quickly notifies emergency contacts. The notification is made using Twilio's API, and location information is also included.

[2477] Input: Emergency voice command

[2478] Output: Emergency call notification

[2479] Step 10:

[2480] As an emotional care feature, a personalized response is generated according to the user's emotional state, and in cooperation with the emotion engine, advice is given to help the user relax if they are feeling stressed.

[2481] Input: Emotional state data

[2482] Output: Personalized response text and voice data

[2483] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2484] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2485] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2486] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2487] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2488] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2489] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2490] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2491] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2492] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2493] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2494] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2495] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2496] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data...

Claims

1. means for receiving a voice input from the senior citizen; means for converting the received voice into text data; means for transmitting the converted text data to a server; means at the server for parsing the text data and generating an appropriate response; means for transmitting the generated response in text format from the server to the terminal; means for converting the received response into speech and notifying the user; A system including:

2. means for collecting health data of a user; means for transmitting the collected health data to a server; a means for analyzing the health data on the server and assessing the user's health status; means for generating health advice based on the assessment results; means for transmitting the generated advice from the server to the terminal; means for converting the received advice into voice and notifying the user; The system of claim 1 , comprising:

3. A means for managing a user's daily schedule; means for receiving a user's voice instruction; means for sending voice instructions to a server and obtaining schedule information; means for analyzing the acquired schedule information and generating a response to the user; means for transmitting the generated response from the server to the terminal; means for converting the received response into speech and notifying the user; The system of claim 1 , comprising:

Images