System

Abstract

A system is provided.SOLUTION: The system includes a means for providing topics based on weather, news, and hobbies and tastes to the elderly person, a means for receiving a message from the relative and notifying the relative of the message by voice, a means for communicating with an ambulance or the relative by a simple voice command, a means for sending a notice to the relative when there is no response from the elderly person for a certain period of time, and a means for analyzing the conversation of the elderly person and issuing a warning when there is a possibility of fraud.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When elderly people live alone or in a facility, daily conversations decrease, which can lead to feelings of loneliness and a decline in cognitive ability. Furthermore, relatives who live far away often worry about the elderly's daily life and how to respond in emergencies. Furthermore, there is a risk that elderly people may become involved in fraud or other troubles. The present invention aims to solve these problems by providing a system that allows elderly people to enjoy rich conversations in their daily lives while their relatives can safely watch over them. [Means for solving the problem]

[0005] The present invention provides a system that supports elderly conversations and allows relatives to remotely check on the elderly's status. This system includes a means for providing the elderly with topics based on the weather, news, and hobbies and preferences, a means for receiving messages from relatives and notifying the elderly by voice, a means for contacting an ambulance or relatives using simple voice commands, a means for notifying relatives if the elderly does not respond for a certain period of time, and a means for analyzing the elderly's conversation and issuing a warning if there is a possibility of fraud. This makes it possible to realize a system that allows elderly people to enjoy everyday conversations while allowing relatives to watch over them with peace of mind. Furthermore, by providing a means for collecting elderly voice samples to improve voice recognition accuracy and a means for ordering daily necessities via voice operation to support online shopping, the system can further support the elderly's daily lives.

[0006] "Weather, news, and topics based on hobbies and preferences" refers to information related to the daily lives of the elderly, such as the day's weather information, the latest news, and topics that match the interests and concerns of the elderly.

[0007] "Messages from relatives" refers to messages or instructions that can be sent remotely by relatives who live far away from the elderly, and are notified to the elderly via voice.

[0008] "Voice commands" are voice instructions given by the elderly person to the system, such as simple commands like "call an ambulance" or "call XX."

[0009] "Means for notifying relatives if there is no response from the elderly person for a certain period of time" refers to a means for automatically notifying relatives of the situation if the system detects that there has been no conversation with the elderly person for a certain period of time (for example, three days).

[0010] "Means of analyzing elderly people's conversations and issuing warnings in the event of a possible fraud" refers to a means of analyzing the content of elderly people's conversations using technology such as AI, and issuing warnings to the elderly themselves or their relatives in the event of a possible fraud.

[0011] A "voice sample" is voice data provided by the elderly person during initial setup and is used to improve the accuracy of voice recognition.

[0012] "Online Shopping Support" provides a function that allows seniors to easily order daily necessities and other products using voice commands. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention is a system that supports conversations between elderly people and enables their relatives living far away to check on their status. This system has functions to provide topics based on weather, news, and hobbies and preferences, to receive messages from relatives and notify them by voice, to recognize voice commands in emergencies and respond immediately, to notify relatives if there is no response from the elderly person for a certain period of time, and to prevent fraud by using voice analysis.

[0035] System Configuration

[0036] The system consists of the following main components:

[0037] 1. Terminal: A device used by the elderly that has voice recognition and notification functions. It provides the elderly with information about the weather, news, and topics based on their hobbies and preferences.

[0038] 2. Server: Manages and processes various data. Responsible for analyzing voice data and sending notifications in emergencies.

[0039] 3. Relatives' interface: A web interface for relatives to send messages remotely and check on the status of the elderly.

[0040] Program processing overview

[0041] 1. User registration and initial settings

[0042] User: Enters name, address, and contact details of relatives into the app or device, and provides a voice sample to the system during initial setup.

[0043] Terminal: Sends entered information and voice samples to the server.

[0044] Server: Stores this information in a database and updates the speech recognition model.

[0045] 2. Providing everyday conversation

[0046] Server: Generates the day's topics (weather, news, hobbies, etc.) based on the user's profile information and past conversation history.

[0047] Terminal: Provides the elderly with the topics generated at the specified time via audio.

[0048] Users: Enjoy conversations and talk to their devices.

[0049] Terminal: Recognizes the user's speech and sends it to the server.

[0050] Server: Generates an appropriate response and sends it back to the device.

[0051] Specific examples

[0052] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[0053] User: Yes, please.

[0054] Terminal: According to this article, roses grow best in the spring.

[0055] 3. Message function from relatives

[0056] Relatives: Enter and send messages via the web interface.

[0057] Server: Receives the message and sends it to the elderly person's device at the specified time.

[0058] Device: Receives voice notification of messages from relatives at the specified time.

[0059] User: Responds verbally if necessary.

[0060] Terminal: The response is sent to the server and the relatives are notified.

[0061] Specific examples

[0062] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[0063] Terminal: Have you taken your medicine yet?

[0064] User: Yes, I did.

[0065] Device: Your reply has been notified to your family.

[0066] The server sends a notification to the relative: Father replied that he had taken his medicine.

[0067] 4. Emergency Response

[0068] User: Says a voice command such as "Call an ambulance" or "Call ____."

[0069] Device: Recognizes voice commands and immediately sends them to the server.

[0070] Server: Receives emergency voice commands and automatically notifies necessary contacts (119 or relatives).

[0071] Specific examples

[0072] User: Call an ambulance

[0073] Terminal: Okay, I'll call 911.

[0074] An automatic call will be made to 119 via the server.

[0075] Server: An emergency has occurred. We will notify you to respond.

[0076] 5. Monitoring conversations and notifying relatives

[0077] Terminal: If there is no conversation for a certain period of time (e.g., 3 days), the server will be automatically notified.

[0078] Server: Receives the notification and notifies relatives.

[0079] Specific examples

[0080] (No conversation for 3 days)

[0081] Terminal: We haven't spoken in a while, I'll contact my family.

[0082] Server sends notification to relatives: Father hasn't spoken in 3 days, please check.

[0083] 6. Online shopping support

[0084] User: Use voice commands to order the items they need.

[0085] Terminal: Recognizes voice and sends orders to the server.

[0086] Server: Completes the purchase procedure and notifies the terminal that the order is complete.

[0087] Terminal: Notifies the user that the order is complete.

[0088] Specific examples

[0089] User: Order some tissues.

[0090] Terminal: Would you like to order one tissue?

[0091] User: Yes

[0092] Device: I ordered it and it will arrive in 2 days.

[0093] 7. Fraud Prevention Features

[0094] Terminal: Constantly monitors the user's conversation and sends the voice data to the server.

[0095] Server: Uses AI models to analyze conversations and detect potential fraud.

[0096] Server: If it determines that there is a high possibility of fraud, it sends a warning to the device.

[0097] Terminal: Provides audio fraud warnings to users.

[0098] Server: Notify relatives of possible fraud.

[0099] Specific examples

[0100] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[0101] Device: This is a potential scam message. Do you want to continue?

[0102] Server: Notify relatives: A potentially fraudulent conversation has been detected.

[0103] With the above configuration and processing, the present invention provides a system that allows elderly people to enjoy everyday conversations while their relatives can watch over them with peace of mind.

[0104] The processing flow will be explained below.

[0105] User registration and initial settings

[0106] Step 1:

[0107] User: Launches app or device, enters name, address, and contact details of next of kin. They are prompted to provide a voice sample.

[0108] Step 2:

[0109] Terminal: Sends entered information and voice samples to the server.

[0110] Step 3:

[0111] Server: Stores the received information in a database and updates the speech recognition model.

[0112] Providing everyday conversation

[0113] Step 1:

[0114] Server: Generates topics for the day based on the user's profile information and past conversation history. Sends the generated topics to the device.

[0115] Step 2:

[0116] Terminal: Provides the elderly with audio information about topics generated at the specified time.

[0117] Step 3:

[0118] User: Talk to the device about a topic.

[0119] Step 4:

[0120] Terminal: Recognizes the user's speech and sends it to the server.

[0121] Step 5:

[0122] Server: Generates an appropriate response and sends it back to the device.

[0123] Step 6:

[0124] Terminal: The response from the server is conveyed to the user by voice.

[0125] Message function from relatives

[0126] Step 1:

[0127] Relatives: Enter and send messages via the web interface.

[0128] Step 2:

[0129] Server: Receives messages and sends them to the terminal at the specified time.

[0130] Step 3:

[0131] Terminal: Notifies the elderly of messages from relatives via voice at the specified time.

[0132] Step 4:

[0133] User: Responds verbally if necessary.

[0134] Step 5:

[0135] Terminal: Sends the response to the server.

[0136] Step 6:

[0137] Server: Notifies the relatives of the response.

[0138] Emergency response

[0139] Step 1:

[0140] User: Issues an emergency voice command such as "Call an ambulance" or "Call ____."

[0141] Step 2:

[0142] Terminal: Recognizes voice commands and immediately notifies the server.

[0143] Step 3:

[0144] Server: Receives emergency voice commands and automatically notifies necessary contacts (119 or relatives).

[0145] Conversation monitoring and notification to relatives

[0146] Step 1:

[0147] Device: If there is no conversation for a certain period of time (e.g., 3 days), the server will be automatically notified.

[0148] Step 2:

[0149] Server: Receives the notification and notifies relatives.

[0150] Online shopping support

[0151] Step 1:

[0152] User: Use voice commands to order the items they need.

[0153] Step 2:

[0154] Terminal: Recognizes the voice and sends it to the server.

[0155] Step 3:

[0156] Server: Receives the order details and processes the purchase on the e-commerce site.

[0157] Step 4:

[0158] Server: After the purchase procedure is completed, send the completion information to the terminal.

[0159] Step 5:

[0160] Terminal: Notifies the user by voice that the order has been completed.

[0161] Fraud prevention features

[0162] Step 1:

[0163] Terminal: Constantly monitors the user's conversation and sends the voice data to the server.

[0164] Step 2:

[0165] Server: Uses AI models to analyze conversations and detect potential fraud.

[0166] Step 3:

[0167] Server: If it determines that there is a high possibility of fraud, it sends a warning to the device.

[0168] Step 4:

[0169] Terminal: Provides audio fraud warnings to users.

[0170] Step 5:

[0171] Server: Notify relatives of possible fraud.

[0172] Example 1

[0173] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0174] In modern society, elderly people are often isolated, and situations in which they require support in daily life are increasing. It is also difficult for relatives living far away to understand the situation of elderly people, which creates the risk of delayed response in emergencies. Furthermore, elderly people are often exposed to risks such as fraud. There is a need for a system that addresses these issues and provides an environment in which elderly people can live with peace of mind.

[0175] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0176] In the present invention, the server includes the following means, which makes it possible to:

[0177] A means of providing weather, news, and topics based on hobbies and preferences to seniors,

[0178] A means of receiving messages from relatives and notifying the elderly by voice;

[0179] A means to contact emergency contacts and relatives with simple voice commands,

[0180] A means of sending a notice to relatives if there is no response from the elderly person for a certain period of time;

[0181] A method for analyzing elderly people's conversations and issuing warnings in cases of possible fraud;

[0182] means for recognizing and transmitting voice input from the senior citizen to a server, which processes the data and provides an appropriate response to the senior citizen;

[0183] A way to order daily necessities online using voice commands,

[0184] a means for automatically taking appropriate emergency action when a voice command is issued; and

[0185] A method to automatically notify relatives if there is no communication for a certain period of time.

[0186] This will enable elderly people to enjoy everyday conversations and allow relatives to watch over them from a distance with peace of mind. It will also enable quick responses in emergencies and prevent fraud, improving the quality of life for the elderly.

[0187] "Elderly" refers to people over a certain age, who often require special support.

[0188] "Relative" refers to a person who has a close relationship with an older person based on blood or legal ties.

[0189] "Topics based on weather, news, and hobbies and preferences" refers to information that is likely to attract the individual user's interest and attention, and is a topic that enriches daily life.

[0190] "Messages" refer to messages or information sent by relatives to elderly people.

[0191] "Voice commands" refer to instructions that seniors can give using their voice to trigger specific actions.

[0192] "Emergency Contact" refers to contact information that should be used to contact you promptly if necessary.

[0193] "Speech recognition" refers to the technology of analyzing voice as a digital signal and converting it into text data, etc.

[0194] "Voice notification" refers to the function of converting specific information into voice and conveying it to the elderly.

[0195] "Potential fraud" refers to patterns of conversation or behavior intended to deceive seniors.

[0196] "Voice input" refers to voice data provided by a user to a system via a microphone or the like.

[0197] "Data processing" refers to the process by which the server analyzes the data it receives and generates an appropriate response or action.

[0198] "Online shopping" refers to the act of purchasing goods over the Internet.

[0199] "Daily necessities" refers to consumables and necessary products used in daily life.

[0200] "Emergency response" refers to measures taken to respond quickly when an unexpected situation occurs.

[0201] "Notification" refers to a message or signal intended to convey specific information to a target audience.

[0202] The present invention is a system that supports elderly people's conversations and allows their relatives living far away to check on their status. This system is composed of a terminal used by the elderly person, a server that manages and processes various data, and a web interface used by the relatives. Specific embodiments of the present invention will be described below.

[0203] 1. Main system components

[0204] The system consists of the following main components:

[0205] Terminal: A device used by seniors that has voice recognition and notification functions. It provides seniors with information about the weather, news, and topics based on their hobbies and preferences. It also has the ability to recognize voice commands in emergencies and respond immediately.

[0206] Server: Manages and processes various data. Responsible for analyzing voice data, sending emergency notifications, and detecting and warning about fraud. The server uses a voice recognition engine (e.g., Google® Speech-to-Text API) and a natural language processing engine (e.g., OpenAI® GPT-3®).

[0207] Relatives' interface: A web interface that allows relatives to send messages remotely and check on the elderly's status. From here, relatives can input and check messages, and receive notifications in case of emergencies.

[0208] 2. Program Processing Overview

[0209] User registration and initial settings

[0210] First, the user enters their information (name, address, contact information for relatives) on the device or a dedicated app, and provides a voice sample during the initial setup. This information is sent from the device to the server, where it is stored in a database and the voice recognition model is updated, improving voice recognition accuracy.

[0211] Providing everyday conversation

[0212] The server generates the day's topics (weather, news, hobbies, etc.) based on the user's profile information and past conversation history. This information is collected using news APIs and weather forecast APIs. The generated topics are provided to the elderly via voice at the specified time via the device. When the user speaks, the device recognizes the voice and sends the text data to the server. The server uses a natural language processing engine to generate an appropriate response, which is sent to the device and played aloud.

[0213] Specific examples

[0214] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[0215] User: Yes, please.

[0216] Terminal: According to this article, roses grow best in the spring.

[0217] Message function from relatives

[0218] Relatives can input messages through a web interface, which are then received by the server. At the designated time, the message is sent to the elderly person's device via voice notification. When the elderly person responds by voice, the message is sent back to the server and notified to the relative.

[0219] Specific examples

[0220] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[0221] Terminal: Have you taken your medicine yet?

[0222] User: Yes, I did.

[0223] Device: Your reply has been notified to your family.

[0224] The server sends a notification to the relative: Father replied that he had taken his medicine.

[0225] Emergency response

[0226] When an elderly person issues an emergency voice command such as "call an ambulance" or "call XX," the device recognizes the command and immediately sends it to the server, which then receives it and automatically notifies the necessary contacts (119 or relatives).

[0227] Specific examples

[0228] User: Call an ambulance

[0229] Terminal: Okay, I'll call 911.

[0230] The server will automatically call 119.

[0231] Server: An emergency has occurred. We will notify you to respond.

[0232] Conversation monitoring and notification to relatives

[0233] If a device has not had a conversation for a certain period of time (e.g., 3 days), it will automatically notify the server, which will then notify the family.

[0234] Specific examples

[0235] (No conversation for 3 days)

[0236] Terminal: It's been a while since we last spoke, I'll contact my family.

[0237] Server sends notification to relatives: Father hasn't spoken in 3 days, please check.

[0238] Online shopping support

[0239] When an elderly person orders a product by voice, the device recognizes the voice and sends the order to the server, which then processes the purchase and notifies the device that the order is complete.

[0240] Specific examples

[0241] User: Order some tissues.

[0242] Terminal: Would you like to order one tissue?

[0243] User: Yes

[0244] Device: I ordered it and it will arrive in 2 days.

[0245] Fraud prevention features

[0246] The device constantly monitors the user's conversations and sends the audio data to the server. The server analyzes the conversation and sends a warning to the device if there is a possibility of fraud. The device notifies the user by voice and also notifies their relatives of the possibility of fraud.

[0247] Specific examples

[0248] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[0249] Device: This is a potential scam message. Do you want to continue?

[0250] Server: Notify relatives: A potentially fraudulent conversation has been detected.

[0251] With the above configuration and processing, the present invention provides a system that allows elderly people to enjoy everyday conversations while allowing relatives to watch over them from a remote location with peace of mind.

[0252] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0253] Step 1:

[0254] The user enters their name, address, contact details of their next of kin into the app or device and provides a voice sample.

[0255] Input: Name, address, contact details of relatives, voice sample

[0256] What happens: The user fills in a form on the device with details and records audio into the microphone.

[0257] Output: Input data and audio samples are saved to the device.

[0258] Step 2:

[0259] The device sends the entered information and voice samples to the server.

[0260] Input: User-entered information and voice samples

[0261] Specific operation: The device packs data into packets and sends them over the Internet to a server.

[0262] Output: Data received by the server

[0263] Step 3:

[0264] The server stores the received information in a database and updates the speech recognition model.

[0265] Input: User information and voice sample

[0266] What happens: The server inserts data into the database and updates the model using a speech recognition engine (e.g., Google Speech-to-Text API).

[0267] Output: Updated database and speech recognition model

[0268] Step 4:

[0269] The server generates the topic for the day based on the user's profile information and past conversation history.

[0270] Input: User profile information, past conversation history

[0271] Specific operation: The server obtains information using news APIs or weather forecast APIs and generates topics using a generative AI model (e.g., OpenAI GPT-3).

[0272] Output: Generated topics

[0273] Step 5:

[0274] The device provides the elderly with the topics generated at the specified time via voice.

[0275] Input: Generated topics

[0276] Specific operation: The device uses a speech synthesis engine (for example, Google Text-to-Speech API) to convert the topic into audio and play it back to the user.

[0277] Output: Elderly people listen to the topic

[0278] Step 6:

[0279] When the user speaks, the device recognizes the voice and sends the text data to the server.

[0280] Input: User voice input

[0281] Specific operation: The device converts the speech into text using a speech recognition engine and sends it to the server.

[0282] Output: Sends audio to the server as text data

[0283] Step 7:

[0284] The server generates an appropriate response and sends it to the terminal, which plays it aloud.

[0285] Input: Sends audio to the server as text data

[0286] Specific operation: The server generates a response using a natural language processing engine (e.g., OpenAI GPT-3) and sends it to the device. The device then converts the response into speech using a speech synthesis engine and plays it back to the user.

[0287] Output: The user hears the reply aloud

[0288] Step 8:

[0289] Relatives can enter messages through a web interface and send them to the server.

[0290] Input: Relative's text message

[0291] Specific actions: A relative logs into the web interface, types a message, and presses send.

[0292] Output: The server receives the message.

[0293] Step 9:

[0294] The server receives the message and sends it to the elderly person's device at the specified time.

[0295] Input: Message from relative

[0296] Specific operation: The server's schedule management system sends notification instructions to the terminal at the specified time.

[0297] Output: The device receives the message.

[0298] Step 10:

[0299] The device will notify you of messages from relatives via voice at the specified time.

[0300] Input: Message notification from the server

[0301] Specific operation: The device uses a speech synthesis engine to convert the message into voice and notify the user.

[0302] Output: User listens to the message

[0303] Step 11:

[0304] The user can respond by voice if necessary, and the terminal sends the content to the server.

[0305] Input: User's voice response

[0306] Specific operation: The device recognizes the voice and sends it to the server as text data.

[0307] Output: Response data is sent to the server

[0308] Step 12:

[0309] The server notifies the relatives of the received reply.

[0310] Input: User response data

[0311] What happens: The server sends emails and SMS to the relatives' contacts.

[0312] Output: The relative receives the reply

[0313] Step 13:

[0314] When an elderly person issues an emergency voice command (e.g., "Call an ambulance"), the device recognizes it and sends it to the server.

[0315] Input: Emergency voice command

[0316] Specific operation: The device analyzes the emergency command using a voice recognition engine and sends it to the server.

[0317] Output: Server receives urgent command

[0318] Step 14:

[0319] The server receives emergency voice commands and automatically notifies the necessary contacts (e.g., 119 or relatives).

[0320] Input: Emergency voice command

[0321] Specific operation: The server automatically notifies the emergency contact via the phone API or message API.

[0322] Output: Emergency is handled

[0323] Step 15:

[0324] If a device has not had a conversation for a certain period of time (e.g., 3 days), it will automatically notify the server.

[0325] Input: Conversation status data

[0326] Specific operation: The device monitors conversation events using an internal counter and notifies the server if there is no update within a certain period of time.

[0327] Output: The server receives the notification

[0328] Step 16:

[0329] The server receives the notification and notifies the relatives.

[0330] Input: Notification from device

[0331] What happens: The server sends emails and SMS to the relatives' contacts.

[0332] Output: Relatives receive notification

[0333] Step 17:

[0334] When an elderly person orders a product by voice, the terminal recognizes the voice and sends the order to the server.

[0335] Input: Voice command to order product

[0336] Specific operation: The device converts the speech into text using a speech recognition engine and sends it to the server.

[0337] Output: The server receives the order.

[0338] Step 18:

[0339] The server completes the purchase process and notifies the terminal that the order has been completed.

[0340] Input: Product order data

[0341] What happens: The server uses the e-commerce API to complete the purchase and confirm the order.

[0342] Output: The terminal will be notified that the order has been completed.

[0343] Step 19:

[0344] The terminal notifies the user that the order is complete.

[0345] Input: Order completion notification

[0346] What happens: The device uses a speech synthesis engine to announce the completion of the order.

[0347] Output: User hears order completed

[0348] Step 20:

[0349] The device constantly monitors the user's conversation and transmits the voice data to the server.

[0350] Input: Conversational audio data

[0351] Specific operation: The device records audio using an audio capture device and streams it to the server.

[0352] Output: The server receives the audio data.

[0353] Step 21:

[0354] The server uses AI models to analyze the conversation and sends a warning to the device if there is a possibility of fraud.

[0355] Input: Conversational audio data

[0356] What it does: The server analyzes the data using natural language processing models and runs algorithms to detect fraudulent patterns.

[0357] Output: Fraud alert data

[0358] Step 22:

[0359] The server receives the fraud warning data and sends it to the terminal, which then notifies the user of the fraud warning by voice.

[0360] Input: Fraud Alert Data

[0361] What happens: The server sends a fraud warning to the device, which then plays it back to the user using its speech synthesis engine.

[0362] Output: User hears fraud warning

[0363] Step 23:

[0364] The server also notifies relatives of possible fraud.

[0365] Input: Fraud Alert Data

[0366] What happens: The server sends emails and SMS to the relatives' contacts.

[0367] Output: Relative receives fraud alert

[0368] (Application example 1)

[0369] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0370] Systems already exist that support the daily conversations of elderly people and allow relatives to check their status remotely, but these systems lack sufficient dialogue generation capabilities and flexible means for relatives to send messages. Furthermore, their emergency response and fraud prevention features for elderly people are often incomplete. Therefore, there is a need for systems with more advanced conversation support and security features that allow elderly people to live their daily lives with peace of mind.

[0371] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0372] In this invention, the server is a system that supports conversations between elderly people and allows relatives to check their status from a remote location, and includes: means for providing the elderly with topics based on the weather, news, and hobbies and preferences; means for receiving messages from relatives and notifying the elderly by voice; means for contacting an ambulance or relatives using simple voice commands; means for notifying relatives if there is no response from the elderly for a certain period of time; means for analyzing the elderly's conversations and issuing a warning if there is a possibility of fraud; means for generating conversation content using a generative artificial intelligence model and providing daily conversation support; and means for relatives to easily send messages using prompt sentences. This provides more advanced conversation support, faster emergency response, and fraud prevention functions than conventional systems, allowing elderly people to live their lives with peace of mind.

[0373] The "elderly conversation support system" is a system that supports the everyday conversations of the elderly and allows relatives in remote locations to check on their status.

[0374] The "topic provision means" is a function that provides elderly people with topics generated based on weather, news, and hobbies and preferences via voice.

[0375] The "message notification means" is a function that receives messages sent by relatives and notifies the elderly person by voice.

[0376] The "emergency contact method" is a function that allows elderly people to contact an ambulance or relatives in an emergency by issuing simple voice commands.

[0377] The "no response notification method" is a function that automatically sends a notification to relatives if there is no response from the elderly person for a certain period of time.

[0378] The "fraud warning feature" is a function that analyzes the elderly person's conversation and issues an audio warning if there is a possibility of fraud.

[0379] A "generative artificial intelligence model" is an artificial intelligence technology that generates appropriate conversation content based on data such as a user's profile information and past conversation history.

[0380] A "prompt sentence" is an instruction sentence that is input to a generative artificial intelligence model, allowing relatives to easily send messages.

[0381] The present invention is a system that supports conversations between elderly people and allows their remote relatives to check on their status. This system includes a means for providing topics based on weather, news, and hobbies and preferences, a means for receiving messages from relatives and notifying them by voice, a means for recognizing and immediately responding to voice commands in emergencies, a means for notifying relatives if there is no response from the elderly person for a certain period of time, a means for preventing fraud through voice analysis, a means for generating conversation content using a generative artificial intelligence model to support daily conversations, and a means for relatives to easily send messages using prompt sentences.

[0382] System Configuration

[0383] This system consists of the following main components:

[0384] 1. Device:

[0385] This device, used by seniors, has voice recognition and notification functions, and provides them with information about the weather, news, and topics based on their hobbies and preferences.

[0386] 2. Server:

[0387] Manages and processes various data. Analyzes voice data and sends emergency notifications. Generates conversation content using generative artificial intelligence models.

[0388] 3. Kindred Interface:

[0389] A web interface that allows relatives to remotely send messages or check on the status of elderly people. Messages can be sent using prompts.

[0390] Functions and Processing

[0391] User registration and initial settings

[0392] The user (elderly person) enters their name, address, and contact information of their relatives into the app or device, and provides a voice sample to the system during initial setup. The device then sends the entered information and voice sample to the server, which stores this information in a database and updates the voice recognition model.

[0393] Providing everyday conversation

[0394] The server generates the topic of the day (weather, news, hobbies, etc.) based on the user's profile information and past conversation history. The device provides the generated topic to the elderly via voice at the specified time. When the user enjoys a conversation and speaks to the device, the device recognizes the user's speech and sends it to the server. The server generates an appropriate response and sends it back to the device.

[0395] Examples:

[0396] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[0397] User: Yes, please.

[0398] Terminal: According to this article, roses grow best in the spring.

[0399] Message function from relatives

[0400] The relative inputs and sends a message via a web interface. The server receives the message and sends it to the elderly person's device at the specified time. The device then notifies the relative of the message by voice at the specified time, and the user can respond by voice as needed. The device then sends the response to the server, which notifies the relative.

[0401] Examples:

[0402] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[0403] Terminal: Have you taken your medicine yet?

[0404] User: Yes, I did.

[0405] Device: Your reply has been notified to your family.

[0406] The server sends a notification to the relative: Father replied that he had taken his medicine.

[0407] Emergency response

[0408] The user issues a voice command such as "Call an ambulance" or "Call XX." The device recognizes the voice command and immediately sends it to the server. The server receives the emergency voice command and automatically notifies the necessary contacts (119 or relatives).

[0409] Examples:

[0410] User: Call an ambulance

[0411] Terminal: Okay, I'll call 911.

[0412] An automatic call will be made to 119 via the server.

[0413] Server: An emergency has occurred. We will notify you to respond.

[0414] Notification function when there is no response for a certain period of time

[0415] If there is no conversation for a certain period of time (e.g., 3 days), the device automatically notifies the server. The server receives the notification and notifies the relatives.

[0416] Examples:

[0417] If there is no conversation for three days

[0418] Terminal: We haven't spoken in a while, I'll contact my family.

[0419] Server sends notification to relatives: Father hasn't spoken in 3 days, please check.

[0420] Fraud prevention features

[0421] The device constantly monitors the user's conversation and sends the audio data to the server. The server uses a generative artificial intelligence model to analyze the conversation and detect possible fraud. If it determines that there is a high possibility of fraud, it sends a warning to the device and notifies the user of the fraud warning via voice. At the same time, the server notifies relatives of the possibility of fraud.

[0422] Examples:

[0423] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[0424] Device: This is a potential scam message. Do you want to continue?

[0425] Server: Notify relatives: A potentially fraudulent conversation has been detected.

[0426] With the above configuration and processing, the present invention provides a system that allows elderly people to enjoy everyday conversations while their relatives can watch over them with peace of mind.

[0427] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0428] Step 1:

[0429] User registration and initial settings

[0430] The user (elderly person) uses an input device (terminal) to enter basic information such as their name, address, and contact details of their relatives. In addition, they provide several voice samples during the initial setup. The terminal then sends this information and the voice samples to the server, which stores the received information in a database and updates the voice recognition model.

[0431] Input: Name, address, contact details of relatives, voice sample

[0432] Output: Updated speech recognition model, user information stored in the database

[0433] Step 2:

[0434] Providing everyday conversation

[0435] The server uses a generative artificial intelligence model to generate the day's topics (weather, news, hobbies, etc.) based on the user's profile information and past conversation history. The generated topics are sent from the server to the device, which then provides them to the elderly via voice at the specified time.

[0436] Input: User profile information, past conversation history

[0437] Output: Generated topics, information provided in audio

[0438] Specific operation: The server generates a topic called "It's forecast to be sunny," and the device announces "It's forecast to be sunny today" by voice.

[0439] Step 3:

[0440] Message function from relatives

[0441] Relatives use a web interface to input and send messages. The server receives the messages and sends them to the elderly person's device at the specified time. The device then notifies the elderly person of the message by voice at the specified time. The elderly person can then respond by voice if necessary, and the response is sent from the device to the server, where it is then notified to the relative.

[0442] Input: Message from relative, user's voice response

[0443] Output: Voice notification to the elderly, notification of the response to relatives

[0444] Specific operation: A relative sends a message saying, "Please make sure you have taken your medicine today," and the device notifies the user by voice, "Have you taken your medicine yet?", to which the user replies, "Yes, I have."

[0445] Step 4:

[0446] Emergency response

[0447] When a user issues a voice command such as "call an ambulance" or "call XX," the device recognizes the command and immediately sends it to the server. The server receives the emergency voice command and automatically notifies the necessary contacts (such as 119 or relatives).

[0448] Input: Senior emergency voice command

[0449] Output: Automatic call to emergency contacts

[0450] Specific operation: The user says "call an ambulance," the device recognizes the voice and sends it to the server, which then automatically calls 119.

[0451] Step 5:

[0452] Notification function when there is no response for a certain period of time

[0453] If the device does not receive a response from the elderly person for a certain period of time (for example, three days), it automatically sends a notification to the server, which then receives the notification and sends it to the elderly person's relatives.

[0454] Input: No response for a certain period of time

[0455] Output: No response notification from the server to relatives

[0456] Specific operation: If there is no response from the elderly person for three days, the device will notify the family, "We haven't heard from you for a long time. We will contact your family," and the server will notify the relative, "Your father has not spoken for three days. Please check."

[0457] Step 6:

[0458] Fraud prevention features

[0459] The device constantly monitors the user's conversation and sends the audio data to the server. The server uses a generative artificial intelligence model to analyze the conversation and detect possible fraud. If it determines that there is a high possibility of fraud, it sends a warning to the device and notifies the user via audio. At the same time, the server notifies relatives of the possibility of fraud.

[0460] Input: Elderly people's conversation

[0461] Output: Audio notification of fraud warning, fraud notification to relatives

[0462] Specific operation: If an unknown caller says, "You have won 1 million yen," the server detects the possibility of fraud, the device warns the user with a voice message saying, "This is a potentially fraudulent message," and the server notifies relatives that, "A potentially fraudulent conversation has been detected."

[0463] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0464] This invention combines a system that supports conversations between elderly people and allows their relatives in remote locations to check their status with an emotion engine that recognizes the user's emotions. This system provides topics based on weather, news, and hobbies and preferences, receives messages from relatives and notifies them by voice, recognizes voice commands in emergencies and responds immediately, notifies relatives if there is no response from the elderly person for a certain period of time, and prevents fraud through voice analysis. It is also possible to use the emotion engine to analyze the elderly person's emotions and provide appropriate responses or notifications.

[0465] System Configuration

[0466] The system consists of the following main components:

[0467] 1. Terminal: A device used by the elderly that has voice recognition, emotion recognition, and notification functions. It provides the elderly with information about the weather, news, and topics based on their hobbies and preferences via voice.

[0468] 2. Server: Manages and processes various data, analyzes voice and emotion data, and sends notifications in the event of an emergency.

[0469] 3. Relatives' interface: A web interface for relatives to send messages remotely and check on the status of the elderly.

[0470] 4. Emotion engine: It has the ability to analyze the user's voice and identify their emotional state.

[0471] Program processing overview

[0472] 1. User registration and initial settings

[0473] User: Enters name, address, and contact details of relatives into the app or device, and provides a voice sample during initial setup.

[0474] Terminal: Sends entered information and voice samples to the server.

[0475] Server: This information is stored in a database and the speech recognition model and emotion recognition model are updated.

[0476] 2. Providing everyday conversation

[0477] Server: Generates topics for the day based on the user's profile information and past conversation history. Sends the generated topics to the device.

[0478] Terminal: Provides the elderly with audio information about topics generated at the specified time.

[0479] Emotion engine: Analyzes the elderly person's responses and identifies their emotional state (e.g., joy, sadness, anger, etc.).

[0480] User: Talk to the device about a topic.

[0481] Terminal: Recognizes the user's speech and sends it to the server.

[0482] Server: Generates an appropriate response and sends it back to the device. Based on the analysis results of the emotion engine, an appropriate response is given according to the emotion.

[0483] Specific examples

[0484] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[0485] User: Yes, please. (Shows delight)

[0486] Emotion engine: Analyzes user responses as emotions of joy.

[0487] Server: Generates a response based on the emotion of joy.

[0488] Terminal: According to this article, roses grow best in the spring. That's so lovely!

[0489] 3. Message function from relatives

[0490] Relatives: Enter and send messages via the web interface.

[0491] Server: Receives the message and sends it to the elderly person's device at the specified time.

[0492] Device: Receives voice notification of messages from relatives at the specified time.

[0493] User: Responds verbally if necessary.

[0494] Emotion engine: Analyzes the user's responses and identifies their emotional state.

[0495] Terminal: Sends the response content and emotional state to the server.

[0496] Server: Notifies relatives of response and emotional state.

[0497] Specific examples

[0498] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[0499] Terminal: Have you taken your medicine yet?

[0500] User: Yes, I did. (Shows relief)

[0501] Emotion engine: Analyzes user responses as emotions of relief.

[0502] Server: Notify relatives of response along with emotional state of relief.

[0503] Family Interface: Father has taken his medicine. He seems relieved.

[0504] 4. Emergency Response

[0505] User: Issues an emergency voice command such as "Call an ambulance" or "Call ____."

[0506] Terminal: Recognizes voice commands and immediately notifies the server.

[0507] Emotion engine: Analyzes emotions (e.g., surprise, fear, etc.) during emergencies.

[0508] Server: Receives emergency voice commands and emotional state, and automatically notifies necessary contacts (119 or relatives).

[0509] Specific examples

[0510] User: Call an ambulance. (Shows fear)

[0511] Terminal: Okay, I'll call 911.

[0512] Emotion engine: Analyzes user responses as emotions of fear.

[0513] Server: Automatically calls 119 and notifies relatives along with the emotional state of fear.

[0514] Kinship Interface: An emergency has occurred. They appear frightened.

[0515] 5. Monitoring conversations and notifying relatives

[0516] Terminal: If there is no conversation for a certain period of time (e.g., 3 days), the server will be automatically notified.

[0517] Server: Receives the notification and notifies relatives.

[0518] Specific examples

[0519] (No conversation for 3 days)

[0520] Terminal: We haven't spoken in a while, I'll contact my family.

[0521] Server: Send notification to relatives.

[0522] Kinship Interface: Father hasn't spoken in 3 days. Please check.

[0523] 6. Online shopping support

[0524] User: Use voice commands to order the items they need.

[0525] Terminal: Recognizes voice and sends orders to the server.

[0526] Server: Receives the order details and processes the purchase on the e-commerce site.

[0527] Server: After the purchase procedure is completed, send the completion information to the terminal.

[0528] Terminal: A voice message is sent to the user to notify them that the order has been completed.

[0529] Specific examples

[0530] User: Order some tissues.

[0531] Terminal: Would you like to order one tissue?

[0532] User: Yes

[0533] Device: I ordered it and it will arrive in 2 days.

[0534] 7. Fraud Prevention Features

[0535] Terminal: Constantly monitors the user's conversation and sends the voice data to the server.

[0536] Server: Uses AI models to analyze conversations and detect potential fraud.

[0537] Emotion Engine: Analyzes the user's emotional state and immediately warns if there is a high possibility of fraud.

[0538] Server: If it determines that there is a high possibility of fraud, it sends a warning along with the emotional state to the device.

[0539] Device: Provides audio fraud warnings along with emotional state.

[0540] Server: Notifies relatives of possible fraud and emotional state.

[0541] Specific examples

[0542] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[0543] Device: This is a potential scam message. Do you want to continue?

[0544] Emotion Engine: Analyzes the user's confused emotional state.

[0545] Server: High likelihood of fraud, along with a confused emotional state, sends out a warning.

[0546] Kinship Interface: A potentially fraudulent conversation has been detected. Your father appears confused.

[0547] The above configuration and processing not only allows elderly people to enjoy everyday conversations and allows relatives to watch over them with peace of mind, but also makes it possible to provide a system that uses an emotion engine to respond appropriately to the emotional state of the elderly.

[0548] The processing flow will be explained below.

[0549] User registration and initial settings

[0550] Step 1:

[0551] User: Launches the app or device and enters initial information such as name, address, and contact details of next of kin. They are then prompted to provide a voice sample.

[0552] Step 2:

[0553] Terminal: Sends entered information and voice samples to the server.

[0554] Step 3:

[0555] Server: Stores the received information in a database and updates the speech recognition model and emotion recognition model.

[0556] Providing everyday conversation

[0557] Step 1:

[0558] Server: Generates topics for the day (weather, news, hobbies, etc.) based on the user's profile information and past conversation history. Sends the generated topics to the device.

[0559] Step 2:

[0560] Terminal: Provides the elderly with audio information about topics generated at the specified time.

[0561] Step 3:

[0562] Emotion engine: Analyzes the elderly person's responses and identifies their emotional state.

[0563] Step 4:

[0564] User: Talk to the device about a topic.

[0565] Step 5:

[0566] Terminal: Recognizes the user's speech and sends it to the server.

[0567] Step 6:

[0568] Server: Generates an appropriate response and sends a response to the device according to the emotion based on the analysis results of the emotion engine.

[0569] Step 7:

[0570] Terminal: Provides the user with a voice response from the server.

[0571] Specific examples

[0572] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[0573] User: Yes, please. (Shows delight)

[0574] Emotion engine: Analyzes user responses as emotions of joy.

[0575] Server: Generates a response based on the emotion of joy.

[0576] Terminal: According to this article, roses grow best in the spring. That's so lovely!

[0577] Message function from relatives

[0578] Step 1:

[0579] Relatives: Enter and send messages via the web interface.

[0580] Step 2:

[0581] Server: Receives the message and sends it to the elderly person's device at the specified time.

[0582] Step 3:

[0583] Terminal: Notifies the elderly of messages from relatives via voice at the specified time.

[0584] Step 4:

[0585] User: Responds verbally if necessary.

[0586] Step 5:

[0587] Emotion engine: Analyzes the user's responses and identifies their emotional state.

[0588] Step 6:

[0589] Terminal: Sends the response content and emotional state to the server.

[0590] Step 7:

[0591] Server: Notifies relatives of response and emotional state.

[0592] Specific examples

[0593] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[0594] Terminal: Have you taken your medicine yet?

[0595] User: Yes, I did. (Shows relief)

[0596] Emotion engine: Analyzes user responses as emotions of relief.

[0597] Server: Notify relatives of response along with emotional state of relief.

[0598] Family Interface: Father has taken his medicine. He seems relieved.

[0599] Emergency response

[0600] Step 1:

[0601] User: Issues an emergency voice command such as "Call an ambulance" or "Call ____."

[0602] Step 2:

[0603] Terminal: Recognizes voice commands and immediately notifies the server.

[0604] Step 3:

[0605] Emotion engine: Analyzes emotions (e.g., surprise, fear, etc.) during emergencies.

[0606] Step 4:

[0607] Server: Receives emergency voice commands and emotional state, and automatically notifies necessary contacts (119 or relatives).

[0608] Step 5:

[0609] Server: Sends notification completion information along with the emotional state to the terminal.

[0610] Step 6:

[0611] Terminal: Notifies the elderly person by voice when the report is complete.

[0612] Specific examples

[0613] User: Call an ambulance. (Shows fear)

[0614] Terminal: Okay, I'll call 911.

[0615] Emotion engine: Analyzes user responses as emotions of fear.

[0616] Server: Automatically calls 119 and notifies relatives along with the emotional state of fear.

[0617] Kinship Interface: An emergency has occurred. They appear frightened.

[0618] Conversation monitoring and notification to relatives

[0619] Step 1:

[0620] Terminal: If there is no conversation for a certain period of time (e.g., 3 days), the server will be automatically notified.

[0621] Step 2:

[0622] Server: Receives the notification and notifies relatives.

[0623] Specific examples

[0624] (No conversation for 3 days)

[0625] Terminal: We haven't spoken in a while, I'll contact my family.

[0626] Server: Send notification to relatives.

[0627] Kinship Interface: Father hasn't spoken in 3 days. Please check.

[0628] Online shopping support

[0629] Step 1:

[0630] User: Use voice commands to order the items they need.

[0631] Step 2:

[0632] Terminal: Recognizes the voice and sends it to the server.

[0633] Step 3:

[0634] Server: Receives the order details and processes the purchase on the e-commerce site.

[0635] Step 4:

[0636] Server: After the purchase procedure is completed, send the completion information to the terminal.

[0637] Step 5:

[0638] Terminal: A voice message is sent to the user to notify them that the order has been completed.

[0639] Specific examples

[0640] User: Order some tissues.

[0641] Terminal: Would you like to order one tissue?

[0642] User: Yes

[0643] Device: I ordered it and it will arrive in 2 days.

[0644] Fraud prevention features

[0645] Step 1:

[0646] Terminal: Constantly monitors the user's conversation and sends the voice data to the server.

[0647] Step 2:

[0648] Emotion Engine: Analyzes the user's emotional state.

[0649] Step 3:

[0650] Server: Uses AI models to analyze conversations and detect potential fraud, taking into account the results of the sentiment engine.

[0651] Step 4:

[0652] Server: If it determines that there is a high possibility of fraud, it sends a warning along with the emotional state to the device.

[0653] Step 5:

[0654] Device: Provides audio fraud warnings along with emotional state.

[0655] Step 6:

[0656] Server: Notifies relatives of possible fraud and emotional state.

[0657] Specific examples

[0658] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[0659] Device: This is a potential scam message. Do you want to continue?

[0660] Emotion Engine: Analyzes the user's confused emotional state.

[0661] Server: High likelihood of fraud, along with a confused emotional state, sends out a warning.

[0662] Kinship Interface: A potentially fraudulent conversation has been detected. Your father appears confused.

[0663] The above configuration and processing not only allows elderly people to enjoy everyday conversations and allows relatives to watch over them with peace of mind, but also makes it possible to provide a system that uses an emotion engine to respond appropriately to the emotional state of the elderly.

[0664] Example 2

[0665] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0666] Elderly people often feel lonely and anxious in their daily lives. In particular, when relatives live far away, it is difficult for them to keep track of the elderly's condition, increasing safety and health risks. There are also situations where a quick response is required in an emergency, and there are risks such as fraud. There is a need for a system that can solve these issues and support the lives of the elderly while allowing relatives to keep an eye on them with peace of mind.

[0667] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for supporting the elderly person's conversation and allowing relatives to check the elderly person's status from a remote location; means for providing topics based on the weather, news, and hobbies and preferences; means for receiving messages from relatives and notifying them by voice; means for contacting emergency contact information and relatives with simple voice commands; means for sending a notification to relatives if there is no response for a certain period of time; means for analyzing the conversation and issuing a warning if there is a possibility of fraud; means for analyzing the elderly person's voice and identifying their emotional state; and means for providing appropriate responses and notifications based on the emotional state. This supports the elderly person's daily life and enables relatives to know the elderly person's status from a remote location in real time.

[0668] "Elderly" refers to an individual who is elderly and requires assistance with daily living.

[0669] "Relative" refers to a person who is related to an older person by blood or marriage, or who has the responsibility to care for an older person.

[0670] A "conversation support system" is a system that supports the elderly in their everyday conversations and allows relatives to check on their status from a remote location.

[0671] "Topic provision means" refers to a means of providing topics to elderly people via audio based on the weather, news, hobbies, and preferences.

[0672] "Message notification means" refers to a means of notifying elderly people by voice of messages received from relatives.

[0673] "Emergency contact methods" refer to methods that allow seniors to contact emergency contacts and relatives using simple voice commands.

[0674] "Response monitoring measures" refer to measures that send notifications to relatives if there is no response from the elderly person for a certain period of time.

[0675] "Fraud warning measures" refer to measures that analyze elderly people's conversations and issue warnings if there is a possibility of fraud.

[0676] "Voice analysis means" refers to a means for analyzing the voice of an elderly person and identifying their emotional state.

[0677] "Emotional response measures" refer to measures that provide appropriate responses or notifications based on the emotional state of the elderly person.

[0678] "Online shopping support means" refers to a means for ordering daily necessities using voice commands.

[0679] This invention relates to a system that supports conversations between elderly people and allows their remote relatives to check their status in real time. This system provides topics based on weather, news, and hobbies and preferences, and includes functions for notifying messages from relatives via voice, recognizing and responding to voice commands in emergencies, notifying relatives if there is no response from the elderly person for a certain period of time, and preventing fraud through voice analysis. It also uses an emotion engine to analyze the emotions of the elderly person and provide appropriate responses and notifications.

[0680] The system consists of the following main components:

[0681] 1. Terminal: A device used by the elderly that has voice recognition, emotion recognition, and notification functions. It provides the elderly with information about the weather, news, and topics based on their hobbies and preferences via voice.

[0682] 2. Server: Manages and processes various data, analyzes voice and emotion data, and sends notifications in the event of an emergency.

[0683] 3. Relatives' interface: A web interface for relatives to send messages and check on the status of the elderly from remote locations.

[0684] 4. Emotion engine: It has the ability to analyze the user's voice and identify their emotional state.

[0685] Hardware and Software Use Cases

[0686] 1. Device: The device can be a tablet device or smart speaker (e.g., Amazon Echo, GOOGLE HOME®) with internet connectivity, allowing the elderly person to easily start a conversation and respond when presented with a topic.

[0687] 2. Server: For the server, we use high-performance cloud services (e.g., Amazon Web Services, Google Cloud Platform), which enable large-volume data processing and real-time responses.

[0688] 3. Emotion Engine: Using Google Cloud Speech-to-Text API and IBM Watson® Tone Analyzer, the system analyzes the elderly person's voice and identifies their emotional state.

[0689] Detailed System Description

[0690] User registration and initial settings

[0691] Users enter their name, address, and contact information for relatives into the app or their device, and provide a voice sample, which allows the system to register the user's voice and basic information in a database to improve the accuracy of voice and emotion recognition.

[0692] Providing everyday conversation

[0693] The server uses a generative AI model to generate appropriate topics based on the user's profile information and past conversation history, and sends them to the device. The device then provides the topics to the elderly via voice at the specified time. For example, it provides information in the form of, "Good morning. It's forecast to be sunny today. We have a new article about gardening. Would you like to take a look?"

[0694] Message function from relatives

[0695] When a relative sends a message via a web interface, the server receives it and sends it to the device at the specified time. For example, a message such as "Please make sure you have taken your medicine today" can be sent, and the device will notify the elderly person by voice.

[0696] Emergency response

[0697] When a user issues a simple voice command such as "call an ambulance" or "call XX," the device recognizes it and immediately notifies the server, which then automatically notifies the necessary contacts (e.g., 119 or relatives), enabling a prompt response.

[0698] Conversation monitoring and notification to relatives

[0699] If the device has not spoken for a certain period of time (e.g., three days), it notifies the server, and the server sends a notification to the relatives informing them of the situation. For example, a message such as "Your father has not spoken for three days. Please check." will be sent to the relatives.

[0700] Online shopping support

[0701] When a user orders everyday items by voice, the device recognizes the order and sends it to the server, which then processes the purchase through the e-commerce site and notifies the user that the order is complete.

[0702] Fraud prevention features

[0703] The device constantly monitors the user's conversation and sends the audio data to the server, which then uses an AI model to analyze the conversation. If there is a high possibility of fraud, the emotion engine analyzes the user's confusion or anxiety and immediately issues a fraud warning.

[0704] The system will support the elderly in their daily lives, allowing relatives to remotely monitor their condition in real time, and will also use an emotion engine to enable more appropriate responses.

[0705] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0706] Program processing flow

[0707] User registration and initial settings

[0708] Step 1:

[0709] The user enters their name, address, and contact details of their next of kin into the app or device and provides a voice sample.

[0710] Input: User's personal information (name, address, contact details of relatives) and voice sample

[0711] Output: The input data is saved on the device and ready to be sent to the next step.

[0712] Specific behavior:

[0713] Users follow the instructions on the app or device's initial setup screen, enter the required information, and provide a voice sample, which is recorded through the device's microphone.

[0714] Step 2:

[0715] The device sends the entered information and voice samples to the server.

[0716] Input: User personal information and voice sample (output of step 1)

[0717] Output: Sending confirmation and server receiving status

[0718] Specific behavior:

[0719] The terminal encrypts the input data and sends it securely to the server, after which the server confirms receipt of the data and returns a receipt confirmation message to the terminal.

[0720] Step 3:

[0721] The server stores this information in a database and updates the speech recognition model and emotion recognition model.

[0722] Input: User personal information and voice sample (output of step 2)

[0723] Output: Database update status and model update completion

[0724] Specific behavior:

[0725] The server stores the received user information in a database and uses the voice samples to train speech and emotion recognition models.

[0726] Providing everyday conversation

[0727] Step 1:

[0728] The server generates the topic of the day based on the user's profile information and past conversation history.

[0729] Input: User profile information and past conversation history

[0730] Output: Generated topic text data

[0731] Specific behavior:

[0732] The server uses a generative AI model to generate appropriate topics based on the user's hobbies, preferences, and past conversation history, such as weather, news, and hobby articles.

[0733] Step 2:

[0734] The device provides the elderly with audio topics generated at the specified time.

[0735] Input: Generated topic text data (output of Step 1)

[0736] Output: Audio notification completion status

[0737] Specific behavior:

[0738] The device will provide a topic by voice at a set time (e.g., every morning at 9:00). A speech synthesis engine will convert the text data into voice and notify the elderly.

[0739] Step 3:

[0740] The user speaks to the terminal about a topic.

[0741] Input: User's voice response

[0742] Output: Recorded audio data

[0743] Specific behavior:

[0744] The user speaks their thoughts or questions about the topics provided by the device, and the device's microphone records the voice.

[0745] Step 4:

[0746] The terminal recognizes what the user says and sends it to the server.

[0747] Input: User's voice data (output of Step 3)

[0748] Output: Text converted data and transmission completion status

[0749] Specific behavior:

[0750] The terminal uses a speech recognition engine to convert the user's voice into text data and transmits it to the server.

[0751] Step 5:

[0752] The server generates an appropriate response and sends it back to the terminal.

[0753] Input: Text data (output of step 4)

[0754] Output: Generated response text data

[0755] Specific behavior:

[0756] The server uses a generative AI model to generate appropriate responses to the user's text data and sends them to the device. An emotion engine also works in parallel to analyze the user's emotional state.

[0757] Step 6:

[0758] The terminal notifies the user of the response from the server by voice.

[0759] Input: Generated response text data (output of Step 5)

[0760] Output: Audio notification completion status

[0761] Specific behavior:

[0762] The terminal uses a speech synthesis engine to convert the response text data into speech and notify the user.

[0763] Message function from relatives

[0764] Step 1:

[0765] Relatives can enter and send messages via a web interface.

[0766] Input: Relative's text message

[0767] Output: Transmission complete status

[0768] Specific behavior:

[0769] The relative logs into the web interface, enters a message, and presses the send button, which sends the message to the server.

[0770] Step 2:

[0771] The server receives the message and sends it to the elderly person's device at the specified time.

[0772] Input: Relative's text message (output of step 1)

[0773] Output: Confirmation of sending and notification to the terminal that it is ready

[0774] Specific behavior:

[0775] The server schedules and manages the received messages, sending them to the elderly person's device at the specified time.

[0776] Step 3:

[0777] The device will notify you of messages from relatives via voice at the specified time.

[0778] Input: Message text data

[0779] Output: Audio notification completion status

[0780] Specific behavior:

[0781] At the set time, the device uses a voice synthesis engine to convert the relative's message into voice and notify the elderly person.

[0782] Step 4:

[0783] The user responds verbally as needed.

[0784] Input: User's voice reply

[0785] Output: Recorded audio data

[0786] Specific behavior:

[0787] The user responds to the message by voice and speaks it into the terminal.

[0788] Step 5:

[0789] An emotion engine analyzes the user's responses to identify their emotional state.

[0790] Input: User's voice data (output of step 4)

[0791] Output: Emotional state data

[0792] Specific behavior:

[0793] The emotion engine analyzes voice data to identify emotional states such as relief or joy.

[0794] Step 6:

[0795] The device sends the response and emotional state to the server.

[0796] Input: Text-converted response data and emotional state data

[0797] Output: Transmission complete status

[0798] Specific behavior:

[0799] The terminal converts the user's response into text data and transmits it to the server together with the emotional state data.

[0800] Step 7:

[0801] The server notifies the relative of the response and emotional state.

[0802] Input: Response text data and emotional state data (output of Step 6)

[0803] Output: Notification to relatives completed

[0804] Specific behavior:

[0805] Based on the received data, the server sends a message to the relatives, including information about the user's emotional state. A notification such as "Your father has taken his medicine. He seems relieved" is sent.

[0806] Emergency response

[0807] Step 1:

[0808] The user issues an emergency voice command such as "call an ambulance" or "call XX."

[0809] Input: User's voice command

[0810] Output: Recorded audio data

[0811] Specific behavior:

[0812] In an emergency, the user speaks loudly into the device, such as "call an ambulance," and the device's microphone records the voice.

[0813] Step 2:

[0814] The device recognizes the voice command and immediately notifies the server.

[0815] Input: User's voice command data (output of step 1)

[0816] Output: Transmission complete status

[0817] Specific behavior:

[0818] When the device recognizes an emergency voice command, it immediately notifies the server.

[0819] Step 3:

[0820] The emotion engine analyzes emotions (e.g., surprise, fear, etc.) during emergencies.

[0821] Input: User's voice data (output of step 2)

[0822] Output: Emotional state data

[0823] Specific behavior:

[0824] The emotion engine analyzes the voice data to identify the user's emotional state in emergencies, such as surprise or fear.

[0825] Step 4:

[0826] The server receives emergency voice commands and emotional state and automatically notifies the necessary contacts (119 or relatives).

[0827] Input: Emergency voice command data and emotional state data (output of Step 3)

[0828] Output: Status of contact notification completion

[0829] Specific behavior:

[0830] The server automatically calls 119 or relatives based on emergency voice commands and the user's emotional state, such as "call an ambulance."

[0831] Conversation monitoring and notification to relatives

[0832] Step 1:

[0833] If a device has not had a conversation for a certain period of time (e.g., 3 days), it will automatically notify the server.

[0834] Input: Conversation absence period

[0835] Output: Notification data

[0836] Specific behavior:

[0837] The device will automatically send a notification to the server if there is no conversation for three days.

[0838] Step 2:

[0839] The server receives the notification and notifies the relatives.

[0840] Input: Notification data (output of step 1)

[0841] Output: Notification to relatives completed

[0842] Specific behavior:

[0843] The server notifies the relatives that there has been no communication for three days. The relatives receive a message saying, "Your father has not spoken for three days. Please check."

[0844] Online shopping support

[0845] Step 1:

[0846] The user orders the desired products by voice.

[0847] Input: User's voice commands

[0848] Output: Recorded audio data

[0849] Specific behavior:

[0850] The user speaks a voice instruction to the terminal, such as "Please order some tissues."

[0851] Step 2:

[0852] The terminal recognizes the voice and sends the order to the server.

[0853] Input: User's voice data (output of step 1)

[0854] Output: Text converted data and transmission completion status

[0855] Specific behavior:

[0856] The terminal uses a voice recognition engine to convert the voice data into text data and transmits it to the server.

[0857] Step 3:

[0858] The server receives the order details and processes the purchase on the e-commerce site.

[0859] Input: Text-converted order data (output of Step 2)

[0860] Output: Purchase procedure completed

[0861] Specific behavior:

[0862] The server uses the specified e-commerce site API to complete the purchase process based on the order details.

[0863] Step 4:

[0864] After the server completes the purchase procedure, it sends completion information to the terminal.

[0865] Input: Purchase completion data (output of Step 3)

[0866] Output: Transmission complete status

[0867] Specific behavior:

[0868] The server sends purchase completion information to the terminal and prepares for notification to the user.

[0869] Step 5:

[0870] The terminal will notify the user by voice that the order has been completed.

[0871] Input: Purchase completion data (output from Step 4)

[0872] Output: Audio notification completion status

[0873] Specific behavior:

[0874] The device will use a speech synthesis engine to notify the user by voice that the purchase has been completed, with a message such as "Your order has been placed. It will arrive in two days."

[0875] Fraud prevention features

[0876] Step 1:

[0877] The device constantly monitors the user's conversation and transmits the voice data to the server.

[0878] Input: Conversational voice data

[0879] Output: Transmission complete status

[0880] Specific behavior:

[0881] The device monitors the user's conversation and transmits the voice data to the server in real time.

[0882] Step 2:

[0883] The server uses AI models to analyze the conversation and detect potential fraud.

[0884] Input: Transmitted audio data (output of step 1)

[0885] Output: Fraud probability data

[0886] Specific behavior:

[0887] The server uses an AI model to analyze the received voice data and detect trigger words and phrases that may indicate fraud.

[0888] Step 3:

[0889] The emotion engine analyzes the user's emotional state and immediately warns if there is a high possibility of fraud.

[0890] Input: Transmitted voice data (output of Step 1) and analysis results of the AI ​​model (output of Step 2)

[0891] Output: Emotional state and alert data

[0892] Specific behavior:

[0893] The emotion engine analyzes the user's emotional state and issues a warning if confusion, anxiety, etc. are detected.

[0894] Step 4:

[0895] If the server determines that there is a high possibility of fraud, it sends a warning to the device along with the emotional state.

[0896] Input: Emotional state and alert data (output of step 3)

[0897] Output: Transmission complete status

[0898] Specific behavior:

[0899] If the server determines that there is a high possibility of fraud, it sends a warning message along with the user's emotional state to the terminal.

[0900] Step 5:

[0901] The device will provide audible warnings of fraud along with emotional state.

[0902] Input: Emotional state and alert data (output of step 4)

[0903] output: Completion status of the audio notification

[0904] Specific behavior:

[0905] The device uses a voice synthesis engine to notify users of their emotional state and fraud warnings, such as "This message may be fraudulent. Do you want to continue?"

[0906] Step 6:

[0907] The server notifies the relatives of the possible fraud and their emotional state.

[0908] Input: Emotional state and alert data (output of step 4)

[0909] Output: Notification to relatives completed

[0910] Specific behavior:

[0911] The server notifies the relatives of the likelihood of fraud and their emotional state. A notification such as "A conversation with a high likelihood of fraud has been detected. Your father seems confused" is sent to the relatives.

[0912] (Application example 2)

[0913] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0914] There is a need for a system that allows elderly people to communicate with their relatives in remote locations with peace of mind and that can respond quickly in emergencies. Another important issue is how to improve the user experience by supporting elderly people in easily shopping in virtual stores using voice commands and by suggesting appropriate products based on their emotions.

[0915] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for identifying the emotional state of the elderly person based on emotion recognition and providing appropriate responses or notifications as necessary, means for performing voice-based product searches in the virtual store and suggesting products according to the emotions, and means for displaying advice from relatives. This enables the elderly person to enjoy shopping in the virtual store with peace of mind while receiving support according to their emotions.

[0916] The "system" is a complex set of devices and software that assists elderly people in conversation and allows relatives to check on their condition remotely.

[0917] "Means" refers to a method, apparatus, or program for performing a specified function or task.

[0918] "Emotion recognition" is a technology that analyzes and identifies a person's emotional state from their voice or text data.

[0919] "Emotional state" refers to a person's current psychological state, such as happiness, sadness, anger, or relief.

[0920] "Appropriate response" refers to responding or acting in the most appropriate way according to one's emotional state.

[0921] "Notification" is the act of sending information to the user or relatives to inform them.

[0922] A "virtual store" is a virtual store that offers products and services on an online platform.

[0923] "Product search" refers to the action or process of finding a specific product.

[0924] "Product suggestion" is the act of recommending appropriate products based on the user's needs and emotions.

[0925] "Advice from relatives" refers to advice or instructions sent by relatives, and is information that is displayed for the elderly person to refer to.

[0926] "Speech recognition" is a technology that analyzes speech and converts it into text or instructions.

[0927] A "voice command" is a command to give specific instructions by voice.

[0928] An "emergency" refers to a time-sensitive situation in which an older adult requires immediate attention.

[0929] "Fraud" is an illegal act of deceiving people through dishonest means to obtain money, goods, or information.

[0930] This invention is a system that supports elderly people's conversations and allows relatives to check on their condition remotely, and incorporates emotion recognition technology. The main components of this system are a terminal equipped with voice recognition, emotion recognition, and notification functions, a server that manages and processes various data, an interface that allows relatives to send messages and check on the elderly's condition remotely, and an emotion engine.

[0931] System configuration

[0932] 1. Device:

[0933] This device is used by the elderly and has voice recognition, emotion recognition, and notification functions. The device provides the elderly with information on the weather, news, and topics based on their hobbies and preferences. It also understands the elderly's voice commands and communicates with the server as needed.

[0934] 2. Server:

[0935] The server is responsible for managing and processing various data, analyzing voice and emotion data, providing emergency notifications, and suggesting products. It also collects voice samples from elderly people to improve the accuracy of voice recognition.

[0936] 3. Kinship Interface:

[0937] This is a web interface that allows relatives to send messages from remote locations and check on the status of the elderly. Relatives can input advice for the elderly, and the advice is sent to the elderly's device.

[0938] 4. Emotion Engine:

[0939] The emotion engine analyzes the voices of elderly people and identifies their emotional state, enabling it to provide appropriate responses and suggest products based on their emotions.

[0940] Program processing

[0941] The server analyzes the data using a speech recognition engine (Python's speech_recognition library), an emotion recognition engine (a customized EmotionRecognition class), and a notification system (AlertSystem class). When it receives voice input, it converts the speech into text and analyzes it. Based on the analysis results, it suggests topics and products appropriate for the elderly.

[0942] In the event of an emergency, the system recognizes the user's voice commands and immediately notifies the server, which then automatically issues an emergency notification based on the analysis results of the emotion engine and takes the necessary action.

[0943] Example

[0944] For example, consider the case of an elderly person ordering daily necessities from a virtual store.

[0945] Scenario: An elderly person wants to order tissues from a virtual store.

[0946] Example user:

[0947] "Looking for tissue paper"

[0948] "The weather is nice today, so I want to go for a walk."

[0949] Based on this, the system processes a program that follows the user's instructions, suggests appropriate products, and notifies relatives of the situation.

[0950] Example prompt sentence:

[0951] Scenario: Senior citizen orders an item from a virtual store.

[0952] User: "Find me some tissues."

[0953] The system recognizes your voice and presents a list of related products.

[0954] Emotion Engine: Recognizes the emotions of the elderly and displays the emotion of joy.

[0955] Server: Make appropriate product suggestions.

[0956] This system not only allows relatives to keep a watchful eye on the elderly while they enjoy everyday conversation, but also uses an emotion engine to respond appropriately to the elderly's emotional state.

[0957] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0958] Step 1:

[0959] The user inputs voice commands through the device, which then receives the elderly person's voice instructions and collects voice data. For example, the user might say, "Find me some tissues."

[0960] Step 2:

[0961] The device sends the collected voice data to a speech recognition engine (speech_recognition library) and converts it into text data. The converted text becomes "Look for tissue paper."

[0962] Step 3:

[0963] The device sends the converted text data to an emotion recognition engine (EmotionRecognition class) for emotion analysis. The emotion engine identifies the elderly person's emotions (e.g., joy, excitement) from the text data.

[0964] Step 4:

[0965] The device transmits text data and emotion data to a server, which receives the data and generates a list of candidate products to suggest appropriate products according to the user's emotion.

[0966] Step 5:

[0967] The server then sends the generated product candidate list back to the emotion engine, which selects the product that best matches the emotion. For example, a user who expresses joy might be offered a product with beautiful packaging.

[0968] Step 6:

[0969] The server sends information about the selected product to the terminal. The terminal notifies the user of this information by voice and suggests the product, such as "We suggest this product as tissue paper. Please check it out."

[0970] Step 7:

[0971] The user checks the products and selects or orders by voice. The terminal sends this voice data back to the voice recognition engine and converts it into text data.

[0972] Step 8:

[0973] The terminal sends the converted text data to the server for order processing, and the server communicates with the e-commerce site to process the order.

[0974] Step 9:

[0975] Once the order process is complete, the server sends the completion information to the terminal. The terminal then notifies the user by voice that the order has been completed. For example, it may say, "Your tissue order has been completed. It will arrive in two days."

[0976] Step 10:

[0977] Through the relative interface, the server notifies the relative of the order and the elderly person's emotional state. The relative can then check the elderly person's status through a web interface and send additional advice or instructions if necessary.

[0978] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0979] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0980] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0981] [Second embodiment]

[0982] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0983] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0984] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0985] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0986] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0987] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0988] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0989] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0990] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0991] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0992] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0993] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0994] This invention is a system that supports conversations between elderly people and enables their relatives living far away to check on their status. This system has functions to provide topics based on weather, news, and hobbies and preferences, to receive messages from relatives and notify them by voice, to recognize voice commands in emergencies and respond immediately, to notify relatives if there is no response from the elderly person for a certain period of time, and to prevent fraud by using voice analysis.

[0995] System Configuration

[0996] The system consists of the following main components:

[0997] 1. Terminal: A device used by the elderly that has voice recognition and notification functions. It provides the elderly with information about the weather, news, and topics based on their hobbies and preferences.

[0998] 2. Server: Manages and processes various data. Responsible for analyzing voice data and sending notifications in emergencies.

[0999] 3. Relatives' interface: A web interface for relatives to send messages remotely and check on the status of the elderly.

[1000] Program processing overview

[1001] 1. User registration and initial settings

[1002] User: Enters name, address, and contact details of relatives into the app or device, and provides a voice sample to the system during initial setup.

[1003] Terminal: Sends entered information and voice samples to the server.

[1004] Server: Stores this information in a database and updates the speech recognition model.

[1005] 2. Providing everyday conversation

[1006] Server: Generates the day's topics (weather, news, hobbies, etc.) based on the user's profile information and past conversation history.

[1007] Terminal: Provides the elderly with the topics generated at the specified time via audio.

[1008] Users: Enjoy conversations and talk to their devices.

[1009] Terminal: Recognizes the user's speech and sends it to the server.

[1010] Server: Generates an appropriate response and sends it back to the device.

[1011] Specific examples

[1012] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[1013] User: Yes, please.

[1014] Terminal: According to this article, roses grow best in the spring.

[1015] 3. Message function from relatives

[1016] Relatives: Enter and send messages via the web interface.

[1017] Server: Receives the message and sends it to the elderly person's device at the specified time.

[1018] Device: Receives voice notification of messages from relatives at the specified time.

[1019] User: Responds verbally if necessary.

[1020] Terminal: The response is sent to the server and the relatives are notified.

[1021] Specific examples

[1022] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[1023] Terminal: Have you taken your medicine yet?

[1024] User: Yes, I did.

[1025] Device: Your reply has been notified to your family.

[1026] The server sends a notification to the relative: Father replied that he had taken his medicine.

[1027] 4. Emergency Response

[1028] User: Says a voice command such as "Call an ambulance" or "Call ____."

[1029] Device: Recognizes voice commands and immediately sends them to the server.

[1030] Server: Receives emergency voice commands and automatically notifies necessary contacts (119 or relatives).

[1031] Specific examples

[1032] User: Call an ambulance

[1033] Terminal: Okay, I'll call 911.

[1034] An automatic call will be made to 119 via the server.

[1035] Server: An emergency has occurred. We will notify you to respond.

[1036] 5. Monitoring conversations and notifying relatives

[1037] Terminal: If there is no conversation for a certain period of time (e.g., 3 days), the server will be automatically notified.

[1038] Server: Receives the notification and notifies relatives.

[1039] Specific examples

[1040] (No conversation for 3 days)

[1041] Terminal: We haven't spoken in a while, I'll contact my family.

[1042] Server sends notification to relatives: Father hasn't spoken in 3 days, please check.

[1043] 6. Online shopping support

[1044] User: Use voice commands to order the items they need.

[1045] Terminal: Recognizes voice and sends orders to the server.

[1046] Server: Completes the purchase procedure and notifies the terminal that the order is complete.

[1047] Terminal: Notifies the user that the order is complete.

[1048] Specific examples

[1049] User: Order some tissues.

[1050] Terminal: Would you like to order one tissue?

[1051] User: Yes

[1052] Device: I ordered it and it will arrive in 2 days.

[1053] 7. Fraud Prevention Features

[1054] Terminal: Constantly monitors the user's conversation and sends the voice data to the server.

[1055] Server: Uses AI models to analyze conversations and detect potential fraud.

[1056] Server: If it determines that there is a high possibility of fraud, it sends a warning to the device.

[1057] Terminal: Provides audio fraud warnings to users.

[1058] Server: Notify relatives of possible fraud.

[1059] Specific examples

[1060] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[1061] Device: This is a potential scam message. Do you want to continue?

[1062] Server: Notify relatives: A potentially fraudulent conversation has been detected.

[1063] With the above configuration and processing, the present invention provides a system that allows elderly people to enjoy everyday conversations while their relatives can watch over them with peace of mind.

[1064] The processing flow will be explained below.

[1065] User registration and initial settings

[1066] Step 1:

[1067] User: Launches app or device, enters name, address, and contact details of next of kin. They are prompted to provide a voice sample.

[1068] Step 2:

[1069] Terminal: Sends entered information and voice samples to the server.

[1070] Step 3:

[1071] Server: Stores the received information in a database and updates the speech recognition model.

[1072] Providing everyday conversation

[1073] Step 1:

[1074] Server: Generates topics for the day based on the user's profile information and past conversation history. Sends the generated topics to the device.

[1075] Step 2:

[1076] Terminal: Provides the elderly with audio information about topics generated at the specified time.

[1077] Step 3:

[1078] User: Talk to the device about a topic.

[1079] Step 4:

[1080] Terminal: Recognizes the user's speech and sends it to the server.

[1081] Step 5:

[1082] Server: Generates an appropriate response and sends it back to the device.

[1083] Step 6:

[1084] Terminal: The response from the server is conveyed to the user by voice.

[1085] Message function from relatives

[1086] Step 1:

[1087] Relatives: Enter and send messages via the web interface.

[1088] Step 2:

[1089] Server: Receives messages and sends them to the terminal at the specified time.

[1090] Step 3:

[1091] Terminal: Notifies the elderly of messages from relatives via voice at the specified time.

[1092] Step 4:

[1093] User: Responds verbally if necessary.

[1094] Step 5:

[1095] Terminal: Sends the response to the server.

[1096] Step 6:

[1097] Server: Notifies the relatives of the response.

[1098] Emergency response

[1099] Step 1:

[1100] User: Issues an emergency voice command such as "Call an ambulance" or "Call ____."

[1101] Step 2:

[1102] Terminal: Recognizes voice commands and immediately notifies the server.

[1103] Step 3:

[1104] Server: Receives emergency voice commands and automatically notifies necessary contacts (119 or relatives).

[1105] Conversation monitoring and notification to relatives

[1106] Step 1:

[1107] Device: If there is no conversation for a certain period of time (e.g., 3 days), the server will be automatically notified.

[1108] Step 2:

[1109] Server: Receives the notification and notifies relatives.

[1110] Online shopping support

[1111] Step 1:

[1112] User: Use voice commands to order the items they need.

[1113] Step 2:

[1114] Terminal: Recognizes the voice and sends it to the server.

[1115] Step 3:

[1116] Server: Receives the order details and processes the purchase on the e-commerce site.

[1117] Step 4:

[1118] Server: After the purchase procedure is completed, send the completion information to the terminal.

[1119] Step 5:

[1120] Terminal: Notifies the user by voice that the order has been completed.

[1121] Fraud prevention features

[1122] Step 1:

[1123] Terminal: Constantly monitors the user's conversation and sends the voice data to the server.

[1124] Step 2:

[1125] Server: Uses AI models to analyze conversations and detect potential fraud.

[1126] Step 3:

[1127] Server: If it determines that there is a high possibility of fraud, it sends a warning to the device.

[1128] Step 4:

[1129] Terminal: Provides audio fraud warnings to users.

[1130] Step 5:

[1131] Server: Notify relatives of possible fraud.

[1132] Example 1

[1133] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1134] In modern society, elderly people are often isolated, and situations in which they require support in daily life are increasing. It is also difficult for relatives living far away to understand the situation of elderly people, which creates the risk of delayed response in emergencies. Furthermore, elderly people are often exposed to risks such as fraud. There is a need for a system that addresses these issues and provides an environment in which elderly people can live with peace of mind.

[1135] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1136] In the present invention, the server includes the following means, which makes it possible to:

[1137] A means of providing weather, news, and topics based on hobbies and preferences to seniors,

[1138] A means of receiving messages from relatives and notifying the elderly by voice;

[1139] A means to contact emergency contacts and relatives with simple voice commands,

[1140] A means of sending a notice to relatives if there is no response from the elderly person for a certain period of time;

[1141] A method for analyzing elderly people's conversations and issuing warnings in cases of possible fraud;

[1142] means for recognizing and transmitting voice input from the senior citizen to a server, which processes the data and provides an appropriate response to the senior citizen;

[1143] A way to order daily necessities online using voice commands,

[1144] a means for automatically taking appropriate emergency action when a voice command is issued; and

[1145] A method to automatically notify relatives if there is no communication for a certain period of time.

[1146] This will enable elderly people to enjoy everyday conversations and allow relatives to watch over them from a distance with peace of mind. It will also enable quick responses in emergencies and prevent fraud, improving the quality of life for the elderly.

[1147] "Elderly" refers to people over a certain age, who often require special support.

[1148] "Relative" refers to a person who has a close relationship with an older person based on blood or legal ties.

[1149] "Topics based on weather, news, and hobbies and preferences" refers to information that is likely to attract the individual user's interest and attention, and is a topic that enriches daily life.

[1150] "Messages" refer to messages or information sent by relatives to elderly people.

[1151] "Voice commands" refer to instructions that seniors can give using their voice to trigger specific actions.

[1152] "Emergency Contact" refers to contact information that should be used to contact you promptly if necessary.

[1153] "Speech recognition" refers to the technology of analyzing voice as a digital signal and converting it into text data, etc.

[1154] "Voice notification" refers to the function of converting specific information into voice and conveying it to the elderly.

[1155] "Potential fraud" refers to patterns of conversation or behavior intended to deceive seniors.

[1156] "Voice input" refers to voice data provided by a user to a system via a microphone or the like.

[1157] "Data processing" refers to the process by which the server analyzes the data it receives and generates an appropriate response or action.

[1158] "Online shopping" refers to the act of purchasing goods over the Internet.

[1159] "Daily necessities" refers to consumables and necessary products used in daily life.

[1160] "Emergency response" refers to measures taken to respond quickly when an unexpected situation occurs.

[1161] "Notification" refers to a message or signal intended to convey specific information to a target audience.

[1162] The present invention is a system that supports elderly people's conversations and allows their relatives living far away to check on their status. This system is composed of a terminal used by the elderly person, a server that manages and processes various data, and a web interface used by the relatives. Specific embodiments of the present invention will be described below.

[1163] 1. Main system components

[1164] The system consists of the following main components:

[1165] Terminal: A device used by seniors that has voice recognition and notification functions. It provides seniors with information about the weather, news, and topics based on their hobbies and preferences. It also has the ability to recognize voice commands in emergencies and respond immediately.

[1166] Server: Manages and processes various data. Responsible for analyzing voice data, providing emergency notifications, and detecting and warning fraud. The server uses a speech recognition engine (such as the Google Speech-to-Text API) and a natural language processing engine (such as OpenAI GPT-3).

[1167] Relatives' interface: A web interface that allows relatives to send messages remotely and check on the elderly's status. From here, relatives can input and check messages, and receive notifications in case of emergencies.

[1168] 2. Program Processing Overview

[1169] User registration and initial settings

[1170] First, the user enters their information (name, address, contact information for relatives) on the device or a dedicated app, and provides a voice sample during the initial setup. This information is sent from the device to the server, where it is stored in a database and the voice recognition model is updated, improving voice recognition accuracy.

[1171] Providing everyday conversation

[1172] The server generates the day's topics (weather, news, hobbies, etc.) based on the user's profile information and past conversation history. This information is collected using news APIs and weather forecast APIs. The generated topics are provided to the elderly via voice at the specified time via the device. When the user speaks, the device recognizes the voice and sends the text data to the server. The server uses a natural language processing engine to generate an appropriate response, which is sent to the device and played aloud.

[1173] Specific examples

[1174] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[1175] User: Yes, please.

[1176] Terminal: According to this article, roses grow best in the spring.

[1177] Message function from relatives

[1178] Relatives can input messages through a web interface, which are then received by the server. At the designated time, the message is sent to the elderly person's device via voice notification. When the elderly person responds by voice, the message is sent back to the server and notified to the relative.

[1179] Specific examples

[1180] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[1181] Terminal: Have you taken your medicine yet?

[1182] User: Yes, I did.

[1183] Device: Your reply has been notified to your family.

[1184] The server sends a notification to the relative: Father replied that he had taken his medicine.

[1185] Emergency response

[1186] When an elderly person issues an emergency voice command such as "call an ambulance" or "call XX," the device recognizes the command and immediately sends it to the server, which then receives it and automatically notifies the necessary contacts (119 or relatives).

[1187] Specific examples

[1188] User: Call an ambulance

[1189] Terminal: Okay, I'll call 911.

[1190] The server will automatically call 119.

[1191] Server: An emergency has occurred. We will notify you to respond.

[1192] Conversation monitoring and notification to relatives

[1193] If a device has not had a conversation for a certain period of time (e.g., 3 days), it will automatically notify the server, which will then notify the family.

[1194] Specific examples

[1195] (No conversation for 3 days)

[1196] Terminal: It's been a while since we last spoke, I'll contact my family.

[1197] Server sends notification to relatives: Father hasn't spoken in 3 days, please check.

[1198] Online shopping support

[1199] When an elderly person orders a product by voice, the device recognizes the voice and sends the order to the server, which then processes the purchase and notifies the device that the order is complete.

[1200] Specific examples

[1201] User: Order some tissues.

[1202] Terminal: Would you like to order one tissue?

[1203] User: Yes

[1204] Device: I ordered it and it will arrive in 2 days.

[1205] Fraud prevention features

[1206] The device constantly monitors the user's conversations and sends the audio data to the server. The server analyzes the conversation and sends a warning to the device if there is a possibility of fraud. The device notifies the user by voice and also notifies their relatives of the possibility of fraud.

[1207] Specific examples

[1208] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[1209] Device: This is a potential scam message. Do you want to continue?

[1210] Server: Notify relatives: A potentially fraudulent conversation has been detected.

[1211] With the above configuration and processing, the present invention provides a system that allows elderly people to enjoy everyday conversations while allowing relatives to watch over them from a remote location with peace of mind.

[1212] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1213] Step 1:

[1214] The user enters their name, address, contact details of their next of kin into the app or device and provides a voice sample.

[1215] Input: Name, address, contact details of relatives, voice sample

[1216] What happens: The user fills in a form on the device with details and records audio into the microphone.

[1217] Output: Input data and audio samples are saved to the device.

[1218] Step 2:

[1219] The device sends the entered information and voice samples to the server.

[1220] Input: User-entered information and voice samples

[1221] Specific operation: The device packs data into packets and sends them over the Internet to a server.

[1222] Output: Data received by the server

[1223] Step 3:

[1224] The server stores the received information in a database and updates the speech recognition model.

[1225] Input: User information and voice sample

[1226] What happens: The server inserts data into the database and updates the model using a speech recognition engine (e.g., Google Speech-to-Text API).

[1227] Output: Updated database and speech recognition model

[1228] Step 4:

[1229] The server generates the topic for the day based on the user's profile information and past conversation history.

[1230] Input: User profile information, past conversation history

[1231] Specific operation: The server obtains information using news APIs or weather forecast APIs and generates topics using a generative AI model (e.g., OpenAI GPT-3).

[1232] Output: Generated topics

[1233] Step 5:

[1234] The device provides the elderly with the topics generated at the specified time via voice.

[1235] Input: Generated topics

[1236] Specific operation: The device uses a speech synthesis engine (for example, Google Text-to-Speech API) to convert the topic into audio and play it back to the user.

[1237] Output: Elderly people listen to the topic

[1238] Step 6:

[1239] When the user speaks, the device recognizes the voice and sends the text data to the server.

[1240] Input: User voice input

[1241] Specific operation: The device converts the speech into text using a speech recognition engine and sends it to the server.

[1242] Output: Sends audio to the server as text data

[1243] Step 7:

[1244] The server generates an appropriate response and sends it to the terminal, which plays it aloud.

[1245] Input: Sends audio to the server as text data

[1246] Specific operation: The server generates a response using a natural language processing engine (e.g., OpenAI GPT-3) and sends it to the device. The device then converts the response into speech using a speech synthesis engine and plays it back to the user.

[1247] Output: The user hears the reply aloud

[1248] Step 8:

[1249] Relatives can enter messages through a web interface and send them to the server.

[1250] Input: Relative's text message

[1251] Specific actions: A relative logs into the web interface, types a message, and presses send.

[1252] Output: The server receives the message.

[1253] Step 9:

[1254] The server receives the message and sends it to the elderly person's device at the specified time.

[1255] Input: Message from relative

[1256] Specific operation: The server's schedule management system sends notification instructions to the terminal at the specified time.

[1257] Output: The device receives the message.

[1258] Step 10:

[1259] The device will notify you of messages from relatives via voice at the specified time.

[1260] Input: Message notification from the server

[1261] Specific operation: The device uses a speech synthesis engine to convert the message into voice and notify the user.

[1262] Output: User listens to the message

[1263] Step 11:

[1264] The user can respond by voice if necessary, and the terminal sends the content to the server.

[1265] Input: User's voice response

[1266] Specific operation: The device recognizes the voice and sends it to the server as text data.

[1267] Output: Response data is sent to the server

[1268] Step 12:

[1269] The server notifies the relatives of the received reply.

[1270] Input: User response data

[1271] What happens: The server sends emails and SMS to the relatives' contacts.

[1272] Output: The relative receives the reply

[1273] Step 13:

[1274] When an elderly person issues an emergency voice command (e.g., "Call an ambulance"), the device recognizes it and sends it to the server.

[1275] Input: Emergency voice command

[1276] Specific operation: The device analyzes the emergency command using a voice recognition engine and sends it to the server.

[1277] Output: Server receives urgent command

[1278] Step 14:

[1279] The server receives emergency voice commands and automatically notifies the necessary contacts (e.g., 119 or relatives).

[1280] Input: Emergency voice command

[1281] Specific operation: The server automatically notifies the emergency contact via the phone API or message API.

[1282] Output: Emergency is handled

[1283] Step 15:

[1284] If a device has not had a conversation for a certain period of time (e.g., 3 days), it will automatically notify the server.

[1285] Input: Conversation status data

[1286] Specific operation: The device monitors conversation events using an internal counter and notifies the server if there is no update within a certain period of time.

[1287] Output: The server receives the notification

[1288] Step 16:

[1289] The server receives the notification and notifies the relatives.

[1290] Input: Notification from device

[1291] What happens: The server sends emails and SMS to the relatives' contacts.

[1292] Output: Relatives receive notification

[1293] Step 17:

[1294] When an elderly person orders a product by voice, the terminal recognizes the voice and sends the order to the server.

[1295] Input: Voice command to order product

[1296] Specific operation: The device converts the speech into text using a speech recognition engine and sends it to the server.

[1297] Output: The server receives the order.

[1298] Step 18:

[1299] The server completes the purchase process and notifies the terminal that the order has been completed.

[1300] Input: Product order data

[1301] What happens: The server uses the e-commerce API to complete the purchase and confirm the order.

[1302] Output: The terminal will be notified that the order has been completed.

[1303] Step 19:

[1304] The terminal notifies the user that the order is complete.

[1305] Input: Order completion notification

[1306] What happens: The device uses a speech synthesis engine to announce the completion of the order.

[1307] Output: User hears order completed

[1308] Step 20:

[1309] The device constantly monitors the user's conversation and transmits the voice data to the server.

[1310] Input: Conversational audio data

[1311] Specific operation: The device records audio using an audio capture device and streams it to the server.

[1312] Output: The server receives the audio data.

[1313] Step 21:

[1314] The server uses AI models to analyze the conversation and sends a warning to the device if there is a possibility of fraud.

[1315] Input: Conversational audio data

[1316] What it does: The server analyzes the data using natural language processing models and runs algorithms to detect fraudulent patterns.

[1317] Output: Fraud alert data

[1318] Step 22:

[1319] The server receives the fraud warning data and sends it to the terminal, which then notifies the user of the fraud warning by voice.

[1320] Input: Fraud Alert Data

[1321] What happens: The server sends a fraud warning to the device, which then plays it back to the user using its speech synthesis engine.

[1322] Output: User hears fraud warning

[1323] Step 23:

[1324] The server also notifies relatives of possible fraud.

[1325] Input: Fraud Alert Data

[1326] What happens: The server sends emails and SMS to the relatives' contacts.

[1327] Output: Relative receives fraud alert

[1328] (Application example 1)

[1329] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1330] Systems already exist that support the daily conversations of elderly people and allow relatives to check their status remotely, but these systems lack sufficient dialogue generation capabilities and flexible means for relatives to send messages. Furthermore, their emergency response and fraud prevention features for elderly people are often incomplete. Therefore, there is a need for systems with more advanced conversation support and security features that allow elderly people to live their daily lives with peace of mind.

[1331] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1332] In this invention, the server is a system that supports conversations between elderly people and allows relatives to check their status from a remote location, and includes: means for providing the elderly with topics based on the weather, news, and hobbies and preferences; means for receiving messages from relatives and notifying the elderly by voice; means for contacting an ambulance or relatives using simple voice commands; means for notifying relatives if there is no response from the elderly for a certain period of time; means for analyzing the elderly's conversations and issuing a warning if there is a possibility of fraud; means for generating conversation content using a generative artificial intelligence model and providing daily conversation support; and means for relatives to easily send messages using prompt sentences. This provides more advanced conversation support, faster emergency response, and fraud prevention functions than conventional systems, allowing elderly people to live their lives with peace of mind.

[1333] The "elderly conversation support system" is a system that supports the everyday conversations of the elderly and allows relatives in remote locations to check on their status.

[1334] The "topic provision means" is a function that provides elderly people with topics generated based on weather, news, and hobbies and preferences via voice.

[1335] The "message notification means" is a function that receives messages sent by relatives and notifies the elderly person by voice.

[1336] The "emergency contact method" is a function that allows elderly people to contact an ambulance or relatives in an emergency by issuing simple voice commands.

[1337] The "no response notification method" is a function that automatically sends a notification to relatives if there is no response from the elderly person for a certain period of time.

[1338] The "fraud warning feature" is a function that analyzes the elderly person's conversation and issues an audio warning if there is a possibility of fraud.

[1339] A "generative artificial intelligence model" is an artificial intelligence technology that generates appropriate conversation content based on data such as a user's profile information and past conversation history.

[1340] A "prompt sentence" is an instruction sentence that is input to a generative artificial intelligence model, allowing relatives to easily send messages.

[1341] The present invention is a system that supports conversations between elderly people and allows their remote relatives to check on their status. This system includes a means for providing topics based on weather, news, and hobbies and preferences, a means for receiving messages from relatives and notifying them by voice, a means for recognizing and immediately responding to voice commands in emergencies, a means for notifying relatives if there is no response from the elderly person for a certain period of time, a means for preventing fraud through voice analysis, a means for generating conversation content using a generative artificial intelligence model to support daily conversations, and a means for relatives to easily send messages using prompt sentences.

[1342] System Configuration

[1343] This system consists of the following main components:

[1344] 1. Device:

[1345] This device, used by seniors, has voice recognition and notification functions, and provides them with information about the weather, news, and topics based on their hobbies and preferences.

[1346] 2. Server:

[1347] Manages and processes various data. Analyzes voice data and sends emergency notifications. Generates conversation content using generative artificial intelligence models.

[1348] 3. Kindred Interface:

[1349] A web interface that allows relatives to remotely send messages or check on the status of elderly people. Messages can be sent using prompts.

[1350] Functions and Processing

[1351] User registration and initial settings

[1352] The user (elderly person) enters their name, address, and contact information of their relatives into the app or device, and provides a voice sample to the system during initial setup. The device then sends the entered information and voice sample to the server, which stores this information in a database and updates the voice recognition model.

[1353] Providing everyday conversation

[1354] The server generates the topic of the day (weather, news, hobbies, etc.) based on the user's profile information and past conversation history. The device provides the generated topic to the elderly via voice at the specified time. When the user enjoys a conversation and speaks to the device, the device recognizes the user's speech and sends it to the server. The server generates an appropriate response and sends it back to the device.

[1355] Examples:

[1356] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[1357] User: Yes, please.

[1358] Terminal: According to this article, roses grow best in the spring.

[1359] Message function from relatives

[1360] The relative inputs and sends a message via a web interface. The server receives the message and sends it to the elderly person's device at the specified time. The device then notifies the relative of the message by voice at the specified time, and the user can respond by voice as needed. The device then sends the response to the server, which notifies the relative.

[1361] Examples:

[1362] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[1363] Terminal: Have you taken your medicine yet?

[1364] User: Yes, I did.

[1365] Device: Your reply has been notified to your family.

[1366] The server sends a notification to the relative: Father replied that he had taken his medicine.

[1367] Emergency response

[1368] The user issues a voice command such as "Call an ambulance" or "Call XX." The device recognizes the voice command and immediately sends it to the server. The server receives the emergency voice command and automatically notifies the necessary contacts (119 or relatives).

[1369] Examples:

[1370] User: Call an ambulance

[1371] Terminal: Okay, I'll call 911.

[1372] An automatic call will be made to 119 via the server.

[1373] Server: An emergency has occurred. We will notify you to respond.

[1374] Notification function when there is no response for a certain period of time

[1375] If there is no conversation for a certain period of time (e.g., 3 days), the device automatically notifies the server. The server receives the notification and notifies the relatives.

[1376] Examples:

[1377] If there is no conversation for three days

[1378] Terminal: We haven't spoken in a while, I'll contact my family.

[1379] Server sends notification to relatives: Father hasn't spoken in 3 days, please check.

[1380] Fraud prevention features

[1381] The device constantly monitors the user's conversation and sends the audio data to the server. The server uses a generative artificial intelligence model to analyze the conversation and detect possible fraud. If it determines that there is a high possibility of fraud, it sends a warning to the device and notifies the user of the fraud warning via voice. At the same time, the server notifies relatives of the possibility of fraud.

[1382] Examples:

[1383] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[1384] Device: This is a potential scam message. Do you want to continue?

[1385] Server: Notify relatives: A potentially fraudulent conversation has been detected.

[1386] With the above configuration and processing, the present invention provides a system that allows elderly people to enjoy everyday conversations while their relatives can watch over them with peace of mind.

[1387] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1388] Step 1:

[1389] User registration and initial settings

[1390] The user (elderly person) uses an input device (terminal) to enter basic information such as their name, address, and contact details of their relatives. In addition, they provide several voice samples during the initial setup. The terminal then sends this information and the voice samples to the server, which stores the received information in a database and updates the voice recognition model.

[1391] Input: Name, address, contact details of relatives, voice sample

[1392] Output: Updated speech recognition model, user information stored in the database

[1393] Step 2:

[1394] Providing everyday conversation

[1395] The server uses a generative artificial intelligence model to generate the day's topics (weather, news, hobbies, etc.) based on the user's profile information and past conversation history. The generated topics are sent from the server to the device, which then provides them to the elderly via voice at the specified time.

[1396] Input: User profile information, past conversation history

[1397] Output: Generated topics, information provided in audio

[1398] Specific operation: The server generates a topic called "It's forecast to be sunny," and the device announces "It's forecast to be sunny today" by voice.

[1399] Step 3:

[1400] Message function from relatives

[1401] Relatives use a web interface to input and send messages. The server receives the messages and sends them to the elderly person's device at the specified time. The device then notifies the elderly person of the message by voice at the specified time. The elderly person can then respond by voice if necessary, and the response is sent from the device to the server, where it is then notified to the relative.

[1402] Input: Message from relative, user's voice response

[1403] Output: Voice notification to the elderly, notification of the response to relatives

[1404] Specific operation: A relative sends a message saying, "Please make sure you have taken your medicine today," and the device notifies the user by voice, "Have you taken your medicine yet?", to which the user replies, "Yes, I have."

[1405] Step 4:

[1406] Emergency response

[1407] When a user issues a voice command such as "call an ambulance" or "call XX," the device recognizes the command and immediately sends it to the server. The server receives the emergency voice command and automatically notifies the necessary contacts (such as 119 or relatives).

[1408] Input: Senior emergency voice command

[1409] Output: Automatic call to emergency contacts

[1410] Specific operation: The user says "call an ambulance," the device recognizes the voice and sends it to the server, which then automatically calls 119.

[1411] Step 5:

[1412] Notification function when there is no response for a certain period of time

[1413] If the device does not receive a response from the elderly person for a certain period of time (for example, three days), it automatically sends a notification to the server, which then receives the notification and sends it to the elderly person's relatives.

[1414] Input: No response for a certain period of time

[1415] Output: No response notification from the server to relatives

[1416] Specific operation: If there is no response from the elderly person for three days, the device will notify the family, "We haven't heard from you for a long time. We will contact your family," and the server will notify the relative, "Your father has not spoken for three days. Please check."

[1417] Step 6:

[1418] Fraud prevention features

[1419] The device constantly monitors the user's conversation and sends the audio data to the server. The server uses a generative artificial intelligence model to analyze the conversation and detect possible fraud. If it determines that there is a high possibility of fraud, it sends a warning to the device and notifies the user via audio. At the same time, the server notifies relatives of the possibility of fraud.

[1420] Input: Elderly people's conversation

[1421] Output: Audio notification of fraud warning, fraud notification to relatives

[1422] Specific operation: If an unknown caller says, "You have won 1 million yen," the server detects the possibility of fraud, the device warns the user with a voice message saying, "This is a potentially fraudulent message," and the server notifies relatives that, "A potentially fraudulent conversation has been detected."

[1423] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1424] This invention combines a system that supports conversations between elderly people and allows their relatives in remote locations to check their status with an emotion engine that recognizes the user's emotions. This system provides topics based on weather, news, and hobbies and preferences, receives messages from relatives and notifies them by voice, recognizes voice commands in emergencies and responds immediately, notifies relatives if there is no response from the elderly person for a certain period of time, and prevents fraud through voice analysis. It is also possible to use the emotion engine to analyze the elderly person's emotions and provide appropriate responses or notifications.

[1425] System Configuration

[1426] The system consists of the following main components:

[1427] 1. Terminal: A device used by the elderly that has voice recognition, emotion recognition, and notification functions. It provides the elderly with information about the weather, news, and topics based on their hobbies and preferences via voice.

[1428] 2. Server: Manages and processes various data, analyzes voice and emotion data, and sends notifications in the event of an emergency.

[1429] 3. Relatives' interface: A web interface for relatives to send messages remotely and check on the status of the elderly.

[1430] 4. Emotion engine: It has the ability to analyze the user's voice and identify their emotional state.

[1431] Program processing overview

[1432] 1. User registration and initial settings

[1433] User: Enters name, address, and contact details of relatives into the app or device, and provides a voice sample during initial setup.

[1434] Terminal: Sends entered information and voice samples to the server.

[1435] Server: This information is stored in a database and the speech recognition model and emotion recognition model are updated.

[1436] 2. Providing everyday conversation

[1437] Server: Generates topics for the day based on the user's profile information and past conversation history. Sends the generated topics to the device.

[1438] Terminal: Provides the elderly with audio information about topics generated at the specified time.

[1439] Emotion engine: Analyzes the elderly person's responses and identifies their emotional state (e.g., joy, sadness, anger, etc.).

[1440] User: Talk to the device about a topic.

[1441] Terminal: Recognizes the user's speech and sends it to the server.

[1442] Server: Generates an appropriate response and sends it back to the device. Based on the analysis results of the emotion engine, an appropriate response is given according to the emotion.

[1443] Specific examples

[1444] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[1445] User: Yes, please. (Shows delight)

[1446] Emotion engine: Analyzes user responses as emotions of joy.

[1447] Server: Generates a response based on the emotion of joy.

[1448] Terminal: According to this article, roses grow best in the spring. That's so lovely!

[1449] 3. Message function from relatives

[1450] Relatives: Enter and send messages via the web interface.

[1451] Server: Receives the message and sends it to the elderly person's device at the specified time.

[1452] Device: Receives voice notification of messages from relatives at the specified time.

[1453] User: Responds verbally if necessary.

[1454] Emotion engine: Analyzes the user's responses and identifies their emotional state.

[1455] Terminal: Sends the response content and emotional state to the server.

[1456] Server: Notifies relatives of response and emotional state.

[1457] Specific examples

[1458] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[1459] Terminal: Have you taken your medicine yet?

[1460] User: Yes, I did. (Shows relief)

[1461] Emotion engine: Analyzes user responses as emotions of relief.

[1462] Server: Notify relatives of response along with emotional state of relief.

[1463] Family Interface: Father has taken his medicine. He seems relieved.

[1464] 4. Emergency Response

[1465] User: Issues an emergency voice command such as "Call an ambulance" or "Call ____."

[1466] Terminal: Recognizes voice commands and immediately notifies the server.

[1467] Emotion engine: Analyzes emotions (e.g., surprise, fear, etc.) during emergencies.

[1468] Server: Receives emergency voice commands and emotional state, and automatically notifies necessary contacts (119 or relatives).

[1469] Specific examples

[1470] User: Call an ambulance. (Shows fear)

[1471] Terminal: Okay, I'll call 911.

[1472] Emotion engine: Analyzes user responses as emotions of fear.

[1473] Server: Automatically calls 119 and notifies relatives along with the emotional state of fear.

[1474] Kinship Interface: An emergency has occurred. They appear frightened.

[1475] 5. Monitoring conversations and notifying relatives

[1476] Terminal: If there is no conversation for a certain period of time (e.g., 3 days), the server will be automatically notified.

[1477] Server: Receives the notification and notifies relatives.

[1478] Specific examples

[1479] (No conversation for 3 days)

[1480] Terminal: We haven't spoken in a while, I'll contact my family.

[1481] Server: Send notification to relatives.

[1482] Kinship Interface: Father hasn't spoken in 3 days. Please check.

[1483] 6. Online shopping support

[1484] User: Use voice commands to order the items they need.

[1485] Terminal: Recognizes voice and sends orders to the server.

[1486] Server: Receives the order details and processes the purchase on the e-commerce site.

[1487] Server: After the purchase procedure is completed, send the completion information to the terminal.

[1488] Terminal: A voice message is sent to the user to notify them that the order has been completed.

[1489] Specific examples

[1490] User: Order some tissues.

[1491] Terminal: Would you like to order one tissue?

[1492] User: Yes

[1493] Device: I ordered it and it will arrive in 2 days.

[1494] 7. Fraud Prevention Features

[1495] Terminal: Constantly monitors the user's conversation and sends the voice data to the server.

[1496] Server: Uses AI models to analyze conversations and detect potential fraud.

[1497] Emotion Engine: Analyzes the user's emotional state and immediately warns if there is a high possibility of fraud.

[1498] Server: If it determines that there is a high possibility of fraud, it sends a warning along with the emotional state to the device.

[1499] Device: Provides audio fraud warnings along with emotional state.

[1500] Server: Notifies relatives of possible fraud and emotional state.

[1501] Specific examples

[1502] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[1503] Device: This is a potential scam message. Do you want to continue?

[1504] Emotion Engine: Analyzes the user's confused emotional state.

[1505] Server: High likelihood of fraud, along with a confused emotional state, sends out a warning.

[1506] Kinship Interface: A potentially fraudulent conversation has been detected. Your father appears confused.

[1507] The above configuration and processing not only allows elderly people to enjoy everyday conversations and allows relatives to watch over them with peace of mind, but also makes it possible to provide a system that uses an emotion engine to respond appropriately to the emotional state of the elderly.

[1508] The processing flow will be explained below.

[1509] User registration and initial settings

[1510] Step 1:

[1511] User: Launches the app or device and enters initial information such as name, address, and contact details of next of kin. They are then prompted to provide a voice sample.

[1512] Step 2:

[1513] Terminal: Sends entered information and voice samples to the server.

[1514] Step 3:

[1515] Server: Stores the received information in a database and updates the speech recognition model and emotion recognition model.

[1516] Providing everyday conversation

[1517] Step 1:

[1518] Server: Generates topics for the day (weather, news, hobbies, etc.) based on the user's profile information and past conversation history. Sends the generated topics to the device.

[1519] Step 2:

[1520] Terminal: Provides the elderly with audio information about topics generated at the specified time.

[1521] Step 3:

[1522] Emotion engine: Analyzes the elderly person's responses and identifies their emotional state.

[1523] Step 4:

[1524] User: Talk to the device about a topic.

[1525] Step 5:

[1526] Terminal: Recognizes the user's speech and sends it to the server.

[1527] Step 6:

[1528] Server: Generates an appropriate response and sends a response to the device according to the emotion based on the analysis results of the emotion engine.

[1529] Step 7:

[1530] Terminal: Provides the user with a voice response from the server.

[1531] Specific examples

[1532] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[1533] User: Yes, please. (Shows delight)

[1534] Emotion engine: Analyzes user responses as emotions of joy.

[1535] Server: Generates a response based on the emotion of joy.

[1536] Terminal: According to this article, roses grow best in the spring. That's so lovely!

[1537] Message function from relatives

[1538] Step 1:

[1539] Relatives: Enter and send messages via the web interface.

[1540] Step 2:

[1541] Server: Receives the message and sends it to the elderly person's device at the specified time.

[1542] Step 3:

[1543] Terminal: Notifies the elderly of messages from relatives via voice at the specified time.

[1544] Step 4:

[1545] User: Responds verbally if necessary.

[1546] Step 5:

[1547] Emotion engine: Analyzes the user's responses and identifies their emotional state.

[1548] Step 6:

[1549] Terminal: Sends the response content and emotional state to the server.

[1550] Step 7:

[1551] Server: Notifies relatives of response and emotional state.

[1552] Specific examples

[1553] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[1554] Terminal: Have you taken your medicine yet?

[1555] User: Yes, I did. (Shows relief)

[1556] Emotion engine: Analyzes user responses as emotions of relief.

[1557] Server: Notify relatives of response along with emotional state of relief.

[1558] Family Interface: Father has taken his medicine. He seems relieved.

[1559] Emergency response

[1560] Step 1:

[1561] User: Issues an emergency voice command such as "Call an ambulance" or "Call ____."

[1562] Step 2:

[1563] Terminal: Recognizes voice commands and immediately notifies the server.

[1564] Step 3:

[1565] Emotion engine: Analyzes emotions (e.g., surprise, fear, etc.) during emergencies.

[1566] Step 4:

[1567] Server: Receives emergency voice commands and emotional state, and automatically notifies necessary contacts (119 or relatives).

[1568] Step 5:

[1569] Server: Sends notification completion information along with the emotional state to the terminal.

[1570] Step 6:

[1571] Terminal: Notifies the elderly person by voice when the report is complete.

[1572] Specific examples

[1573] User: Call an ambulance. (Shows fear)

[1574] Terminal: Okay, I'll call 911.

[1575] Emotion engine: Analyzes user responses as emotions of fear.

[1576] Server: Automatically calls 119 and notifies relatives along with the emotional state of fear.

[1577] Kinship Interface: An emergency has occurred. They appear frightened.

[1578] Conversation monitoring and notification to relatives

[1579] Step 1:

[1580] Terminal: If there is no conversation for a certain period of time (e.g., 3 days), the server will be automatically notified.

[1581] Step 2:

[1582] Server: Receives the notification and notifies relatives.

[1583] Specific examples

[1584] (No conversation for 3 days)

[1585] Terminal: We haven't spoken in a while, I'll contact my family.

[1586] Server: Send notification to relatives.

[1587] Kinship Interface: Father hasn't spoken in 3 days. Please check.

[1588] Online shopping support

[1589] Step 1:

[1590] User: Use voice commands to order the items they need.

[1591] Step 2:

[1592] Terminal: Recognizes the voice and sends it to the server.

[1593] Step 3:

[1594] Server: Receives the order details and processes the purchase on the e-commerce site.

[1595] Step 4:

[1596] Server: After the purchase procedure is completed, send the completion information to the terminal.

[1597] Step 5:

[1598] Terminal: A voice message is sent to the user to notify them that the order has been completed.

[1599] Specific examples

[1600] User: Order some tissues.

[1601] Terminal: Would you like to order one tissue?

[1602] User: Yes

[1603] Device: I ordered it and it will arrive in 2 days.

[1604] Fraud prevention features

[1605] Step 1:

[1606] Terminal: Constantly monitors the user's conversation and sends the voice data to the server.

[1607] Step 2:

[1608] Emotion Engine: Analyzes the user's emotional state.

[1609] Step 3:

[1610] Server: Uses AI models to analyze conversations and detect potential fraud, taking into account the results of the sentiment engine.

[1611] Step 4:

[1612] Server: If it determines that there is a high possibility of fraud, it sends a warning along with the emotional state to the device.

[1613] Step 5:

[1614] Device: Provides audio fraud warnings along with emotional state.

[1615] Step 6:

[1616] Server: Notifies relatives of possible fraud and emotional state.

[1617] Specific examples

[1618] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[1619] Device: This is a potential scam message. Do you want to continue?

[1620] Emotion Engine: Analyzes the user's confused emotional state.

[1621] Server: High likelihood of fraud, along with a confused emotional state, sends out a warning.

[1622] Kinship Interface: A potentially fraudulent conversation has been detected. Your father appears confused.

[1623] The above configuration and processing not only allows elderly people to enjoy everyday conversations and allows relatives to watch over them with peace of mind, but also makes it possible to provide a system that uses an emotion engine to respond appropriately to the emotional state of the elderly.

[1624] Example 2

[1625] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1626] Elderly people often feel lonely and anxious in their daily lives. In particular, when relatives live far away, it is difficult for them to keep track of the elderly's condition, increasing safety and health risks. There are also situations where a quick response is required in an emergency, and there are risks such as fraud. There is a need for a system that can solve these issues and support the lives of the elderly while allowing relatives to keep an eye on them with peace of mind.

[1627] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for supporting the elderly person's conversation and allowing relatives to check the elderly person's status from a remote location; means for providing topics based on the weather, news, and hobbies and preferences; means for receiving messages from relatives and notifying them by voice; means for contacting emergency contact information and relatives with simple voice commands; means for sending a notification to relatives if there is no response for a certain period of time; means for analyzing the conversation and issuing a warning if there is a possibility of fraud; means for analyzing the elderly person's voice and identifying their emotional state; and means for providing appropriate responses and notifications based on the emotional state. This supports the elderly person's daily life and enables relatives to know the elderly person's status from a remote location in real time.

[1628] "Elderly" refers to an individual who is elderly and requires assistance with daily living.

[1629] "Relative" refers to a person who is related to an older person by blood or marriage, or who has the responsibility to care for an older person.

[1630] A "conversation support system" is a system that supports the elderly in their everyday conversations and allows relatives to check on their status from a remote location.

[1631] "Topic provision means" refers to a means of providing topics to elderly people via audio based on the weather, news, hobbies, and preferences.

[1632] "Message notification means" refers to a means of notifying elderly people by voice of messages received from relatives.

[1633] "Emergency contact methods" refer to methods that allow seniors to contact emergency contacts and relatives using simple voice commands.

[1634] "Response monitoring measures" refer to measures that send notifications to relatives if there is no response from the elderly person for a certain period of time.

[1635] "Fraud warning measures" refer to measures that analyze elderly people's conversations and issue warnings if there is a possibility of fraud.

[1636] "Voice analysis means" refers to a means for analyzing the voice of an elderly person and identifying their emotional state.

[1637] "Emotional response measures" refer to measures that provide appropriate responses or notifications based on the emotional state of the elderly person.

[1638] "Online shopping support means" refers to a means for ordering daily necessities using voice commands.

[1639] This invention relates to a system that supports conversations between elderly people and allows their remote relatives to check their status in real time. This system provides topics based on weather, news, and hobbies and preferences, and includes functions for notifying messages from relatives via voice, recognizing and responding to voice commands in emergencies, notifying relatives if there is no response from the elderly person for a certain period of time, and preventing fraud through voice analysis. It also uses an emotion engine to analyze the emotions of the elderly person and provide appropriate responses and notifications.

[1640] The system consists of the following main components:

[1641] 1. Terminal: A device used by the elderly that has voice recognition, emotion recognition, and notification functions. It provides the elderly with information about the weather, news, and topics based on their hobbies and preferences via voice.

[1642] 2. Server: Manages and processes various data, analyzes voice and emotion data, and sends notifications in the event of an emergency.

[1643] 3. Relatives' interface: A web interface for relatives to send messages and check on the status of the elderly from remote locations.

[1644] 4. Emotion engine: It has the ability to analyze the user's voice and identify their emotional state.

[1645] Hardware and Software Use Cases

[1646] 1. Devices: Devices can be tablet devices or smart speakers (e.g., Amazon Echo, Google Home) with internet connectivity, allowing seniors to easily initiate conversations and respond when presented with a topic.

[1647] 2. Server: For the server, we use high-performance cloud services (e.g., Amazon Web Services, Google Cloud Platform), which enable large-volume data processing and real-time responses.

[1648] 3. Emotion Engine: Using Google Cloud Speech-to-Text API and IBM Watson Tone Analyzer, analyze the elderly person's voice to identify their emotional state.

[1649] Detailed System Description

[1650] User registration and initial settings

[1651] Users enter their name, address, and contact information for relatives into the app or their device, and provide a voice sample, which allows the system to register the user's voice and basic information in a database to improve the accuracy of voice and emotion recognition.

[1652] Providing everyday conversation

[1653] The server uses a generative AI model to generate appropriate topics based on the user's profile information and past conversation history, and sends them to the device. The device then provides the topics to the elderly via voice at the specified time. For example, it provides information in the form of, "Good morning. It's forecast to be sunny today. We have a new article about gardening. Would you like to take a look?"

[1654] Message function from relatives

[1655] When a relative sends a message via a web interface, the server receives it and sends it to the device at the specified time. For example, a message such as "Please make sure you have taken your medicine today" can be sent, and the device will notify the elderly person by voice.

[1656] Emergency response

[1657] When a user issues a simple voice command such as "call an ambulance" or "call XX," the device recognizes it and immediately notifies the server, which then automatically notifies the necessary contacts (e.g., 119 or relatives), enabling a prompt response.

[1658] Conversation monitoring and notification to relatives

[1659] If the device has not spoken for a certain period of time (e.g., three days), it notifies the server, and the server sends a notification to the relatives informing them of the situation. For example, a message such as "Your father has not spoken for three days. Please check." will be sent to the relatives.

[1660] Online shopping support

[1661] When a user orders everyday items by voice, the device recognizes the order and sends it to the server, which then processes the purchase through the e-commerce site and notifies the user that the order is complete.

[1662] Fraud prevention features

[1663] The device constantly monitors the user's conversation and sends the audio data to the server, which then uses an AI model to analyze the conversation. If there is a high possibility of fraud, the emotion engine analyzes the user's confusion or anxiety and immediately issues a fraud warning.

[1664] The system will support the elderly in their daily lives, allowing relatives to remotely monitor their condition in real time, and will also use an emotion engine to enable more appropriate responses.

[1665] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1666] Program processing flow

[1667] User registration and initial settings

[1668] Step 1:

[1669] The user enters their name, address, and contact details of their next of kin into the app or device and provides a voice sample.

[1670] Input: User's personal information (name, address, contact details of relatives) and voice sample

[1671] Output: The input data is saved on the device and ready to be sent to the next step.

[1672] Specific behavior:

[1673] Users follow the instructions on the app or device's initial setup screen, enter the required information, and provide a voice sample, which is recorded through the device's microphone.

[1674] Step 2:

[1675] The device sends the entered information and voice samples to the server.

[1676] Input: User personal information and voice sample (output of step 1)

[1677] Output: Sending confirmation and server receiving status

[1678] Specific behavior:

[1679] The terminal encrypts the input data and sends it securely to the server, after which the server confirms receipt of the data and returns a receipt confirmation message to the terminal.

[1680] Step 3:

[1681] The server stores this information in a database and updates the speech recognition model and emotion recognition model.

[1682] Input: User personal information and voice sample (output of step 2)

[1683] Output: Database update status and model update completion

[1684] Specific behavior:

[1685] The server stores the received user information in a database and uses the voice samples to train speech and emotion recognition models.

[1686] Providing everyday conversation

[1687] Step 1:

[1688] The server generates the topic of the day based on the user's profile information and past conversation history.

[1689] Input: User profile information and past conversation history

[1690] Output: Generated topic text data

[1691] Specific behavior:

[1692] The server uses a generative AI model to generate appropriate topics based on the user's hobbies, preferences, and past conversation history, such as weather, news, and hobby articles.

[1693] Step 2:

[1694] The device provides the elderly with audio topics generated at the specified time.

[1695] Input: Generated topic text data (output of Step 1)

[1696] Output: Audio notification completion status

[1697] Specific behavior:

[1698] The device will provide a topic by voice at a set time (e.g., every morning at 9:00). A speech synthesis engine will convert the text data into voice and notify the elderly.

[1699] Step 3:

[1700] The user speaks to the terminal about a topic.

[1701] Input: User's voice response

[1702] Output: Recorded audio data

[1703] Specific behavior:

[1704] The user speaks their thoughts or questions about the topics provided by the device, and the device's microphone records the voice.

[1705] Step 4:

[1706] The terminal recognizes what the user says and sends it to the server.

[1707] Input: User's voice data (output of Step 3)

[1708] Output: Text converted data and transmission completion status

[1709] Specific behavior:

[1710] The terminal uses a speech recognition engine to convert the user's voice into text data and transmits it to the server.

[1711] Step 5:

[1712] The server generates an appropriate response and sends it back to the terminal.

[1713] Input: Text data (output of step 4)

[1714] Output: Generated response text data

[1715] Specific behavior:

[1716] The server uses a generative AI model to generate appropriate responses to the user's text data and sends them to the device. An emotion engine also works in parallel to analyze the user's emotional state.

[1717] Step 6:

[1718] The terminal notifies the user of the response from the server by voice.

[1719] Input: Generated response text data (output of Step 5)

[1720] Output: Audio notification completion status

[1721] Specific behavior:

[1722] The terminal uses a speech synthesis engine to convert the response text data into speech and notify the user.

[1723] Message function from relatives

[1724] Step 1:

[1725] Relatives can enter and send messages via a web interface.

[1726] Input: Relative's text message

[1727] Output: Transmission complete status

[1728] Specific behavior:

[1729] The relative logs into the web interface, enters a message, and presses the send button, which sends the message to the server.

[1730] Step 2:

[1731] The server receives the message and sends it to the elderly person's device at the specified time.

[1732] Input: Relative's text message (output of step 1)

[1733] Output: Confirmation of sending and notification to the terminal that it is ready

[1734] Specific behavior:

[1735] The server schedules and manages the received messages, sending them to the elderly person's device at the specified time.

[1736] Step 3:

[1737] The device will notify you of messages from relatives via voice at the specified time.

[1738] Input: Message text data

[1739] Output: Audio notification completion status

[1740] Specific behavior:

[1741] At the set time, the device uses a voice synthesis engine to convert the relative's message into voice and notify the elderly person.

[1742] Step 4:

[1743] The user responds verbally as needed.

[1744] Input: User's voice reply

[1745] Output: Recorded audio data

[1746] Specific behavior:

[1747] The user responds to the message by voice and speaks it into the terminal.

[1748] Step 5:

[1749] An emotion engine analyzes the user's responses to identify their emotional state.

[1750] Input: User's voice data (output of step 4)

[1751] Output: Emotional state data

[1752] Specific behavior:

[1753] The emotion engine analyzes voice data to identify emotional states such as relief or joy.

[1754] Step 6:

[1755] The device sends the response and emotional state to the server.

[1756] Input: Text-converted response data and emotional state data

[1757] Output: Transmission complete status

[1758] Specific behavior:

[1759] The terminal converts the user's response into text data and transmits it to the server together with the emotional state data.

[1760] Step 7:

[1761] The server notifies the relative of the response and emotional state.

[1762] Input: Response text data and emotional state data (output of Step 6)

[1763] Output: Notification to relatives completed

[1764] Specific behavior:

[1765] Based on the received data, the server sends a message to the relatives, including information about the user's emotional state. A notification such as "Your father has taken his medicine. He seems relieved" is sent.

[1766] Emergency response

[1767] Step 1:

[1768] The user issues an emergency voice command such as "call an ambulance" or "call XX."

[1769] Input: User's voice command

[1770] Output: Recorded audio data

[1771] Specific behavior:

[1772] In an emergency, the user speaks loudly into the device, such as "call an ambulance," and the device's microphone records the voice.

[1773] Step 2:

[1774] The device recognizes the voice command and immediately notifies the server.

[1775] Input: User's voice command data (output of step 1)

[1776] Output: Transmission complete status

[1777] Specific behavior:

[1778] When the device recognizes an emergency voice command, it immediately notifies the server.

[1779] Step 3:

[1780] The emotion engine analyzes emotions (e.g., surprise, fear, etc.) during emergencies.

[1781] Input: User's voice data (output of step 2)

[1782] Output: Emotional state data

[1783] Specific behavior:

[1784] The emotion engine analyzes the voice data to identify the user's emotional state in emergencies, such as surprise or fear.

[1785] Step 4:

[1786] The server receives emergency voice commands and emotional state and automatically notifies the necessary contacts (119 or relatives).

[1787] Input: Emergency voice command data and emotional state data (output of Step 3)

[1788] Output: Status of contact notification completion

[1789] Specific behavior:

[1790] The server automatically calls 119 or relatives based on emergency voice commands and the user's emotional state, such as "call an ambulance."

[1791] Conversation monitoring and notification to relatives

[1792] Step 1:

[1793] If a device has not had a conversation for a certain period of time (e.g., 3 days), it will automatically notify the server.

[1794] Input: Conversation absence period

[1795] Output: Notification data

[1796] Specific behavior:

[1797] The device will automatically send a notification to the server if there is no conversation for three days.

[1798] Step 2:

[1799] The server receives the notification and notifies the relatives.

[1800] Input: Notification data (output of step 1)

[1801] Output: Notification to relatives completed

[1802] Specific behavior:

[1803] The server notifies the relatives that there has been no communication for three days. The relatives receive a message saying, "Your father has not spoken for three days. Please check."

[1804] Online shopping support

[1805] Step 1:

[1806] The user orders the desired products by voice.

[1807] Input: User's voice commands

[1808] Output: Recorded audio data

[1809] Specific behavior:

[1810] The user speaks a voice instruction to the terminal, such as "Please order some tissues."

[1811] Step 2:

[1812] The terminal recognizes the voice and sends the order to the server.

[1813] Input: User's voice data (output of step 1)

[1814] Output: Text converted data and transmission completion status

[1815] Specific behavior:

[1816] The terminal uses a voice recognition engine to convert the voice data into text data and transmits it to the server.

[1817] Step 3:

[1818] The server receives the order details and processes the purchase on the e-commerce site.

[1819] Input: Text-converted order data (output of Step 2)

[1820] Output: Purchase procedure completed

[1821] Specific behavior:

[1822] The server uses the specified e-commerce site API to complete the purchase process based on the order details.

[1823] Step 4:

[1824] After the server completes the purchase procedure, it sends completion information to the terminal.

[1825] Input: Purchase completion data (output of Step 3)

[1826] Output: Transmission complete status

[1827] Specific behavior:

[1828] The server sends purchase completion information to the terminal and prepares for notification to the user.

[1829] Step 5:

[1830] The terminal will notify the user by voice that the order has been completed.

[1831] Input: Purchase completion data (output from Step 4)

[1832] Output: Audio notification completion status

[1833] Specific behavior:

[1834] The device will use a speech synthesis engine to notify the user by voice that the purchase has been completed, with a message such as "Your order has been placed. It will arrive in two days."

[1835] Fraud prevention features

[1836] Step 1:

[1837] The device constantly monitors the user's conversation and transmits the voice data to the server.

[1838] Input: Conversational voice data

[1839] Output: Transmission complete status

[1840] Specific behavior:

[1841] The device monitors the user's conversation and transmits the voice data to the server in real time.

[1842] Step 2:

[1843] The server uses AI models to analyze the conversation and detect potential fraud.

[1844] Input: Transmitted audio data (output of step 1)

[1845] Output: Fraud probability data

[1846] Specific behavior:

[1847] The server uses an AI model to analyze the received voice data and detect trigger words and phrases that may indicate fraud.

[1848] Step 3:

[1849] The emotion engine analyzes the user's emotional state and immediately warns if there is a high possibility of fraud.

[1850] Input: Transmitted voice data (output of Step 1) and analysis results of the AI ​​model (output of Step 2)

[1851] Output: Emotional state and alert data

[1852] Specific behavior:

[1853] The emotion engine analyzes the user's emotional state and issues a warning if confusion, anxiety, etc. are detected.

[1854] Step 4:

[1855] If the server determines that there is a high possibility of fraud, it sends a warning to the device along with the emotional state.

[1856] Input: Emotional state and alert data (output of step 3)

[1857] Output: Transmission complete status

[1858] Specific behavior:

[1859] If the server determines that there is a high possibility of fraud, it sends a warning message along with the user's emotional state to the terminal.

[1860] Step 5:

[1861] The device will provide audible warnings of fraud along with emotional state.

[1862] Input: Emotional state and alert data (output of step 4)

[1863] output: Completion status of the audio notification

[1864] Specific behavior:

[1865] The device uses a voice synthesis engine to notify users of their emotional state and fraud warnings, such as "This message may be fraudulent. Do you want to continue?"

[1866] Step 6:

[1867] The server notifies the relatives of the possible fraud and their emotional state.

[1868] Input: Emotional state and alert data (output of step 4)

[1869] Output: Notification to relatives completed

[1870] Specific behavior:

[1871] The server notifies the relatives of the likelihood of fraud and their emotional state. A notification such as "A conversation with a high likelihood of fraud has been detected. Your father seems confused" is sent to the relatives.

[1872] (Application example 2)

[1873] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1874] There is a need for a system that allows elderly people to communicate with their relatives in remote locations with peace of mind and that can respond quickly in emergencies. Another important issue is how to improve the user experience by supporting elderly people in easily shopping in virtual stores using voice commands and by suggesting appropriate products based on their emotions.

[1875] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for identifying the emotional state of the elderly person based on emotion recognition and providing appropriate responses or notifications as necessary, means for performing voice-based product searches in the virtual store and suggesting products according to the emotions, and means for displaying advice from relatives. This enables the elderly person to enjoy shopping in the virtual store with peace of mind while receiving support according to their emotions.

[1876] The "system" is a complex set of devices and software that assists elderly people in conversation and allows relatives to check on their condition remotely.

[1877] "Means" refers to a method, apparatus, or program for performing a specified function or task.

[1878] "Emotion recognition" is a technology that analyzes and identifies a person's emotional state from their voice or text data.

[1879] "Emotional state" refers to a person's current psychological state, such as happiness, sadness, anger, or relief.

[1880] "Appropriate response" refers to responding or acting in the most appropriate way according to one's emotional state.

[1881] "Notification" is the act of sending information to the user or relatives to inform them.

[1882] A "virtual store" is a virtual store that offers products and services on an online platform.

[1883] "Product search" refers to the action or process of finding a specific product.

[1884] "Product suggestion" is the act of recommending appropriate products based on the user's needs and emotions.

[1885] "Advice from relatives" refers to advice or instructions sent by relatives, and is information that is displayed for the elderly person to refer to.

[1886] "Speech recognition" is a technology that analyzes speech and converts it into text or instructions.

[1887] A "voice command" is a command to give specific instructions by voice.

[1888] An "emergency" refers to a time-sensitive situation in which an older adult requires immediate attention.

[1889] "Fraud" is an illegal act of deceiving people through dishonest means to obtain money, goods, or information.

[1890] This invention is a system that supports elderly people's conversations and allows relatives to check on their condition remotely, and incorporates emotion recognition technology. The main components of this system are a terminal equipped with voice recognition, emotion recognition, and notification functions, a server that manages and processes various data, an interface that allows relatives to send messages and check on the elderly's condition remotely, and an emotion engine.

[1891] System configuration

[1892] 1. Device:

[1893] This device is used by the elderly and has voice recognition, emotion recognition, and notification functions. The device provides the elderly with information on the weather, news, and topics based on their hobbies and preferences. It also understands the elderly's voice commands and communicates with the server as needed.

[1894] 2. Server:

[1895] The server is responsible for managing and processing various data, analyzing voice and emotion data, providing emergency notifications, and suggesting products. It also collects voice samples from elderly people to improve the accuracy of voice recognition.

[1896] 3. Kinship Interface:

[1897] This is a web interface that allows relatives to send messages from remote locations and check on the status of the elderly. Relatives can input advice for the elderly, and the advice is sent to the elderly's device.

[1898] 4. Emotion Engine:

[1899] The emotion engine analyzes the voices of elderly people and identifies their emotional state, enabling it to provide appropriate responses and suggest products based on their emotions.

[1900] Program processing

[1901] The server analyzes the data using a speech recognition engine (Python's speech_recognition library), an emotion recognition engine (a customized EmotionRecognition class), and a notification system (AlertSystem class). When it receives voice input, it converts the speech into text and analyzes it. Based on the analysis results, it suggests topics and products appropriate for the elderly.

[1902] In the event of an emergency, the system recognizes the user's voice commands and immediately notifies the server, which then automatically issues an emergency notification based on the analysis results of the emotion engine and takes the necessary action.

[1903] Example

[1904] For example, consider the case of an elderly person ordering daily necessities from a virtual store.

[1905] Scenario: An elderly person wants to order tissues from a virtual store.

[1906] Example user:

[1907] "Looking for tissue paper"

[1908] "The weather is nice today, so I want to go for a walk."

[1909] Based on this, the system processes a program that follows the user's instructions, suggests appropriate products, and notifies relatives of the situation.

[1910] Example prompt sentence:

[1911] Scenario: Senior citizen orders an item from a virtual store.

[1912] User: "Find me some tissues."

[1913] The system recognizes your voice and presents a list of related products.

[1914] Emotion Engine: Recognizes the emotions of the elderly and displays the emotion of joy.

[1915] Server: Make appropriate product suggestions.

[1916] This system not only allows relatives to keep a watchful eye on the elderly while they enjoy everyday conversation, but also uses an emotion engine to respond appropriately to the elderly's emotional state.

[1917] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1918] Step 1:

[1919] The user inputs voice commands through the device, which then receives the elderly person's voice instructions and collects voice data. For example, the user might say, "Find me some tissues."

[1920] Step 2:

[1921] The device sends the collected voice data to a speech recognition engine (speech_recognition library) and converts it into text data. The converted text becomes "Look for tissue paper."

[1922] Step 3:

[1923] The device sends the converted text data to an emotion recognition engine (EmotionRecognition class) for emotion analysis. The emotion engine identifies the elderly person's emotions (e.g., joy, excitement) from the text data.

[1924] Step 4:

[1925] The device transmits text data and emotion data to a server, which receives the data and generates a list of candidate products to suggest appropriate products according to the user's emotion.

[1926] Step 5:

[1927] The server then sends the generated product candidate list back to the emotion engine, which selects the product that best matches the emotion. For example, a user who expresses joy might be offered a product with beautiful packaging.

[1928] Step 6:

[1929] The server sends information about the selected product to the terminal. The terminal notifies the user of this information by voice and suggests the product, such as "We suggest this product as tissue paper. Please check it out."

[1930] Step 7:

[1931] The user checks the products and selects or orders by voice. The terminal sends this voice data back to the voice recognition engine and converts it into text data.

[1932] Step 8:

[1933] The terminal sends the converted text data to the server for order processing, and the server communicates with the e-commerce site to process the order.

[1934] Step 9:

[1935] Once the order process is complete, the server sends the completion information to the terminal. The terminal then notifies the user by voice that the order has been completed. For example, it may say, "Your tissue order has been completed. It will arrive in two days."

[1936] Step 10:

[1937] Through the relative interface, the server notifies the relative of the order and the elderly person's emotional state. The relative can then check the elderly person's status through a web interface and send additional advice or instructions if necessary.

[1938] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1939] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1940] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1941] [Third embodiment]

[1942] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1943] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1944] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1945] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1946] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1947] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1948] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1949] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1950] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1951] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1952] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1953] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1954] This invention is a system that supports conversations between elderly people and enables their relatives living far away to check on their status. This system has functions to provide topics based on weather, news, and hobbies and preferences, to receive messages from relatives and notify them by voice, to recognize voice commands in emergencies and respond immediately, to notify relatives if there is no response from the elderly person for a certain period of time, and to prevent fraud by using voice analysis.

[1955] System Configuration

[1956] The system consists of the following main components:

[1957] 1. Terminal: A device used by the elderly that has voice recognition and notification functions. It provides the elderly with information about the weather, news, and topics based on their hobbies and preferences.

[1958] 2. Server: Manages and processes various data. Responsible for analyzing voice data and sending notifications in emergencies.

[1959] 3. Relatives' interface: A web interface for relatives to send messages remotely and check on the status of the elderly.

[1960] Program processing overview

[1961] 1. User registration and initial settings

[1962] User: Enters name, address, and contact details of relatives into the app or device, and provides a voice sample to the system during initial setup.

[1963] Terminal: Sends entered information and voice samples to the server.

[1964] Server: Stores this information in a database and updates the speech recognition model.

[1965] 2. Providing everyday conversation

[1966] Server: Generates the day's topics (weather, news, hobbies, etc.) based on the user's profile information and past conversation history.

[1967] Terminal: Provides the elderly with the topics generated at the specified time via audio.

[1968] Users: Enjoy conversations and talk to their devices.

[1969] Terminal: Recognizes the user's speech and sends it to the server.

[1970] Server: Generates an appropriate response and sends it back to the device.

[1971] Specific examples

[1972] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[1973] User: Yes, please.

[1974] Terminal: According to this article, roses grow best in the spring.

[1975] 3. Message function from relatives

[1976] Relatives: Enter and send messages via the web interface.

[1977] Server: Receives the message and sends it to the elderly person's device at the specified time.

[1978] Device: Receives voice notification of messages from relatives at the specified time.

[1979] User: Responds verbally if necessary.

[1980] Terminal: The response is sent to the server and the relatives are notified.

[1981] Specific examples

[1982] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[1983] Terminal: Have you taken your medicine yet?

[1984] User: Yes, I did.

[1985] Device: Your reply has been notified to your family.

[1986] The server sends a notification to the relative: Father replied that he had taken his medicine.

[1987] 4. Emergency Response

[1988] User: Says a voice command such as "Call an ambulance" or "Call ____."

[1989] Device: Recognizes voice commands and immediately sends them to the server.

[1990] Server: Receives emergency voice commands and automatically notifies necessary contacts (119 or relatives).

[1991] Specific examples

[1992] User: Call an ambulance

[1993] Terminal: Okay, I'll call 911.

[1994] An automatic call will be made to 119 via the server.

[1995] Server: An emergency has occurred. We will notify you to respond.

[1996] 5. Monitoring conversations and notifying relatives

[1997] Terminal: If there is no conversation for a certain period of time (e.g., 3 days), the server will be automatically notified.

[1998] Server: Receives the notification and notifies relatives.

[1999] Specific examples

[2000] (No conversation for 3 days)

[2001] Terminal: We haven't spoken in a while, I'll contact my family.

[2002] Server sends notification to relatives: Father hasn't spoken in 3 days, please check.

[2003] 6. Online shopping support

[2004] User: Use voice commands to order the items they need.

[2005] Terminal: Recognizes voice and sends orders to the server.

[2006] Server: Completes the purchase procedure and notifies the terminal that the order is complete.

[2007] Terminal: Notifies the user that the order is complete.

[2008] Specific examples

[2009] User: Order some tissues.

[2010] Terminal: Would you like to order one tissue?

[2011] User: Yes

[2012] Device: I ordered it and it will arrive in 2 days.

[2013] 7. Fraud Prevention Features

[2014] Terminal: Constantly monitors the user's conversation and sends the voice data to the server.

[2015] Server: Uses AI models to analyze conversations and detect potential fraud.

[2016] Server: If it determines that there is a high possibility of fraud, it sends a warning to the device.

[2017] Terminal: Provides audio fraud warnings to users.

[2018] Server: Notify relatives of possible fraud.

[2019] Specific examples

[2020] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[2021] Device: This is a potential scam message. Do you want to continue?

[2022] Server: Notify relatives: A potentially fraudulent conversation has been detected.

[2023] With the above configuration and processing, the present invention provides a system that allows elderly people to enjoy everyday conversations while their relatives can watch over them with peace of mind.

[2024] The processing flow will be explained below.

[2025] User registration and initial settings

[2026] Step 1:

[2027] User: Launches app or device, enters name, address, and contact details of next of kin. They are prompted to provide a voice sample.

[2028] Step 2:

[2029] Terminal: Sends entered information and voice samples to the server.

[2030] Step 3:

[2031] Server: Stores the received information in a database and updates the speech recognition model.

[2032] Providing everyday conversation

[2033] Step 1:

[2034] Server: Generates topics for the day based on the user's profile information and past conversation history. Sends the generated topics to the device.

[2035] Step 2:

[2036] Terminal: Provides the elderly with audio information about topics generated at the specified time.

[2037] Step 3:

[2038] User: Talk to the device about a topic.

[2039] Step 4:

[2040] Terminal: Recognizes the user's speech and sends it to the server.

[2041] Step 5:

[2042] Server: Generates an appropriate response and sends it back to the device.

[2043] Step 6:

[2044] Terminal: The response from the server is conveyed to the user by voice.

[2045] Message function from relatives

[2046] Step 1:

[2047] Relatives: Enter and send messages via the web interface.

[2048] Step 2:

[2049] Server: Receives messages and sends them to the terminal at the specified time.

[2050] Step 3:

[2051] Terminal: Notifies the elderly of messages from relatives via voice at the specified time.

[2052] Step 4:

[2053] User: Responds verbally if necessary.

[2054] Step 5:

[2055] Terminal: Sends the response to the server.

[2056] Step 6:

[2057] Server: Notifies the relatives of the response.

[2058] Emergency response

[2059] Step 1:

[2060] User: Issues an emergency voice command such as "Call an ambulance" or "Call ____."

[2061] Step 2:

[2062] Terminal: Recognizes voice commands and immediately notifies the server.

[2063] Step 3:

[2064] Server: Receives emergency voice commands and automatically notifies necessary contacts (119 or relatives).

[2065] Conversation monitoring and notification to relatives

[2066] Step 1:

[2067] Device: If there is no conversation for a certain period of time (e.g., 3 days), the server will be automatically notified.

[2068] Step 2:

[2069] Server: Receives the notification and notifies relatives.

[2070] Online shopping support

[2071] Step 1:

[2072] User: Use voice commands to order the items they need.

[2073] Step 2:

[2074] Terminal: Recognizes the voice and sends it to the server.

[2075] Step 3:

[2076] Server: Receives the order details and processes the purchase on the e-commerce site.

[2077] Step 4:

[2078] Server: After the purchase procedure is completed, send the completion information to the terminal.

[2079] Step 5:

[2080] Terminal: Notifies the user by voice that the order has been completed.

[2081] Fraud prevention features

[2082] Step 1:

[2083] Terminal: Constantly monitors the user's conversation and sends the voice data to the server.

[2084] Step 2:

[2085] Server: Uses AI models to analyze conversations and detect potential fraud.

[2086] Step 3:

[2087] Server: If it determines that there is a high possibility of fraud, it sends a warning to the device.

[2088] Step 4:

[2089] Terminal: Provides audio fraud warnings to users.

[2090] Step 5:

[2091] Server: Notify relatives of possible fraud.

[2092] Example 1

[2093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2094] In modern society, elderly people are often isolated, and situations in which they require support in daily life are increasing. It is also difficult for relatives living far away to understand the situation of elderly people, which creates the risk of delayed response in emergencies. Furthermore, elderly people are often exposed to risks such as fraud. There is a need for a system that addresses these issues and provides an environment in which elderly people can live with peace of mind.

[2095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2096] In the present invention, the server includes the following means, which makes it possible to:

[2097] A means of providing weather, news, and topics based on hobbies and preferences to seniors,

[2098] A means of receiving messages from relatives and notifying the elderly by voice;

[2099] A means to contact emergency contacts and relatives with simple voice commands,

[2100] A means of sending a notice to relatives if there is no response from the elderly person for a certain period of time;

[2101] A method for analyzing elderly people's conversations and issuing warnings in cases of possible fraud;

[2102] means for recognizing and transmitting voice input from the senior citizen to a server, which processes the data and provides an appropriate response to the senior citizen;

[2103] A way to order daily necessities online using voice commands,

[2104] a means for automatically taking appropriate emergency action when a voice command is issued; and

[2105] A method to automatically notify relatives if there is no communication for a certain period of time.

[2106] This will enable elderly people to enjoy everyday conversations and allow relatives to watch over them from a distance with peace of mind. It will also enable quick responses in emergencies and prevent fraud, improving the quality of life for the elderly.

[2107] "Elderly" refers to people over a certain age, who often require special support.

[2108] "Relative" refers to a person who has a close relationship with an older person based on blood or legal ties.

[2109] "Topics based on weather, news, and hobbies and preferences" refers to information that is likely to attract the individual user's interest and attention, and is a topic that enriches daily life.

[2110] "Messages" refer to messages or information sent by relatives to elderly people.

[2111] "Voice commands" refer to instructions that seniors can give using their voice to trigger specific actions.

[2112] "Emergency Contact" refers to contact information that should be used to contact you promptly if necessary.

[2113] "Speech recognition" refers to the technology of analyzing voice as a digital signal and converting it into text data, etc.

[2114] "Voice notification" refers to the function of converting specific information into voice and conveying it to the elderly.

[2115] "Potential fraud" refers to patterns of conversation or behavior intended to deceive seniors.

[2116] "Voice input" refers to voice data provided by a user to a system via a microphone or the like.

[2117] "Data processing" refers to the process by which the server analyzes the data it receives and generates an appropriate response or action.

[2118] "Online shopping" refers to the act of purchasing goods over the Internet.

[2119] "Daily necessities" refers to consumables and necessary products used in daily life.

[2120] "Emergency response" refers to measures taken to respond quickly when an unexpected situation occurs.

[2121] "Notification" refers to a message or signal intended to convey specific information to a target audience.

[2122] The present invention is a system that supports elderly people's conversations and allows their relatives living far away to check on their status. This system is composed of a terminal used by the elderly person, a server that manages and processes various data, and a web interface used by the relatives. Specific embodiments of the present invention will be described below.

[2123] 1. Main system components

[2124] The system consists of the following main components:

[2125] Terminal: A device used by seniors that has voice recognition and notification functions. It provides seniors with information about the weather, news, and topics based on their hobbies and preferences. It also has the ability to recognize voice commands in emergencies and respond immediately.

[2126] Server: Manages and processes various data. Responsible for analyzing voice data, providing emergency notifications, and detecting and warning fraud. The server uses a speech recognition engine (such as the Google Speech-to-Text API) and a natural language processing engine (such as OpenAI GPT-3).

[2127] Relatives' interface: A web interface that allows relatives to send messages remotely and check on the elderly's status. From here, relatives can input and check messages, and receive notifications in case of emergencies.

[2128] 2. Program Processing Overview

[2129] User registration and initial settings

[2130] First, the user enters their information (name, address, contact information for relatives) on the device or a dedicated app, and provides a voice sample during the initial setup. This information is sent from the device to the server, where it is stored in a database and the voice recognition model is updated, improving voice recognition accuracy.

[2131] Providing everyday conversation

[2132] The server generates the day's topics (weather, news, hobbies, etc.) based on the user's profile information and past conversation history. This information is collected using news APIs and weather forecast APIs. The generated topics are provided to the elderly via voice at the specified time via the device. When the user speaks, the device recognizes the voice and sends the text data to the server. The server uses a natural language processing engine to generate an appropriate response, which is sent to the device and played aloud.

[2133] Specific examples

[2134] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[2135] User: Yes, please.

[2136] Terminal: According to this article, roses grow best in the spring.

[2137] Message function from relatives

[2138] Relatives can input messages through a web interface, which are then received by the server. At the designated time, the message is sent to the elderly person's device via voice notification. When the elderly person responds by voice, the message is sent back to the server and notified to the relative.

[2139] Specific examples

[2140] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[2141] Terminal: Have you taken your medicine yet?

[2142] User: Yes, I did.

[2143] Device: Your reply has been notified to your family.

[2144] The server sends a notification to the relative: Father replied that he had taken his medicine.

[2145] Emergency response

[2146] When an elderly person issues an emergency voice command such as "call an ambulance" or "call XX," the device recognizes the command and immediately sends it to the server, which then receives it and automatically notifies the necessary contacts (119 or relatives).

[2147] Specific examples

[2148] User: Call an ambulance

[2149] Terminal: Okay, I'll call 911.

[2150] The server will automatically call 119.

[2151] Server: An emergency has occurred. We will notify you to respond.

[2152] Conversation monitoring and notification to relatives

[2153] If a device has not had a conversation for a certain period of time (e.g., 3 days), it will automatically notify the server, which will then notify the family.

[2154] Specific examples

[2155] (No conversation for 3 days)

[2156] Terminal: It's been a while since we last spoke, I'll contact my family.

[2157] Server sends notification to relatives: Father hasn't spoken in 3 days, please check.

[2158] Online shopping support

[2159] When an elderly person orders a product by voice, the device recognizes the voice and sends the order to the server, which then processes the purchase and notifies the device that the order is complete.

[2160] Specific examples

[2161] User: Order some tissues.

[2162] Terminal: Would you like to order one tissue?

[2163] User: Yes

[2164] Device: I ordered it and it will arrive in 2 days.

[2165] Fraud prevention features

[2166] The device constantly monitors the user's conversations and sends the audio data to the server. The server analyzes the conversation and sends a warning to the device if there is a possibility of fraud. The device notifies the user by voice and also notifies their relatives of the possibility of fraud.

[2167] Specific examples

[2168] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[2169] Device: This is a potential scam message. Do you want to continue?

[2170] Server: Notify relatives: A potentially fraudulent conversation has been detected.

[2171] With the above configuration and processing, the present invention provides a system that allows elderly people to enjoy everyday conversations while allowing relatives to watch over them from a remote location with peace of mind.

[2172] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2173] Step 1:

[2174] The user enters their name, address, contact details of their next of kin into the app or device and provides a voice sample.

[2175] Input: Name, address, contact details of relatives, voice sample

[2176] What happens: The user fills in a form on the device with details and records audio into the microphone.

[2177] Output: Input data and audio samples are saved to the device.

[2178] Step 2:

[2179] The device sends the entered information and voice samples to the server.

[2180] Input: User-entered information and voice samples

[2181] Specific operation: The device packs data into packets and sends them over the Internet to a server.

[2182] Output: Data received by the server

[2183] Step 3:

[2184] The server stores the received information in a database and updates the speech recognition model.

[2185] Input: User information and voice sample

[2186] What happens: The server inserts data into the database and updates the model using a speech recognition engine (e.g., Google Speech-to-Text API).

[2187] Output: Updated database and speech recognition model

[2188] Step 4:

[2189] The server generates the topic for the day based on the user's profile information and past conversation history.

[2190] Input: User profile information, past conversation history

[2191] Specific operation: The server obtains information using news APIs or weather forecast APIs and generates topics using a generative AI model (e.g., OpenAI GPT-3).

[2192] Output: Generated topics

[2193] Step 5:

[2194] The device provides the elderly with the topics generated at the specified time via voice.

[2195] Input: Generated topics

[2196] Specific operation: The device uses a speech synthesis engine (for example, Google Text-to-Speech API) to convert the topic into audio and play it back to the user.

[2197] Output: Elderly people listen to the topic

[2198] Step 6:

[2199] When the user speaks, the device recognizes the voice and sends the text data to the server.

[2200] Input: User voice input

[2201] Specific operation: The device converts the speech into text using a speech recognition engine and sends it to the server.

[2202] Output: Sends audio to the server as text data

[2203] Step 7:

[2204] The server generates an appropriate response and sends it to the terminal, which plays it aloud.

[2205] Input: Sends audio to the server as text data

[2206] Specific operation: The server generates a response using a natural language processing engine (e.g., OpenAI GPT-3) and sends it to the device. The device then converts the response into speech using a speech synthesis engine and plays it back to the user.

[2207] Output: The user hears the reply aloud

[2208] Step 8:

[2209] Relatives can enter messages through a web interface and send them to the server.

[2210] Input: Relative's text message

[2211] Specific actions: A relative logs into the web interface, types a message, and presses send.

[2212] Output: The server receives the message.

[2213] Step 9:

[2214] The server receives the message and sends it to the elderly person's device at the specified time.

[2215] Input: Message from relative

[2216] Specific operation: The server's schedule management system sends notification instructions to the terminal at the specified time.

[2217] Output: The device receives the message.

[2218] Step 10:

[2219] The device will notify you of messages from relatives via voice at the specified time.

[2220] Input: Message notification from the server

[2221] Specific operation: The device uses a speech synthesis engine to convert the message into voice and notify the user.

[2222] Output: User listens to the message

[2223] Step 11:

[2224] The user can respond by voice if necessary, and the terminal sends the content to the server.

[2225] Input: User's voice response

[2226] Specific operation: The device recognizes the voice and sends it to the server as text data.

[2227] Output: Response data is sent to the server

[2228] Step 12:

[2229] The server notifies the relatives of the received reply.

[2230] Input: User response data

[2231] What happens: The server sends emails and SMS to the relatives' contacts.

[2232] Output: The relative receives the reply

[2233] Step 13:

[2234] When an elderly person issues an emergency voice command (e.g., "Call an ambulance"), the device recognizes it and sends it to the server.

[2235] Input: Emergency voice command

[2236] Specific operation: The device analyzes the emergency command using a voice recognition engine and sends it to the server.

[2237] Output: Server receives urgent command

[2238] Step 14:

[2239] The server receives emergency voice commands and automatically notifies the necessary contacts (e.g., 119 or relatives).

[2240] Input: Emergency voice command

[2241] Specific operation: The server automatically notifies the emergency contact via the phone API or message API.

[2242] Output: Emergency is handled

[2243] Step 15:

[2244] If a device has not had a conversation for a certain period of time (e.g., 3 days), it will automatically notify the server.

[2245] Input: Conversation status data

[2246] Specific operation: The device monitors conversation events using an internal counter and notifies the server if there is no update within a certain period of time.

[2247] Output: The server receives the notification

[2248] Step 16:

[2249] The server receives the notification and notifies the relatives.

[2250] Input: Notification from device

[2251] What happens: The server sends emails and SMS to the relatives' contacts.

[2252] Output: Relatives receive notification

[2253] Step 17:

[2254] When an elderly person orders a product by voice, the terminal recognizes the voice and sends the order to the server.

[2255] Input: Voice command to order product

[2256] Specific operation: The device converts the speech into text using a speech recognition engine and sends it to the server.

[2257] Output: The server receives the order.

[2258] Step 18:

[2259] The server completes the purchase process and notifies the terminal that the order has been completed.

[2260] Input: Product order data

[2261] What happens: The server uses the e-commerce API to complete the purchase and confirm the order.

[2262] Output: The terminal will be notified that the order has been completed.

[2263] Step 19:

[2264] The terminal notifies the user that the order is complete.

[2265] Input: Order completion notification

[2266] What happens: The device uses a speech synthesis engine to announce the completion of the order.

[2267] Output: User hears order completed

[2268] Step 20:

[2269] The device constantly monitors the user's conversation and transmits the voice data to the server.

[2270] Input: Conversational audio data

[2271] Specific operation: The device records audio using an audio capture device and streams it to the server.

[2272] Output: The server receives the audio data.

[2273] Step 21:

[2274] The server uses AI models to analyze the conversation and sends a warning to the device if there is a possibility of fraud.

[2275] Input: Conversational audio data

[2276] What it does: The server analyzes the data using natural language processing models and runs algorithms to detect fraudulent patterns.

[2277] Output: Fraud alert data

[2278] Step 22:

[2279] The server receives the fraud warning data and sends it to the terminal, which then notifies the user of the fraud warning by voice.

[2280] Input: Fraud Alert Data

[2281] What happens: The server sends a fraud warning to the device, which then plays it back to the user using its speech synthesis engine.

[2282] Output: User hears fraud warning

[2283] Step 23:

[2284] The server also notifies relatives of possible fraud.

[2285] Input: Fraud Alert Data

[2286] What happens: The server sends emails and SMS to the relatives' contacts.

[2287] Output: Relative receives fraud alert

[2288] (Application example 1)

[2289] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2290] Systems already exist that support the daily conversations of elderly people and allow relatives to check their status remotely, but these systems lack sufficient dialogue generation capabilities and flexible means for relatives to send messages. Furthermore, their emergency response and fraud prevention features for elderly people are often incomplete. Therefore, there is a need for systems with more advanced conversation support and security features that allow elderly people to live their daily lives with peace of mind.

[2291] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2292] In this invention, the server is a system that supports conversations between elderly people and allows relatives to check their status from a remote location, and includes: means for providing the elderly with topics based on the weather, news, and hobbies and preferences; means for receiving messages from relatives and notifying the elderly by voice; means for contacting an ambulance or relatives using simple voice commands; means for notifying relatives if there is no response from the elderly for a certain period of time; means for analyzing the elderly's conversations and issuing a warning if there is a possibility of fraud; means for generating conversation content using a generative artificial intelligence model and providing daily conversation support; and means for relatives to easily send messages using prompt sentences. This provides more advanced conversation support, faster emergency response, and fraud prevention functions than conventional systems, allowing elderly people to live their lives with peace of mind.

[2293] The "elderly conversation support system" is a system that supports the everyday conversations of the elderly and allows relatives in remote locations to check on their status.

[2294] The "topic provision means" is a function that provides elderly people with topics generated based on weather, news, and hobbies and preferences via voice.

[2295] The "message notification means" is a function that receives messages sent by relatives and notifies the elderly person by voice.

[2296] The "emergency contact method" is a function that allows elderly people to contact an ambulance or relatives in an emergency by issuing simple voice commands.

[2297] The "no response notification method" is a function that automatically sends a notification to relatives if there is no response from the elderly person for a certain period of time.

[2298] The "fraud warning feature" is a function that analyzes the elderly person's conversation and issues an audio warning if there is a possibility of fraud.

[2299] A "generative artificial intelligence model" is an artificial intelligence technology that generates appropriate conversation content based on data such as a user's profile information and past conversation history.

[2300] A "prompt sentence" is an instruction sentence that is input to a generative artificial intelligence model, allowing relatives to easily send messages.

[2301] The present invention is a system that supports conversations between elderly people and allows their remote relatives to check on their status. This system includes a means for providing topics based on weather, news, and hobbies and preferences, a means for receiving messages from relatives and notifying them by voice, a means for recognizing and immediately responding to voice commands in emergencies, a means for notifying relatives if there is no response from the elderly person for a certain period of time, a means for preventing fraud through voice analysis, a means for generating conversation content using a generative artificial intelligence model to support daily conversations, and a means for relatives to easily send messages using prompt sentences.

[2302] System Configuration

[2303] This system consists of the following main components:

[2304] 1. Device:

[2305] This device, used by seniors, has voice recognition and notification functions, and provides them with information about the weather, news, and topics based on their hobbies and preferences.

[2306] 2. Server:

[2307] Manages and processes various data. Analyzes voice data and sends emergency notifications. Generates conversation content using generative artificial intelligence models.

[2308] 3. Kindred Interface:

[2309] A web interface that allows relatives to remotely send messages or check on the status of elderly people. Messages can be sent using prompts.

[2310] Functions and Processing

[2311] User registration and initial settings

[2312] The user (elderly person) enters their name, address, and contact information of their relatives into the app or device, and provides a voice sample to the system during initial setup. The device then sends the entered information and voice sample to the server, which stores this information in a database and updates the voice recognition model.

[2313] Providing everyday conversation

[2314] The server generates the topic of the day (weather, news, hobbies, etc.) based on the user's profile information and past conversation history. The device provides the generated topic to the elderly via voice at the specified time. When the user enjoys a conversation and speaks to the device, the device recognizes the user's speech and sends it to the server. The server generates an appropriate response and sends it back to the device.

[2315] Examples:

[2316] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[2317] User: Yes, please.

[2318] Terminal: According to this article, roses grow best in the spring.

[2319] Message function from relatives

[2320] The relative inputs and sends a message via a web interface. The server receives the message and sends it to the elderly person's device at the specified time. The device then notifies the relative of the message by voice at the specified time, and the user can respond by voice as needed. The device then sends the response to the server, which notifies the relative.

[2321] Examples:

[2322] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[2323] Terminal: Have you taken your medicine yet?

[2324] User: Yes, I did.

[2325] Device: Your reply has been notified to your family.

[2326] The server sends a notification to the relative: Father replied that he had taken his medicine.

[2327] Emergency response

[2328] The user issues a voice command such as "Call an ambulance" or "Call XX." The device recognizes the voice command and immediately sends it to the server. The server receives the emergency voice command and automatically notifies the necessary contacts (119 or relatives).

[2329] Examples:

[2330] User: Call an ambulance

[2331] Terminal: Okay, I'll call 911.

[2332] An automatic call will be made to 119 via the server.

[2333] Server: An emergency has occurred. We will notify you to respond.

[2334] Notification function when there is no response for a certain period of time

[2335] If there is no conversation for a certain period of time (e.g., 3 days), the device automatically notifies the server. The server receives the notification and notifies the relatives.

[2336] Examples:

[2337] If there is no conversation for three days

[2338] Terminal: We haven't spoken in a while, I'll contact my family.

[2339] Server sends notification to relatives: Father hasn't spoken in 3 days, please check.

[2340] Fraud prevention features

[2341] The device constantly monitors the user's conversation and sends the audio data to the server. The server uses a generative artificial intelligence model to analyze the conversation and detect possible fraud. If it determines that there is a high possibility of fraud, it sends a warning to the device and notifies the user of the fraud warning via voice. At the same time, the server notifies relatives of the possibility of fraud.

[2342] Examples:

[2343] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[2344] Device: This is a potential scam message. Do you want to continue?

[2345] Server: Notify relatives: A potentially fraudulent conversation has been detected.

[2346] With the above configuration and processing, the present invention provides a system that allows elderly people to enjoy everyday conversations while their relatives can watch over them with peace of mind.

[2347] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2348] Step 1:

[2349] User registration and initial settings

[2350] The user (elderly person) uses an input device (terminal) to enter basic information such as their name, address, and contact details of their relatives. In addition, they provide several voice samples during the initial setup. The terminal then sends this information and the voice samples to the server, which stores the received information in a database and updates the voice recognition model.

[2351] Input: Name, address, contact details of relatives, voice sample

[2352] Output: Updated speech recognition model, user information stored in the database

[2353] Step 2:

[2354] Providing everyday conversation

[2355] The server uses a generative artificial intelligence model to generate the day's topics (weather, news, hobbies, etc.) based on the user's profile information and past conversation history. The generated topics are sent from the server to the device, which then provides them to the elderly via voice at the specified time.

[2356] Input: User profile information, past conversation history

[2357] Output: Generated topics, information provided in audio

[2358] Specific operation: The server generates a topic called "It's forecast to be sunny," and the device announces "It's forecast to be sunny today" by voice.

[2359] Step 3:

[2360] Message function from relatives

[2361] Relatives use a web interface to input and send messages. The server receives the messages and sends them to the elderly person's device at the specified time. The device then notifies the elderly person of the message by voice at the specified time. The elderly person can then respond by voice if necessary, and the response is sent from the device to the server, where it is then notified to the relative.

[2362] Input: Message from relative, user's voice response

[2363] Output: Voice notification to the elderly, notification of the response to relatives

[2364] Specific operation: A relative sends a message saying, "Please make sure you have taken your medicine today," and the device notifies the user by voice, "Have you taken your medicine yet?", to which the user replies, "Yes, I have."

[2365] Step 4:

[2366] Emergency response

[2367] When a user issues a voice command such as "call an ambulance" or "call XX," the device recognizes the command and immediately sends it to the server. The server receives the emergency voice command and automatically notifies the necessary contacts (such as 119 or relatives).

[2368] Input: Senior emergency voice command

[2369] Output: Automatic call to emergency contacts

[2370] Specific operation: The user says "call an ambulance," the device recognizes the voice and sends it to the server, which then automatically calls 119.

[2371] Step 5:

[2372] Notification function when there is no response for a certain period of time

[2373] If the device does not receive a response from the elderly person for a certain period of time (for example, three days), it automatically sends a notification to the server, which then receives the notification and sends it to the elderly person's relatives.

[2374] Input: No response for a certain period of time

[2375] Output: No response notification from the server to relatives

[2376] Specific operation: If there is no response from the elderly person for three days, the device will notify the family, "We haven't heard from you for a long time. We will contact your family," and the server will notify the relative, "Your father has not spoken for three days. Please check."

[2377] Step 6:

[2378] Fraud prevention features

[2379] The device constantly monitors the user's conversation and sends the audio data to the server. The server uses a generative artificial intelligence model to analyze the conversation and detect possible fraud. If it determines that there is a high possibility of fraud, it sends a warning to the device and notifies the user via audio. At the same time, the server notifies relatives of the possibility of fraud.

[2380] Input: Elderly people's conversation

[2381] Output: Audio notification of fraud warning, fraud notification to relatives

[2382] Specific operation: If an unknown caller says, "You have won 1 million yen," the server detects the possibility of fraud, the device warns the user with a voice message saying, "This is a potentially fraudulent message," and the server notifies relatives that, "A potentially fraudulent conversation has been detected."

[2383] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2384] This invention combines a system that supports conversations between elderly people and allows their relatives in remote locations to check their status with an emotion engine that recognizes the user's emotions. This system provides topics based on weather, news, and hobbies and preferences, receives messages from relatives and notifies them by voice, recognizes voice commands in emergencies and responds immediately, notifies relatives if there is no response from the elderly person for a certain period of time, and prevents fraud through voice analysis. It is also possible to use the emotion engine to analyze the elderly person's emotions and provide appropriate responses or notifications.

[2385] System Configuration

[2386] The system consists of the following main components:

[2387] 1. Terminal: A device used by the elderly that has voice recognition, emotion recognition, and notification functions. It provides the elderly with information about the weather, news, and topics based on their hobbies and preferences via voice.

[2388] 2. Server: Manages and processes various data, analyzes voice and emotion data, and sends notifications in the event of an emergency.

[2389] 3. Relatives' interface: A web interface for relatives to send messages remotely and check on the status of the elderly.

[2390] 4. Emotion engine: It has the ability to analyze the user's voice and identify their emotional state.

[2391] Program processing overview

[2392] 1. User registration and initial settings

[2393] User: Enters name, address, and contact details of relatives into the app or device, and provides a voice sample during initial setup.

[2394] Terminal: Sends entered information and voice samples to the server.

[2395] Server: This information is stored in a database and the speech recognition model and emotion recognition model are updated.

[2396] 2. Providing everyday conversation

[2397] Server: Generates topics for the day based on the user's profile information and past conversation history. Sends the generated topics to the device.

[2398] Terminal: Provides the elderly with audio information about topics generated at the specified time.

[2399] Emotion engine: Analyzes the elderly person's responses and identifies their emotional state (e.g., joy, sadness, anger, etc.).

[2400] User: Talk to the device about a topic.

[2401] Terminal: Recognizes the user's speech and sends it to the server.

[2402] Server: Generates an appropriate response and sends it back to the device. Based on the analysis results of the emotion engine, an appropriate response is given according to the emotion.

[2403] Specific examples

[2404] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[2405] User: Yes, please. (Shows delight)

[2406] Emotion engine: Analyzes user responses as emotions of joy.

[2407] Server: Generates a response based on the emotion of joy.

[2408] Terminal: According to this article, roses grow best in the spring. That's so lovely!

[2409] 3. Message function from relatives

[2410] Relatives: Enter and send messages via the web interface.

[2411] Server: Receives the message and sends it to the elderly person's device at the specified time.

[2412] Device: Receives voice notification of messages from relatives at the specified time.

[2413] User: Responds verbally if necessary.

[2414] Emotion engine: Analyzes the user's responses and identifies their emotional state.

[2415] Terminal: Sends the response content and emotional state to the server.

[2416] Server: Notifies relatives of response and emotional state.

[2417] Specific examples

[2418] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[2419] Terminal: Have you taken your medicine yet?

[2420] User: Yes, I did. (Shows relief)

[2421] Emotion engine: Analyzes user responses as emotions of relief.

[2422] Server: Notify relatives of response along with emotional state of relief.

[2423] Family Interface: Father has taken his medicine. He seems relieved.

[2424] 4. Emergency Response

[2425] User: Issues an emergency voice command such as "Call an ambulance" or "Call ____."

[2426] Terminal: Recognizes voice commands and immediately notifies the server.

[2427] Emotion engine: Analyzes emotions (e.g., surprise, fear, etc.) during emergencies.

[2428] Server: Receives emergency voice commands and emotional state, and automatically notifies necessary contacts (119 or relatives).

[2429] Specific examples

[2430] User: Call an ambulance. (Shows fear)

[2431] Terminal: Okay, I'll call 911.

[2432] Emotion engine: Analyzes user responses as emotions of fear.

[2433] Server: Automatically calls 119 and notifies relatives along with the emotional state of fear.

[2434] Kinship Interface: An emergency has occurred. They appear frightened.

[2435] 5. Monitoring conversations and notifying relatives

[2436] Terminal: If there is no conversation for a certain period of time (e.g., 3 days), the server will be automatically notified.

[2437] Server: Receives the notification and notifies relatives.

[2438] Specific examples

[2439] (No conversation for 3 days)

[2440] Terminal: We haven't spoken in a while, I'll contact my family.

[2441] Server: Send notification to relatives.

[2442] Kinship Interface: Father hasn't spoken in 3 days. Please check.

[2443] 6. Online shopping support

[2444] User: Use voice commands to order the items they need.

[2445] Terminal: Recognizes voice and sends orders to the server.

[2446] Server: Receives the order details and processes the purchase on the e-commerce site.

[2447] Server: After the purchase procedure is completed, send the completion information to the terminal.

[2448] Terminal: A voice message is sent to the user to notify them that the order has been completed.

[2449] Specific examples

[2450] User: Order some tissues.

[2451] Terminal: Would you like to order one tissue?

[2452] User: Yes

[2453] Device: I ordered it and it will arrive in 2 days.

[2454] 7. Fraud Prevention Features

[2455] Terminal: Constantly monitors the user's conversation and sends the voice data to the server.

[2456] Server: Uses AI models to analyze conversations and detect potential fraud.

[2457] Emotion Engine: Analyzes the user's emotional state and immediately warns if there is a high possibility of fraud.

[2458] Server: If it determines that there is a high possibility of fraud, it sends a warning along with the emotional state to the device.

[2459] Device: Provides audio fraud warnings along with emotional state.

[2460] Server: Notifies relatives of possible fraud and emotional state.

[2461] Specific examples

[2462] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[2463] Device: This is a potential scam message. Do you want to continue?

[2464] Emotion Engine: Analyzes the user's confused emotional state.

[2465] Server: High likelihood of fraud, along with a confused emotional state, sends out a warning.

[2466] Kinship Interface: A potentially fraudulent conversation has been detected. Your father appears confused.

[2467] The above configuration and processing not only allows elderly people to enjoy everyday conversations and allows relatives to watch over them with peace of mind, but also makes it possible to provide a system that uses an emotion engine to respond appropriately to the emotional state of the elderly.

[2468] The processing flow will be explained below.

[2469] User registration and initial settings

[2470] Step 1:

[2471] User: Launches the app or device and enters initial information such as name, address, and contact details of next of kin. They are then prompted to provide a voice sample.

[2472] Step 2:

[2473] Terminal: Sends entered information and voice samples to the server.

[2474] Step 3:

[2475] Server: Stores the received information in a database and updates the speech recognition model and emotion recognition model.

[2476] Providing everyday conversation

[2477] Step 1:

[2478] Server: Generates topics for the day (weather, news, hobbies, etc.) based on the user's profile information and past conversation history. Sends the generated topics to the device.

[2479] Step 2:

[2480] Terminal: Provides the elderly with audio information about topics generated at the specified time.

[2481] Step 3:

[2482] Emotion engine: Analyzes the elderly person's responses and identifies their emotional state.

[2483] Step 4:

[2484] User: Talk to the device about a topic.

[2485] Step 5:

[2486] Terminal: Recognizes the user's speech and sends it to the server.

[2487] Step 6:

[2488] Server: Generates an appropriate response and sends a response to the device according to the emotion based on the analysis results of the emotion engine.

[2489] Step 7:

[2490] Terminal: Provides the user with a voice response from the server.

[2491] Specific examples

[2492] Terminal: Good morning. It's sunny today. I have a new article about gardening. Would you like to take a look?

[2493] User: Yes, please. (Shows delight)

[2494] Emotion engine: Analyzes user responses as emotions of joy.

[2495] Server: Generates a response based on the emotion of joy.

[2496] Terminal: According to this article, roses grow best in the spring. That's so lovely!

[2497] Message function from relatives

[2498] Step 1:

[2499] Relatives: Enter and send messages via the web interface.

[2500] Step 2:

[2501] Server: Receives the message and sends it to the elderly person's device at the specified time.

[2502] Step 3:

[2503] Terminal: Notifies the elderly of messages from relatives via voice at the specified time.

[2504] Step 4:

[2505] User: Responds verbally if necessary.

[2506] Step 5:

[2507] Emotion engine: Analyzes the user's responses and identifies their emotional state.

[2508] Step 6:

[2509] Terminal: Sends the response content and emotional state to the server.

[2510] Step 7:

[2511] Server: Notifies relatives of response and emotional state.

[2512] Specific examples

[2513] A relative sends a message via a web interface saying, "Please make sure you took your medicine today."

[2514] Terminal: Have you taken your medicine yet?

[2515] User: Yes, I did. (Shows relief)

[2516] Emotion engine: Analyzes user responses as emotions of relief.

[2517] Server: Notify relatives of response along with emotional state of relief.

[2518] Family Interface: Father has taken his medicine. He seems relieved.

[2519] Emergency response

[2520] Step 1:

[2521] User: Issues an emergency voice command such as "Call an ambulance" or "Call ____."

[2522] Step 2:

[2523] Terminal: Recognizes voice commands and immediately notifies the server.

[2524] Step 3:

[2525] Emotion engine: Analyzes emotions (e.g., surprise, fear, etc.) during emergencies.

[2526] Step 4:

[2527] Server: Receives emergency voice commands and emotional state, and automatically notifies necessary contacts (119 or relatives).

[2528] Step 5:

[2529] Server: Sends notification completion information along with the emotional state to the terminal.

[2530] Step 6:

[2531] Terminal: Notifies the elderly person by voice when the report is complete.

[2532] Specific examples

[2533] User: Call an ambulance. (Shows fear)

[2534] Terminal: Okay, I'll call 911.

[2535] Emotion engine: Analyzes user responses as emotions of fear.

[2536] Server: Automatically calls 119 and notifies relatives along with the emotional state of fear.

[2537] Kinship Interface: An emergency has occurred. They appear frightened.

[2538] Conversation monitoring and notification to relatives

[2539] Step 1:

[2540] Terminal: If there is no conversation for a certain period of time (e.g., 3 days), the server will be automatically notified.

[2541] Step 2:

[2542] Server: Receives the notification and notifies relatives.

[2543] Specific examples

[2544] (No conversation for 3 days)

[2545] Terminal: We haven't spoken in a while, I'll contact my family.

[2546] Server: Send notification to relatives.

[2547] Kinship Interface: Father hasn't spoken in 3 days. Please check.

[2548] Online shopping support

[2549] Step 1:

[2550] User: Use voice commands to order the items they need.

[2551] Step 2:

[2552] Terminal: Recognizes the voice and sends it to the server.

[2553] Step 3:

[2554] Server: Receives the order details and processes the purchase on the e-commerce site.

[2555] Step 4:

[2556] Server: After the purchase procedure is completed, send the completion information to the terminal.

[2557] Step 5:

[2558] Terminal: A voice message is sent to the user to notify them that the order has been completed.

[2559] Specific examples

[2560] User: Order some tissues.

[2561] Terminal: Would you like to order one tissue?

[2562] User: Yes

[2563] Device: I ordered it and it will arrive in 2 days.

[2564] Fraud prevention features

[2565] Step 1:

[2566] Terminal: Constantly monitors the user's conversation and sends the voice data to the server.

[2567] Step 2:

[2568] Emotion Engine: Analyzes the user's emotional state.

[2569] Step 3:

[2570] Server: Uses AI models to analyze conversations and detect potential fraud, taking into account the results of the sentiment engine.

[2571] Step 4:

[2572] Server: If it determines that there is a high possibility of fraud, it sends a warning along with the emotional state to the device.

[2573] Step 5:

[2574] Device: Provides audio fraud warnings along with emotional state.

[2575] Step 6:

[2576] Server: Notifies relatives of possible fraud and emotional state.

[2577] Specific examples

[2578] Unknown caller: Congratulations! You have won 1 million yen. Please transfer 10,000 yen in advance as a handling fee.

[2579] Device: This is a potential scam message. Do you want to continue?

[2580] Emotion Engine: Analyzes the user's confused emotional state.

[2581] Server: High likelihood of fraud, along with a confused emotional state, sends out a warning.

[2582] Kinship Interface: A potentially fraudulent conversation has been detected. Your father appears confused.

[2583] The above configuration and processing not only allows elderly people to enjoy everyday conversations and allows relatives to watch over them with peace of mind, but also makes it possible to provide a system that uses an emotion engine to respond appropriately to the emotional state of the elderly.

[2584] Example 2

[2585] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2586] Elderly people often feel lonely and anxious in their daily lives. In particular, when relatives live far away, it is difficult for them to keep track of the elderly's condition, increasing safety and health risks. There are also situations where a quick response is required in an emergency, and there are risks such as fraud. There is a need for a system that can solve these issues and support the lives of the elderly while allowing relatives to keep an eye on them with peace of mind.

[2587] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for supporting the elderly person's conversation and allowing relatives to check the elderly person's status from a remote location; means for providing topics based on the weather, news, and hobbies and preferences; means for receiving messages from relatives and notifying them by voice; means for contacting emergency contact information and relatives with simple voice commands; means for sending a notification to relatives if there is no response for a certain period of time; means for analyzing the conversation and issuing a warning if there is a possibility of fraud; means for analyzing the elderly person's voice and identifying their emotional state; and means for providing appropriate responses and notifications based on the emotional state. This supports the elderly person's daily life and enables relatives to know the elderly person's status from a remote location in real time.

[2588] "Elderly" refers to an individual who is elderly and requires assistance with daily living.

[2589] "Relative" refers to a person who is related to an older person by blood or marriage, or who has the responsibility to care for an older person.

[2590] A "conversation support system" is a system that supports the elderly in their everyday conversations and allows relatives to check on their status from a remote location.

[2591] "Topic provision means" refers to a means of providing topics to elderly people via audio based on the weather, news, hobbies, and preferences.

[2592] "Message notification means" refers to a means of notifying elderly people by voice of messages received from relatives.

[2593] "Emergency contact methods" refer to methods that allow seniors to contact emergency contacts and relatives using simple voice commands.

[2594] "Response monitoring measures" refer to measures that send notifications to relatives if there is no response from the elderly person for a certain period of time.

[2595] "Fraud warning measures" refer to measures that analyze elderly people's conversations and issue warnings if there is a possibility of fraud.

[2596] "Voice analysis means" refers to a means for analyzing the voice of an elderly person and identifying their emotional state.

[2597] "Emotional response measures" refer to measures that provide appropriate responses or notifications based on the emotional state of the elderly person.

[2598] "Online shopping support means" refers to a means for ordering daily necessities using voice commands.

[2599] This invention relates to a system that supports conversations between elderly people and allows their remote relatives to check their status in real time. This system provides topics based on weather, news, and hobbies and preferences, and includes functions for notifying messages from relatives via voice, recognizing and responding to voice commands in emergencies, notifying relatives if there is no response from the elderly person for a certain period of time, and preventing fraud through voice analysis. It also uses an emotion engine to analyze the emotions of the elderly person and provide appropriate responses and notifications.

[2600] The system consists of the following main components:

[2601] 1. Terminal: A device used by the elderly that has voice recognition, emotion recognition, and notification functions. It provides the elderly with information about the weather, news, and topics based on their hobbies and preferences via voice.

[2602] 2. Server: Manages and processes various data, analyzes voice and emotion data, and sends notifications in the event of an emergency.

[2603] 3. Relatives' interface: A web interface for relatives to send messages and check on the status of the elderly from remote locations.

[2604] 4. Emotion engine: It has the ability to analyze the user's voice and identify their emotional state.

[2605] Hardware and Software Use Cases

[2606] 1. Devices: Devices can be tablet devices or smart speakers (e.g., Amazon Echo, Google Home) with internet connectivity, allowing seniors to easily initiate conversations and respond when presented with a topic.

[2607] 2. Server: For the server, we use high-performance cloud services (e.g., Amazon Web Services, Google Cloud Platform), which enable large-volume data processing and real-time responses.

[2608] 3. Emotion Engine: Using Google Cloud Speech-to-Text API and IBM Watson Tone Analyzer, analyze the elderly person's voice to identify their emotional state.

[2609] Detailed System Description

[2610] User registration and initial settings

[2611] Users enter their name, address, and contact information for relatives into the app or their device, and provide a voice sample, which allows the system to register the user's voice and basic information in a database to improve the accuracy of voice and emotion recognition.

[2612] Providing everyday conversation

[2613] The server uses a generative AI model to generate appropriate topics based on the user's profile information and past conversation history, and sends them to the device. The device then provides the topics to the elderly via voice at the specified time. For example, it provides information in the form of, "Good morning. It's forecast to be sunny today. We have a new article about gardening. Would you like to take a look?"

[2614] Message function from relatives

[2615] When a relative sends a message via a web interface, the server receives it and sends it to the device at the specified time. For example, a message such as "Please make sure you have taken your medicine today" can be sent, and the device will notify the elderly person by voice.

[2616] Emergency response

[2617] When a user issues a simple voice command such as "call an ambulance" or "call XX," the device recognizes it and immediately notifies the server, which then automatically notifies the necessary contacts (e.g., 119 or relatives), enabling a prompt response.

[2618] Conversation monitoring and notification to relatives

[2619] If the device has not spoken for a certain period of time (e.g., three days), it notifies the server, and the server sends a notification to the relatives informing them of the situation. For example, a message such as "Your father has not spoken for three days. Please check." will be sent to the relatives.

[2620] Online shopping support

[2621] When a user orders everyday items by voice, the device recognizes the order and sends it to the server, which then processes the purchase through the e-commerce site and notifies the user that the order is complete.

[2622] Fraud prevention features

[2623] The device constantly monitors the user's conversation and sends the audio data to the server, which then uses an AI model to analyze the conversation. If there is a high possibility of fraud, the emotion engine analyzes the user's confusion or anxiety and immediately issues a fraud warning.

[2624] The system will support the elderly in their daily lives, allowing relatives to remotely monitor their condition in real time, and will also use an emotion engine to enable more appropriate responses.

[2625] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2626] Program processing flow

[2627] User registration and initial settings

[2628] Step 1:

[2629] The user enters their name, address, and contact details of their next of kin into the app or device and provides a voice sample.

[2630] Input: User's personal information (name, address, contact details of relatives) and voice sample

[2631] Output: The input data is saved on the device and ready to be sent to the next step.

[2632] Specific behavior:

[2633] Users follow the instructions on the app or device's initial setup screen, enter the required information, and provide a voice sample, which is recorded through the device's microphone.

[2634] Step 2:

[2635] The device sends the entered information and voice samples to the server.

[2636] Input: User personal information and voice sample (output of step 1)

[2637] Output: Sending confirmation and server receiving status

[2638] Specific behavior:

[2639] The terminal encrypts the input data and sends it securely to the server, after which the server confirms receipt of the data and returns a receipt confirmation message to the terminal.

[2640] Step 3:

[2641] The server stores this information in a database and updates the speech recognition model and emotion recognition model.

[2642] Input: User personal information and voice sample (output of step 2)

[2643] Output: Database update status and model update completion

[2644] Specific behavior:

[2645] The server stores the received user information in a database and uses the voice samples to train speech and emotion recognition models.

[2646] Providing everyday conversation

[2647] Step 1:

[2648] The server generates the topic of the day based on the user's profile information and past conversation history.

[2649] Input: User profile information and past conversation history

[2650] Output: Generated topic text data

[2651] Specific behavior:

[2652] The server uses a generative AI model to generate appropriate topics based on the user's hobbies, preferences, and past conversation history, such as weather, news, and hobby articles.

[2653] Step 2:

[2654] The device provides the elderly with audio topics generated at the specified time.

[2655] Input: Generated topic text data (output of Step 1)

[2656] Output: Audio notification completion status

[2657] Specific behavior:

[2658] The device will provide a topic by voice at a set time (e.g., every morning at 9:00). A speech synthesis engine will convert the text data into voice and notify the elderly.

[2659] Step 3:

[2660] The user speaks to the terminal about a topic.

[2661] Input: User's voice response

[2662] Output: Recorded audio data

[2663] Specific behavior:

[2664] The user speaks their thoughts or questions about the topics provided by the device, and the device's microphone records the voice.

[2665] Step 4:

[2666] The terminal recognizes what the user says and sends it to the server.

[2667] Input: User's voice data (output of Step 3)

[2668] Output: Text converted data and transmission completion status

[2669] Specific behavior:

[2670] The terminal uses a speech recognition engine to convert the user's voice into text data and transmits it to the server.

[2671] Step 5:

[2672] The server generates an appropriate response and sends it back to the terminal.

[2673] Input: Text data (output of step 4)

[2674] Output: Generated response text data

[2675] Specific behavior:

[2676] The server uses a generative AI model to generate appropriate responses to the user's text data and sends them to the device. An emotion engine also works in parallel to analyze the user's emotional state.

[2677] Step 6:

[2678] The terminal notifies the user of the response from the server by voice.

[2679] Input: Generated response text data (output of Step 5)

[2680] Output: Audio notification completion status

[2681] Specific behavior:

[2682] The terminal uses a speech synthesis engine to convert the response text data into speech and notify the user.

[2683] Message function from relatives

[2684] Step 1:

[2685] Relatives can enter and send messages via a web interface.

[2686] Input: Relative's text message

[2687] Output: Transmission complete status

[2688] Specific behavior:

[2689] The relative logs into the web interface, enters a message, and presses the send button, which sends the message to the server.

[2690] Step 2:

[2691] The server receives the message and sends it to the elderly person's device at the specified time.

[2692] Input: Relative's text message (output of step 1)

[2693] Output: Confirmation of sending and notification to the terminal that it is ready

[2694] Specific behavior:

[2695] The server schedules and manages the received messages, sending them to the elderly person's device at the specified time.

[2696] Step 3:

[2697] The device will notify you of messages from relatives via voice at the specified time.

[2698] Input: Message text data

[2699] Output: Audio notification completion status

[2700] Specific behavior:

[2701] At the set time, the device uses a voice synthesis engine to convert the relative's message into voice and notify the elderly person.

[2702] Step 4:

[2703] The user responds verbally as needed.

[2704] Input: User's voice reply

[2705] Output: Recorded audio data

[2706] Specific behavior:

[2707] The user responds to the message by voice and speaks it into the terminal.

[2708] Step 5:

[2709] An emotion engine analyzes the user's responses to identify their emotional state.

[2710] Input: User's voice data (output of step 4)

[2711] Output: Emotional state data

[2712] Specific behavior:

[2713] The emotion engine analyzes voice data to identify emotional states such as relief or joy.

[2714] Step 6:

[2715] The device sends the response and emotional state to the server.

[2716] Input: Text-converted response data and emotional state data

[2717] Output: Transmission complete status

[2718] Specific behavior:

[2719] The terminal converts the user's response into text data and transmits it to the server together with the emotional state data.

[2720] Step 7:

[2721] The server notifies the relative of the response and emotional state.

[2722] Input: Response text data and emotional state data (output of Step 6)

[2723] Output: Notification to relatives completed

[2724] Specific behavior:

[2725] Based on the received data, the server sends a message to the relatives, including information about the user's emotional state. A notification such as "Your father has taken his medicine. He seems relieved" is sent.

[2726] Emergency response

[2727] Step 1:

[2728] The user issues an emergency voice command such as "call an ambulance" or "call XX."

[2729] Input: User's voice command

[2730] Output: Recorded audio data

[2731] Specific behavior:

[2732] In an emergency, the user speaks loudly into the device, such as "call an ambulance," and the device's microphone records the voice.

[2733] Step 2:

[2734] The device recognizes the voice command and immediately notifies the server.

[2735] Input: User's voice command data (output of step 1)

[2736] Output: Transmission complete status

[2737] Specific behavior:

[2738] When the device recognizes an emergency voice command, it immediately notifies the server.

[2739] Step 3:

[2740] The emotion engine analyzes emotions (e.g., surprise, fear, etc.) during emergencies.

[2741] Input: User's voice data (output of step 2)

[2742] Output: Emotional state data

[2743] Specific behavior:

[2744] The emotion engine analyzes the voice data to identify the user's emotional state in emergencies, such as surprise or fear.

[2745] Step 4:

[2746] The server receives emergency voice commands and emotional state and automatically notifies the necessary contacts (119 or relatives).

[2747] Input: Emergency voice command data and emotional state data (output of Step 3)

[2748] Output: Status of contact notification completion

[2749] Specific behavior:

[2750] The server automatically calls 119 or relatives based on emergency voice commands and the user's emotional state, such as "call an ambulance."

[2751] Conversation monitoring and notification to relatives

[2752] Step 1:

[2753] If a device has not had a conversation for a certain period of time (e.g., 3 days), it will automatically notify the server.

[2754] Input: Conversation absence period

[2755] Output: Notification data

[2756] Specific behavior:

[2757] The device will automatically send a notification to the server if there is no conversation for three days.

[2758] Step 2:

[2759] The server receives the notification and notifies the relatives.

[2760] Input: Notification data (output of step 1)

[2761] Output: Notification to relatives completed

[2762] Specific behavior:

[2763] The server notifies the relatives that there has been no communication for three days. The relatives receive a message saying, "Your father has not spoken for three days. Please check."

[2764] Online shopping support

[2765] Step 1:

[2766] The user orders the desired products by voice.

[2767] Input: User's voice commands

[2768] Output: Recorded audio data

[2769] Specific behavior:

[2770] The user speaks a voice instruction to the terminal, such as "Please order some tissues."

[2771] Step 2:

[2772] The terminal recognizes the voice and sends the order to the server.

[2773] Input: User's voice data (output of step 1)

[2774] Output: Text converted data and transmission completion status

[2775] Specific behavior:

[2776] The terminal uses a voice recognition engine to convert the voice data into text data and transmits it to the server.

[2777] Step 3:

[2778] The server receives the order details and processes the purchase on the e-commerce site.

[2779] Input: Text-converted order data (output of Step 2)

[2780] Output: Purchase procedure completed

[2781] Specific behavior:

[2782] The server uses the specified e-commerce site API to complete the purchase process based on the order details.

[2783] Step 4:

[2784] After the server completes the purchase procedure, it sends completion information to the terminal.

[2785] Input: Purchase completion data (output of Step 3)

[2786] Output: Transmission complete status

[2787] Specific behavior:

[2788] The server sends purchase completion information to the terminal and prepares for notification to the user.

[2789] Step 5:

[2790] The terminal will notify the user by voice that the order has been completed.

[2791] Input: Purchase completion data (output from Step 4)

[2792] Output: Audio notification completion status

[2793] Specific behavior:

[2794] The device will use a speech synthesis engine to notify the user by voice that the purchase has been completed, with a message such as "Your order has been placed. It will arrive in two days."

[2795] Fraud prevention features

[2796] Step 1:

[2797] The device constantly monitors the user's conversation and transmits the voice data to the server.

[2798] Input: Conversational voice data

[2799] Output: Transmission complete status

[2800] Specific behavior:

[2801] The device monitors the user's conversation and transmits the voice data to the server in real time.

[2802] Step 2:

[2803] The server uses AI models to analyze the conversation and detect potential fraud.

[2804] Input: Transmitted audio data (output of step 1)

[2805] Output: Fraud probability data

[2806] Specific behavior:

[2807] The server uses an AI model to analyze the received voice data and detect trigger words and phrases that may indicate fraud.

[2808] Step 3:

[2809] The emotion engine analyzes the user's emotional state and immediately warns if there is a high possibility of fraud.

[2810] Input: Transmitted voice data (output of Step 1) and analysis results of the AI ​​model (output of Step 2)

[2811] Output: Emotional state and alert data

[2812] Specific behavior:

[2813] The emotion engine analyzes the user's emotional state and issues a warning if confusion, anxiety, etc. are detected.

[2814] Step 4:

[2815] If the server determines that there is a high possibility of fraud, it sends a warning to the device along with the emotional state.

[2816] Input: Emotional state and alert data (output of step 3)

[2817] Output: Transmission complete status

[2818] Specific behavior:

[2819] If the server determines that there is a high possibility of fraud, it sends a warning message along with the user's emotional state to the terminal.

[2820] Step 5:

[2821] The device will provide audible warnings of fraud along with emotional state.

[2822] Input: Emotional state and alert data (output of step 4)

[2823] output: Completion status of the audio notification

[2824] Specific behavior:

[2825] The device uses a voice synthesis engine to notify users of their emotional state and fraud warnings, such as "This message may be fraudulent. Do you want to continue?"

[2826] Step 6:

[2827] The server notifies the relatives of the possible fraud and their emotional state.

[2828] Input: Emotional state and alert data (output of step 4)

[2829] Output: Notification to relatives completed

[2830] Specific behavior:

[2831] The server notifies the relatives of the likelihood of fraud and their emotional state. A notification such as "A conversation with a high likelihood of fraud has been detected. Your father seems confused" is sent to the relatives.

[2832] (Application example 2)

[2833] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2834] There is a need for a system that allows elderly people to communicate with their relatives in remote locations with peace of mind and that can respond quickly in emergencies. Another important issue is how to improve the user experience by supporting elderly people in easily shopping in virtual stores using voice commands and by suggesting appropriate products based on their emotions.

[2835] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for identifying the emotional state of the elderly person based on emotion recognition and providing appropriate responses or notifications as necessary, means for performing voice-based product searches in the virtual store and suggesting products according to the emotions, and means for displaying advice from relatives. This enables the elderly person to enjoy shopping in the virtual store with peace of mind while receiving support according to their emotions.

[2836] The "system" is a complex set of devices and software that assists elderly people in conversation and allows relatives to check on their condition remotely.

[2837] "Means" refers to a method, apparatus, or program for performing a specified function or task.

[2838] "Emotion recognition" is a technology that analyzes and identifies a person's emotional state from their voice or text data.

[2839] "Emotional state" refers to a person's current psychological state, such as happiness, sadness, anger, or relief.

[2840] "Appropriate response" refers to responding or acting in the most appropriate way according to one's emotional state.

[2841] "Notification" is the act of sending information to the user or relatives to inform them.

[2842] A "virtual store" is a virtual store that offers products and services on an online platform.

[2843] "Product search" refers to the action or process of finding a specific product.

[2844] "Product suggestion" is the act of recommending appropriate products based on the user's needs and emotions.

[2845] "Advice from relatives" refers to advice or instructions sent by relatives, and is information that is displayed for the elderly person to refer to.

[2846] "Speech recognition" is a technology that analyzes speech and converts it into text or instructions.

[2847] A "voice command" is a command to give specific instructions by voice.

[2848] An "emergency" refers to a time-sensitive situation in which an older adult requires immediate attention.

[2849] "Fraud" is an illegal act of deceiving people through dishonest means to obtain money, goods, or information.

[2850] This invention is a system that supports elderly people's conversations and allows relatives to check on their condition remotely, and incorporates emotion recognition technology. The main components of this system are a terminal equipped with voice recognition, emotion recognition, and notification functions, a server that manages and processes various data, an interface that allows relatives to send messages and check on the elderly's condition remotely, and an emotion engine.

[2851] System configuration

[2852] 1. Device:

[2853] This device is used by the elderly and has voice recognition, emotion recognition, and notification functions. The device provides the elderly with information on the weather, news, and topics based on their hobbies and preferences. It also understands the elderly's voice commands and communicates with the server as needed.

[2854] 2. Server:

[2855] The server is responsible for managing and processing various data, analyzing voice and emotion data, providing emergency notifications, and suggesting products. It also collects voice samples from elderly people to improve the accuracy of voice recognition.

[2856] 3. Kinship Interface:

[2857] This is a web interface that allows relatives to send messages from remote locations and check on the status of the elderly. Relatives can input advice for the elderly, and the advice is sent to the elderly's device.

[2858] 4. Emotion Engine:

[2859] The emotion engine analyzes the voices of elderly people and identifies their emotional state, enabling it to provide appropriate responses and suggest products based on their emotions.

[2860] Program processing

[2861] The server analyzes the data using a speech recognition engine (Python's speech_recognition library), an emotion recognition engine (a customized EmotionRecognition class), and a notification system (AlertSystem class). When it receives voice input, it converts the speech into text and analyzes it. Based on the analysis results, it suggests topics and products appropriate for the elderly.

[2862] In the event of an emergency, the system recognizes the user's voice commands and immediately notifies the server, which then automatically issues an emergency notification based on the analysis results of the emotion engine and takes the necessary action.

[2863] Example

[2864] For example, consider the case of an elderly person ordering daily necessities from a virtual store.

[2865] Scenario: An elderly person wants to order tissues from a virtual store.

[2866] Example user:

[2867] "Looking for tissue paper"

[2868] "The weather is nice today, so I want to go for a walk."

[2869] Based on this, the system processes a program that follows the user's instructions, suggests appropriate products, and notifies relatives of the situation.

[2870] Example prompt sentence:

[2871] Scenario: Senior citizen orders an item from a virtual store.

[2872] User: "Find me some tissues."

[2873] The system recognizes your voice and presents a list of related products.

[2874] Emotion Engine: Recognizes the emotions of the elderly and displays the emotion of joy.

[2875] Server: Make appropriate product suggestions.

[2876] This system not only allows relatives to keep a watchful eye on the elderly while they enjoy everyday conversation, but also uses an emotion engine to respond appropriately to the elderly's emotional state.

[2877] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2878] Step 1:

[2879] The user inputs voice commands through the device, which then receives the elderly person's voice instructions and collects voice data. For example, the user might say, "Find me some tissues."

[2880] Step 2:

[2881] The device sends the collected voice data to a speech recognition engine (speech_recognition library) and converts it into text data. The converted text becomes "Look for tissue paper."

[2882] Step 3:

[2883] The device sends the converted text data to an emotion recognition engine (EmotionRecognition class) for emotion analysis. The emotion engine identifies the elderly person's emotions (e.g., joy, excitement) from the text data.

[2884] Step 4:

[2885] The device transmits text data and emotion data to a server, which receives the data and generates a list of candidate products to suggest appropriate products according to the user's emotion.

[2886] Step 5:

[2887] The server then sends the generated product candidate list back to the emotion engine, which selects the product that best matches the emotion. For example, a user who expresses joy might be offered a product with beautiful packaging.

[2888] Step 6:

[2889] The server sends information about the selected product to the terminal. The terminal notifies the user of this information by voice and suggests the product, such as "We suggest this product as tissue paper. Please check it out."

[2890] Step 7:

[2891] The user checks the products and selects or orders by voice. The terminal sends this voice data back to the voice recognition engine and converts it into text data.

[2892] Step 8:

[2893] The terminal sends the converted text data to the server for order processing, and the server communicates with the e-commerce site to process the order.

[2894] Step 9:

[2895] Once the order process is complete, the server sends the completion information to the terminal. The terminal then notifies the user by voice that the order has been completed. For example, it may say, "Your tissue order has been completed. It will arrive in two days."

[2896] Step 10:

[2897] Through the relative interface, the server notifies the relative of the order and the elderly person's emotional state. The relative can then check the elderly person's status through a web interface and send additional advice or instructions if necessary.

[2898] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2899] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2900] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2901] [Fourth embodiment]

[2902] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2903] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2904] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The com...

Claims

1. A system that supports conversations between elderly people and allows relatives to check their status from a remote location. A means of providing weather, news, and topics based on hobbies and preferences to seniors, A means of receiving messages from relatives and notifying the elderly by voice; A way to contact an ambulance or relatives with simple voice commands, A means of sending a notice to relatives if there is no response from the elderly person for a certain period of time; A method to analyze elderly people's conversations and issue warnings in cases of possible fraud Including system.

2. A means for collecting voice samples from seniors to improve speech recognition accuracy; means for recognizing the voice of the elderly person and sending a notification to a relative; The system of claim 1 .

3. Support for online shopping, including the ability to order everyday items using voice commands. The system of claim 1 .

Images