System

Abstract

To provide a system for easily and efficiently supporting an aged person.SOLUTION: A means for transmitting information input by a user to a server through a terminal and generating a user profile based on the information received by the server, a means for recognizing voice input by the user and converting the voice into text, a means for transmitting the text to the server and generating a response based on the text received by the server, a means for converting the response into voice and providing the voice to the user again, and a means for searching the server for information requested by the user from the Internet or a database; The system includes a means for providing a terminal, a means for inputting information on the health condition of a user and transmitting the information to a server through the terminal, and the server automatically contacting an emergency contact, and a means for transmitting information on a commodity desired by the user to the server through the terminal, and the server retrieving the commodity from an affiliated shopping site and performing a purchase procedure.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Elderly people living alone face a variety of anxieties, including loneliness, social isolation, health management, and emergency response. To alleviate these anxieties and improve the quality of their daily lives, a system is needed that allows elderly people to easily communicate and quickly obtain the information they need. However, conventional systems often require users to have advanced technical knowledge, making them difficult to use, especially for the elderly. Therefore, a system that supports the elderly in a simple and efficient manner is needed. [Means for solving the problem]

[0005] The present invention provides a system including: means for transmitting information input by a user to a server via a terminal, and the server generating a user profile based on the information; means for recognizing voice input by the user and converting the voice to text; means for transmitting the text to the server and generating a response based on the text received by the server; means for the terminal converting the response to voice and providing it to the user again; means for the server to search the Internet or a database for information requested by the user and provide it to the terminal; means for inputting information about the user's health condition and transmitting it to the server via the terminal, and the server automatically contacting an emergency contact; and means for transmitting information about a product desired by the user to the server via the terminal, and the server searching for the product on an affiliated shopping site and completing the purchase procedure. This enables elderly people to easily communicate daily, obtain necessary information, and receive prompt support.

[0006] "Users" are individuals, including elderly people, who use the system to receive support for their daily lives.

[0007] A "terminal" is a device that a user uses to perform operations such as voice input and information acquisition, and includes electronic devices such as smartphones and tablets.

[0008] A "server" is a centralized computer system that receives information sent by a user, performs various processes to generate a response, and manages collaboration with other services.

[0009] A "profile" is an individual data set created based on a user's basic information and specific needs, enabling services to be provided that are tailored to the user's characteristics and preferences.

[0010] "Speech recognition" is a technology that electronically analyzes the speech of a user and converts it into text form.

[0011] "Text" means the string of characters generated by speech recognition and used for parsing and response generation on the server.

[0012] "Emergency contact" is a process that automatically contacts emergency services, nearby family members, and local welfare centers when the user's health condition is deemed critical.

[0013] A "shopping site" is a web platform that allows users to purchase daily necessities and groceries online. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone. This system functions by interoperating with three entities: a server, a terminal, and a user.

[0036] Program processing details

[0037] 1. User Registration and Profiling

[0038] The user turns on the device (e.g., a smartphone or tablet) and speaks, "I would like to register my information."

[0039] The device converts the voice to text and displays a screen for entering information.

[0040] The user enters basic information such as name, age, address, and medical history.

[0041] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[0042] The server parses the received JSON data and stores the information in the database.

[0043] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[0044] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[0045] 2. Daily communication

[0046] In the morning, the user says "Good morning" to the terminal.

[0047] The device converts the speech into text and sends the text to the server.

[0048] The server receives the text and generates a response text saying "Good morning. The weather is sunny today."

[0049] The server generates a response and sends it to the terminal.

[0050] The terminal converts the received response text into speech and tells the user, "Good morning. The weather is sunny today."

[0051] 3. Searching for and providing information

[0052] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[0053] The device converts the speech into text and sends the text to the server.

[0054] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[0055] The server sends the contact information it finds to the device.

[0056] The information received by the terminal is displayed and output to the user as audio, such as "The contact information for the nearest welfare center is XXX-XXXX-XXXX."

[0057] 4. Health Management and Emergency Response

[0058] The user tells the terminal that he is "feeling unwell."

[0059] The device detects this sound and goes into emergency call mode.

[0060] The device sends profile data (such as medical history) and status to the server.

[0061] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[0062] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called."

[0063] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[0064] 5. Online shopping support

[0065] The user makes a request to the terminal saying, "I want to buy bread."

[0066] The device converts the speech into text and sends it to the server.

[0067] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[0068] The server sends a list of the found products to the terminal.

[0069] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[0070] If the user answers "yes," the terminal sends the information to the server.

[0071] The server completes the final purchase and sends a confirmation message to the terminal.

[0072] The terminal will display and audibly inform the user that "Purchase completed."

[0073] Specific examples

[0074] For example, when a user wakes up in the morning and says "Good morning" to the device, the device converts the speech into text and sends it to the server. The server generates a response, "Good morning. The weather is sunny today," and sends it back to the device. The device then converts this response into speech and conveys it to the user. This series of steps allows users to engage in everyday communication without feeling isolated.

[0075] This system is divided into detailed processing steps, and by performing specific actions at each step, it provides multifaceted support for elderly people living alone.

[0076] The processing flow will be explained below.

[0077] User Registration and Profiling

[0078] Step 1:

[0079] The user starts up the device and says, "I'd like to register my information."

[0080] Your device uses speech recognition to convert speech into text.

[0081] Step 2:

[0082] The terminal analyzes the converted text and displays a screen for entering user information.

[0083] Step 3:

[0084] The user enters basic information such as name, age, address, and medical history.

[0085] Step 4:

[0086] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[0087] Step 5:

[0088] The server parses the received JSON data and stores the information in the database.

[0089] Step 6:

[0090] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[0091] Step 7:

[0092] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[0093] Daily communication

[0094] Step 1:

[0095] In the morning, the user says "Good morning" to the terminal.

[0096] Step 2:

[0097] The device converts the speech into text and sends the text to the server.

[0098] Step 3:

[0099] The server receives the text and generates a response text saying "Good morning. The weather is sunny today."

[0100] Step 4:

[0101] The server generates a response and sends it to the terminal.

[0102] Step 5:

[0103] The terminal converts the received response text into speech and tells the user, "Good morning. The weather is sunny today."

[0104] Searching for and providing information

[0105] Step 1:

[0106] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[0107] Step 2:

[0108] The device converts the speech into text and sends the text to the server.

[0109] Step 3:

[0110] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[0111] Step 4:

[0112] The server sends the contact information it finds to the device.

[0113] Step 5:

[0114] The information received by the terminal is displayed and output to the user as audio, such as "The contact information for the nearest welfare center is XXX-XXXX-XXXX."

[0115] Health Management and Emergency Response

[0116] Step 1:

[0117] The user tells the terminal that he is "feeling unwell."

[0118] Step 2:

[0119] The device detects this sound and goes into emergency call mode.

[0120] Step 3:

[0121] The device sends profile data (such as medical history) and status to the server.

[0122] Step 4:

[0123] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[0124] Step 5:

[0125] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called."

[0126] Step 6:

[0127] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[0128] Online shopping support

[0129] Step 1:

[0130] The user makes a request to the terminal saying, "I want to buy bread."

[0131] Step 2:

[0132] The device converts the speech into text and sends it to the server.

[0133] Step 3:

[0134] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[0135] Step 4:

[0136] The server sends a list of the found products to the terminal.

[0137] Step 5:

[0138] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[0139] Step 6:

[0140] If the user answers "yes," the terminal sends the information to the server.

[0141] Step 7:

[0142] The server completes the final purchase and sends a confirmation message to the terminal.

[0143] Step 8:

[0144] The terminal will display and audibly inform the user that "Purchase completed."

[0145] By going through the above steps, this system can provide multifaceted support to elderly people living alone.

[0146] Example 1

[0147] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0148] Elderly people living alone face a wide range of challenges, including daily communication, health management, and emergency response. To efficiently and comprehensively address these challenges, an easy-to-use system is required. However, current technology provides solutions for individual problems in a fragmented manner, preventing a consistent user experience.

[0149] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0150] In this invention, the server includes means for transmitting information input by a user to the server via a terminal and for the server to generate a user profile based on the information, means for recognizing a voice input by the user and converting the voice into text, and means for transmitting the text to the server and generating a response based on the text received by the server, thereby enabling multifaceted support for elderly people in their daily activities such as communication, emergency response, health management, and online shopping.

[0151] "Users" refers to the elderly people who use the system and the people who support them.

[0152] A "terminal" is a device operated by a user, and refers to information and communication devices such as smartphones and tablets.

[0153] "Server" refers to a central computing device that processes information sent from terminals and manages it in a database.

[0154] A "user profile" refers to a data set that includes personal information such as a user's basic information and medical history.

[0155] "Voice input" refers to voice commands given by a user through a microphone.

[0156] "Speech recognition" refers to the technology that converts voice input into text form.

[0157] "Text conversion" refers to the process of processing text obtained by speech recognition as data.

[0158] "Secure Protocol" refers to a secure communications protocol such as HTTPS.

[0159] "Generate" refers to the process by which the server generates text or audio information.

[0160] "Search" refers to the operation by which a server retrieves information using the Internet or an internal database.

[0161] "Emergency contact" refers to the procedure by which the server automatically notifies pre-defined contacts in the event of an emergency.

[0162] "Shopping Site" refers to the affiliated e-commerce platform.

[0163] "Voice notification" refers to a function in which a terminal uses a voice conversion function to convey information to a user.

[0164] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone. This system functions by interoperating with three entities: a server, a terminal, and a user.

[0165] First, a user uses a smartphone or tablet as a terminal. These terminals have a speech recognition function and convert the user's voice input into text using the Google (registered trademark) Speech-to-Text API. The terminal also converts text into speech using the Google Text-to-Speech API.

[0166] Next, the information entered by the user is sent to the server via the terminal. For example, when a user enters basic information such as name, age, address, and medical history, this information is converted into JSON format and sent to the server using the HTTPS protocol. The server analyzes the received information, stores it in a MySQL® database, and creates a user profile.

[0167] In everyday communication, when a user speaks to a device, for example, "Good morning," the device converts the speech into text and sends it to the server. The server generates a response text and sends it to the device. The device then converts the received response text into speech and conveys it to the user. For example, "Good morning. The weather is sunny today."

[0168] Also, when a user says to the device, "Please tell me the contact information for the nearest welfare center," the device converts the voice into text and sends that text to the server. The server uses an internal database or an external API (such as Google Maps API) to search for the welfare center's contact information and sends that information to the device. The device then converts the received information into voice and relays it to the user.

[0169] Furthermore, if the user tells the device that they are "feeling unwell," the device will enter emergency call mode and send their profile data and status to the server. The server will retrieve emergency contact information from a database and contact the appropriate resource (e.g., emergency services). The server will then send a confirmation message to the device saying "an ambulance has been called," which the device will then verbally convey to the user.

[0170] For online shopping support, when a user requests the device to "buy bread," the device converts the voice to text and sends it to the server. The server then searches for a product list from affiliated shopping sites and sends it to the device. The device then displays the product list to the user and asks aloud, "Is this OK?" If the user answers "yes," the device completes the final purchase procedure and a confirmation message is spoken by the device.

[0171] Specific examples

[0172] For example, if a user says "Good morning" to a device in the morning, the device converts the speech into text and sends it to the server. The server generates a response such as "Good morning. The weather is sunny today" and sends it back to the device. The device then converts this response into speech and conveys it to the user. This series of steps allows users to engage in everyday communication without feeling isolated.

[0173] Examples of prompts include "Good morning," "Please tell me the contact information of the nearest welfare center," "I'm not feeling well," "I want to buy some bread," etc. Through such prompts, it is possible to provide the multifaceted support that the elderly require in their daily lives.

[0174] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0175] Program processing flow

[0176] User Registration and Profiling

[0177] Step 1:

[0178] The user starts up the device and says, "I'd like to register my information."

[0179] Input: Audio

[0180] Output: Audio data

[0181] Specific operation: The user uses a smartphone or tablet to input voice information.

[0182] Step 2:

[0183] The device converts the speech to text.

[0184] Input: Audio data

[0185] Output: Text data

[0186] What happens: The device converts speech to text using the Google Speech-to-Text API.

[0187] Step 3:

[0188] The user enters basic information such as name, age, address, and medical history.

[0189] Input: User basic information

[0190] Output: User basic information data

[0191] Specific operation: The user enters information into an input form displayed on the device screen.

[0192] Step 4:

[0193] The device converts the information into JSON format.

[0194] Input: User basic information data

[0195] Output: JSON data

[0196] Specific operation: The terminal converts the input data into JSON format.

[0197] Step 5:

[0198] The terminal sends data to the server.

[0199] Input: JSON data

[0200] Output: Send data to the server

[0201] Specific operation: The device sends data to the server using the HTTPS protocol.

[0202] Step 6:

[0203] The server analyzes the data and stores it in a database.

[0204] Input: JSON data

[0205] Output: Store in database

[0206] Specific operation: The server analyzes the received data and stores it in a MySQL database.

[0207] Step 7:

[0208] The server generates a confirmation message and sends it to the terminal.

[0209] Input: Database update information

[0210] Output:Confirmation message

[0211] Specific operation: The server creates a message saying "Profile has been generated" and sends it to the terminal.

[0212] Step 8:

[0213] The device will notify the user by voice.

[0214] Input:Confirmation message

[0215] Output: Audio notification

[0216] Specific behavior: The device uses the Google Text-to-Speech API to convert the text to speech and notifies the user that "Registration is complete."

[0217] Daily communication

[0218] Step 1:

[0219] The user says "Good morning" to the terminal.

[0220] Input: Audio

[0221] Output: Audio data

[0222] Specific operation: The user greets the device by voice.

[0223] Step 2:

[0224] The device converts the speech to text.

[0225] Input: Audio data

[0226] Output: Text data

[0227] What happens: The device converts speech to text using the Google Speech-to-Text API.

[0228] Step 3:

[0229] The device sends the text to the server.

[0230] Input: Text data

[0231] Output: Send text data to the server

[0232] Specific operation: The device sends text data to the server via HTTP.

[0233] Step 4:

[0234] The server parses the text and generates a response.

[0235] Input: Text data

[0236] Output: Response text

[0237] Specific operation: The server analyzes the received text data and generates a response saying, "Good morning. The weather is sunny today."

[0238] Step 5:

[0239] The server sends the response text to the terminal.

[0240] Input: Response text

[0241] Output: Response sent from the server

[0242] Specific operation: The response text generated by the server is sent to the terminal.

[0243] Step 6:

[0244] The terminal converts the response into voice and conveys it to the user.

[0245] Input: Response text

[0246] Output: Audio notification

[0247] What happens: The device uses the Google Text-to-Speech API to convert text to speech and tells the user, "Good morning. The weather is sunny today."

[0248] Searching for and providing information

[0249] Step 1:

[0250] The user speaks to the device, saying, "Please tell me the contact information for the nearest welfare center."

[0251] Input: Audio

[0252] Output: Audio data

[0253] Specific behavior: The user makes a request by voice.

[0254] Step 2:

[0255] The device converts the speech to text.

[0256] Input: Audio data

[0257] Output: Text data

[0258] What happens: The device converts speech to text using the Google Speech-to-Text API.

[0259] Step 3:

[0260] The device sends the text to the server.

[0261] Input: Text data

[0262] Output: Send data to the server

[0263] Specific operation: The device sends text data to the server via HTTP.

[0264] Step 4:

[0265] The server retrieves the information.

[0266] Input: Text data

[0267] Output: Search results

[0268] Specific operation: The server searches for "welfare center contact information" using its internal database and Google Maps API.

[0269] Step 5:

[0270] The server sends the found information to the device.

[0271] Input: Search results

[0272] Output: Information sent from the server

[0273] Specific operation: The server sends the search results to the terminal.

[0274] Step 6:

[0275] The device converts the information into voice and conveys it to the user.

[0276] Input: Search results

[0277] Output: Audio notification

[0278] What it does: The device uses the Google Text-to-Speech API to convert the text into speech and tells the user, "The contact information for the nearest welfare center is XXX-XXXX-XXXX."

[0279] Health Management and Emergency Response

[0280] Step 1:

[0281] The user tells the terminal that he is "feeling unwell."

[0282] Input: Audio

[0283] Output: Audio data

[0284] Specific operation: The user vocally complains about feeling unwell.

[0285] Step 2:

[0286] The device converts the speech to text.

[0287] Input: Audio data

[0288] Output: Text data

[0289] What happens: The device converts speech to text using the Google Speech-to-Text API.

[0290] Step 3:

[0291] The device enters emergency call mode.

[0292] Input: Text data

[0293] Output: Emergency call mode activated

[0294] Specific operation: The device receives the text data and activates emergency call mode.

[0295] Step 4:

[0296] The device sends profile data and status to the server.

[0297] Input: Profile data, text data

[0298] Output: Send data to the server

[0299] Specific operation: The device sends profile data and status to the server.

[0300] Step 5:

[0301] The server retrieves the emergency contact information.

[0302] Input: Profile data

[0303] Output: Emergency contact information

[0304] Specific operation: The server retrieves emergency contact information from the database.

[0305] Step 6:

[0306] The server contacts emergency services.

[0307] Input: Emergency Contact Information

[0308] Output: Emergency contact execution

[0309] Specific Actions: The server contacts emergency services.

[0310] Step 7:

[0311] The server sends a confirmation message to the terminal.

[0312] Input: Contact status

[0313] Output:Confirmation message

[0314] Specific operation: The server sends a confirmation message to the terminal saying "An ambulance has been called."

[0315] Step 8:

[0316] The device will notify the user by voice.

[0317] Input:Confirmation message

[0318] Output: Audio notification

[0319] What happens: The device uses the Google Text-to-Speech API to convert the text into speech and notifies the user, "An ambulance has been called. Please remain calm."

[0320] Online shopping support

[0321] Step 1:

[0322] The user makes a request to the terminal saying, "I want to buy bread."

[0323] Input: Audio

[0324] Output: Audio data

[0325] Specific operation: The user expresses their purchase intention by voice.

[0326] Step 2:

[0327] The device converts the speech to text.

[0328] Input: Audio data

[0329] Output: Text data

[0330] What happens: The device converts speech to text using the Google Speech-to-Text API.

[0331] Step 3:

[0332] The device sends the text to the server.

[0333] Input: Text data

[0334] Output: Send data to the server

[0335] Specific operation: The device sends text data to the server via HTTP.

[0336] Step 4:

[0337] The server parses the text and searches for products.

[0338] Input: Text data

[0339] Output: Product list

[0340] Specific operation: The server searches for a list of bread products using the API of a shopping site it is affiliated with.

[0341] Step 5:

[0342] The server sends the product list to the terminal.

[0343] Input: Product list

[0344] Output: List sent from server

[0345] Specific operation: The server sends the list of found products to the terminal.

[0346] Step 6:

[0347] The device displays a list of products and asks aloud, "Is this okay?"

[0348] Input: Product list

[0349] Output: Display, voice prompts

[0350] Specific operation: The device displays a list of products to the user and asks aloud, "Is this okay?"

[0351] Step 7:

[0352] The user responds "Yes."

[0353] Input: Audio

[0354] Output: Audio data

[0355] What happens: The user confirms the purchase and responds verbally.

[0356] Step 8:

[0357] The terminal sends this response to the server.

[0358] Input: Audio data

[0359] Output: Text data

[0360] Specific operation: The device converts the voice data into text and sends it to the server.

[0361] Step 9:

[0362] The server completes the final purchase and sends a confirmation message to the terminal.

[0363] Input: Text data

[0364] Output:Confirmation message

[0365] Specific operation: The server performs the final purchase procedure, generates a message saying "Purchase completed" and sends it to the terminal.

[0366] Step 10:

[0367] The device will notify you by voice that the purchase is complete.

[0368] Input:Confirmation message

[0369] Output: Audio notification

[0370] What happens: The device uses the Google Text-to-Speech API to convert the text to speech and tells the user, "Your purchase is complete."

[0371] (Application example 1)

[0372] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0373] The lack of comprehensive support systems for the various problems that elderly people face in their daily lives is a challenge. In particular, systems that can quickly respond when elderly people order meals or in emergencies are needed, but the current means to achieve this are insufficient. Furthermore, interactive interfaces are also needed to reduce the burden that elderly people feel when performing these tasks alone.

[0374] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0375] In this invention, the server includes: means for transmitting information input by a user to the server through a terminal and for the server to generate a user profile based on the information; means for recognizing a voice input by the user and converting the voice to text; means for transmitting the text to the server and generating a response based on the text received by the server; means for the terminal to convert the response to voice and provide it to the user again; means for the server to search the Internet or a database for information corresponding to the user's request and provide it to the terminal; means for transmitting an order for a meal desired by the user to the server through the terminal, searching for product lists from affiliated food delivery services with the server, and completing the purchase procedure; means for the server to check the delivery status and obtain information and send it to the terminal when the user says "Tell me the delivery status" to the terminal; and means for the terminal to enter emergency response mode when the user says to the terminal that they are feeling unwell, notifying emergency contacts via the server, and arranging appropriate responses. This enables elderly people to easily meet various needs in their daily lives and respond to emergencies.

[0376] 1. A "terminal" is an electronic device that allows users to input information or give voice instructions. Examples include smartphones, tablets, and robots.

[0377] 2. A "server" is a computer system that receives data sent from a terminal via the Internet or a network, performs the necessary processing, and then sends a response back to the terminal.

[0378] 3. A "user profile" is a set of detailed data such as a user's personal information, past behavioral history, preferences, and allergy information.

[0379] 4. "Voice input" means the user speaks through a microphone, which is converted into text by the system.

[0380] 5. "Text conversion" is the process of analyzing voice input and converting it into text information.

[0381] 6. "Response" means a reply message generated by the server based on received text and profile information.

[0382] 7. "Voice output" is the process of converting the generated response text from text information back into voice and delivering it to the user.

[0383] 8. "Internet" means a worldwide collection of computer networks that enable the transmission and reception of information.

[0384] 9. A "database" is a collection of structured data that organizes large amounts of information and allows for quick search and retrieval.

[0385] 10. "Food delivery service" means a service that allows a user to order a meal they want and delivers the meal to a location specified by the user.

[0386] 11. "Emergency Response Mode" is a function that allows the system to automatically take special action and notify emergency contacts when a user reports a health problem.

[0387] 12. "Purchase procedure" refers to a series of processes for purchasing products from affiliated shopping sites based on information about the products desired by the user.

[0388] 13. "Delivery status" is information that indicates the current location of the ordered meal or product and how long it will take to arrive at the user's location.

[0389] 14. "Application" means a specific example of applying this invention to a specific field or use.

[0390] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone, and it functions by interoperating with three main entities: a terminal, a server, and a user. The system has a wide range of functions, including user profile generation, voice input recognition, text conversion, response generation, information search, food ordering, delivery status tracking, and emergency response.

[0391] 1. User profiling

[0392] Users input information through devices such as smartphones and tablets. The devices convert the input speech into text and send it to a server in JSON format using the HTTPS protocol. The server analyzes the data and creates a user profile including name, age, address, medical history, dietary preferences, and allergy information, and stores it in a database.

[0393] 2. Voice Input and Response Generation

[0394] When a user says "I'd like to order lunch" to the device, the device converts the speech into text and sends it to the server. The server analyzes the received text and generates a list of recommended menu items based on past order history and profile information. This menu list is sent back to the device, which then asks aloud, "Is this OK?" This allows the user to complete their meal order using only the voice interface.

[0395] 3. Track your delivery status

[0396] When the user speaks to the device saying, "Tell me the delivery status," the device converts the speech into text and sends it to the server. The server obtains the delivery status from the partner food delivery service and sends the real-time status to the device. The device then notifies the user by notifying them, for example, "The delivery person will arrive in 5 minutes."

[0397] 4. Emergency Response

[0398] When a user notifies the device that they are "feeling unwell," the device enters emergency response mode and sends an emergency notification along with their profile data to the server. The server notifies emergency contacts and arranges for the necessary response. The device then notifies the user by voice, saying, "An ambulance has been called. Please remain calm."

[0399] Hardware and software used

[0400] Hardware: smartphones, tablets, robots

[0401] software:

[0402] Natural Language Processing: Google Cloud Speech-to-Text

[0403] Data transmission / reception: HTTPS protocol

[0404] Database: MySQL

[0405] Server: AWS (registered trademark) EC2

[0406] Speech synthesis: Google Cloud Text-to-Speech

[0407] Specific examples

[0408] Consider a scenario where an elderly person wants to order lunch. When the user says "I'd like to order lunch," the following dialogue takes place:

[0409] User: "I'd like to order lunch."

[0410] Device: "The recommended menu items registered are curry rice, sandwiches, and udon. Which would you like?"

[0411] User: "Curry rice"

[0412] Terminal: "I ordered curry rice."

[0413] Example prompt sentence:

[0414] "A user is trying to order lunch using a food delivery assistance assistant for seniors. Generate a series of dialogues with corresponding menu choices."

[0415] This will enable elderly people to easily meet various needs in their daily lives and respond to emergencies.

[0416] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0417] Step 1:

[0418] The user turns on a device such as a smartphone or tablet and says, "I would like to register as a user." The device converts the voice to text and displays an information entry screen. The user enters basic information such as name, address, dietary preferences, and allergy information, and the device converts this information into JSON format. This data is sent to the server using the HTTPS protocol.

[0419] Input: User voice input "I would like to register as a user", basic information

[0420] Output: JSON formatted data sent to the server

[0421] Step 2:

[0422] The server analyzes the received JSON data and generates a user profile including the user's name, age, address, medical history, dietary preferences, and allergy information. The generated user profile is stored in a database. The server confirms that the profile has been successfully generated and notifies the device that "The profile has been generated."

[0423] Input: User information in JSON format

[0424] Output: User profile stored in database, notification to terminal

[0425] Step 3:

[0426] The user speaks to the device, saying, "I'd like to order lunch." The device converts the speech into text and sends it to the server. The server analyzes the text and generates a list of recommended menu items based on past order history and user profile information. This menu list is then sent to the device.

[0427] Input: User speaks "I want to order lunch"

[0428] Output: Menu list sent to the terminal

[0429] Step 4:

[0430] The terminal displays a menu list and asks the user aloud, "Is this OK?" If the user answers "Yes," the terminal sends the information to the server, which then processes the final order. The server then sends a confirmation message to the terminal, informing the user that the order has been completed.

[0431] Input: User voice input "Yes"

[0432] Output: Order information sent to the server, confirmation message sent to the terminal

[0433] Step 5:

[0434] The user speaks to the device, saying, "Tell me the delivery status." The device converts the speech into text and sends it to the server. The server obtains the delivery status from the partner food delivery service and sends the real-time information to the device. The device then notifies the user, "The delivery person will arrive in 5 minutes."

[0435] Input: User's voice input "Tell me the delivery status"

[0436] Output: Delivery status information sent to the terminal

[0437] Step 6:

[0438] The user says "I'm feeling unwell" to the device. The device converts the speech to text and enters emergency response mode. It sends an emergency notification along with the profile data to the server. The server notifies emergency contacts and arranges for the necessary response. The device then tells the user by voice, "An ambulance has been called. Please remain calm."

[0439] Input: User's voice input "I feel sick"

[0440] Output: Notification to emergency contacts, feedback to device

[0441] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0442] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone, and is equipped with a function that recognizes the user's emotions and provides appropriate responses and support based on those emotions. This system functions in cooperation with three entities: a server, a terminal, and a user, and combines an emotion engine to improve the user experience.

[0443] Program processing details

[0444] User Registration and Profiling

[0445] The user starts up the device and says, "I'd like to register my information."

[0446] Your device uses speech recognition to convert speech into text.

[0447] The terminal analyzes the converted text and displays a screen for entering user information.

[0448] The user enters basic information such as name, age, address, and medical history.

[0449] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[0450] The server parses the received JSON data and stores the information in the database.

[0451] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[0452] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[0453] Daily communication

[0454] In the morning, the user says "Good morning" to the terminal.

[0455] The device converts the speech into text and sends the text to the server.

[0456] The server receives the text and analyzes the user's emotions using an emotion engine.

[0457] Based on the analysis results, the server generates a response text saying, "Good morning. Today's weather is sunny and it looks like it's going to be a pleasant day."

[0458] The server generates a response and sends it to the terminal.

[0459] The terminal converts the received response text into speech and tells the user, "Good morning. The weather is sunny today, and it looks like it's going to be a pleasant day."

[0460] Searching for and providing information

[0461] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[0462] The device converts the speech into text and sends the text to the server.

[0463] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[0464] The server analyzes the contact information it finds using an emotion engine and presents it in a format that corresponds to the user's emotions.

[0465] The server sends the information to the terminal.

[0466] The information received by the device is displayed and output to the user as audio, saying, "The contact number for the nearest welfare center is XXX-XXXX-XXXX. If you have any concerns, you can contact them immediately."

[0467] Health Management and Emergency Response

[0468] The user tells the terminal that he is "feeling unwell."

[0469] The device detects this sound and goes into emergency call mode.

[0470] The device sends profile data (such as medical history) and status to the server.

[0471] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[0472] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called. Please wait a moment."

[0473] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[0474] Online shopping support

[0475] The user makes a request to the terminal saying, "I want to buy bread."

[0476] The device converts the speech into text and sends it to the server.

[0477] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[0478] The server analyzes the found product list using an emotion engine and presents it in an appropriate format.

[0479] The server sends the product list to the terminal.

[0480] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[0481] If the user answers "yes," the terminal sends the information to the server.

[0482] The server completes the final purchase and sends a confirmation message to the terminal.

[0483] The terminal will display and audibly inform the user that "Purchase completed."

[0484] Specific examples

[0485] For example, if a user says "Good morning" to a device in the morning, the device converts the speech into text and sends it to the server. The server analyzes the text and the user's emotions, and generates a response that best suits the user's state of mind. This process allows the user to receive a response that reflects their emotions, enabling everyday communication while reducing feelings of isolation.

[0486] By combining this system with an emotion engine, it provides multifaceted support for elderly people living alone and offers more personalized services.

[0487] The processing flow will be explained below.

[0488] User Registration and Profiling

[0489] Step 1:

[0490] The user starts up the device and says, "I'd like to register my information."

[0491] Your device uses speech recognition to convert speech into text.

[0492] Step 2:

[0493] The terminal analyzes the converted text and displays a screen for entering user information.

[0494] Step 3:

[0495] The user enters basic information such as name, age, address, and medical history.

[0496] Step 4:

[0497] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[0498] Step 5:

[0499] The server parses the received JSON data and stores the information in the database.

[0500] Step 6:

[0501] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[0502] Step 7:

[0503] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[0504] Daily communication

[0505] Step 1:

[0506] In the morning, the user says "Good morning" to the terminal.

[0507] Step 2:

[0508] The device converts the speech into text and sends the text to the server.

[0509] Step 3:

[0510] The server receives the text and analyzes the user's emotions using an emotion engine.

[0511] Step 4:

[0512] Based on the results of the sentiment analysis, the server generates a response text such as "Good morning. The weather is sunny today and you look like you're in a good mood."

[0513] Step 5:

[0514] The server generates a response and sends it to the terminal.

[0515] Step 6:

[0516] The device converts the received response text into speech and tells the user, "Good morning. The weather is sunny today and you look like you're in a good mood."

[0517] Searching for and providing information

[0518] Step 1:

[0519] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[0520] Step 2:

[0521] The device converts the speech into text and sends the text to the server.

[0522] Step 3:

[0523] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[0524] Step 4:

[0525] The server analyzes the contact information it finds using an emotion engine and creates information in a format that corresponds to the user's emotions.

[0526] Step 5:

[0527] The server sends the information to the terminal.

[0528] Step 6:

[0529] The information received by the terminal is displayed and output to the user as audio, saying, "The contact number for the nearest welfare center is XXX-XXXX-XXXX. Please feel free to contact us at any time."

[0530] Health Management and Emergency Response

[0531] Step 1:

[0532] The user tells the terminal that he is "feeling unwell."

[0533] Step 2:

[0534] The device detects this sound and goes into emergency call mode.

[0535] Step 3:

[0536] The device sends profile data (such as medical history) and status to the server.

[0537] Step 4:

[0538] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[0539] Step 5:

[0540] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called. Please wait a moment."

[0541] Step 6:

[0542] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[0543] Online shopping support

[0544] Step 1:

[0545] The user makes a request to the terminal saying, "I want to buy bread."

[0546] Step 2:

[0547] The device converts the speech into text and sends it to the server.

[0548] Step 3:

[0549] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[0550] Step 4:

[0551] The server analyzes the found product list using an emotion engine and presents it in an appropriate format.

[0552] Step 5:

[0553] The server sends the product list to the terminal.

[0554] Step 6:

[0555] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[0556] Step 7:

[0557] If the user answers "yes," the terminal sends the information to the server.

[0558] Step 8:

[0559] The server completes the final purchase and sends a confirmation message to the terminal.

[0560] Step 9:

[0561] The terminal will display and audibly inform the user that "Purchase completed."

[0562] Through these steps, the system can provide multifaceted support for elderly people living alone and provide services that take users' emotions into consideration.

[0563] Example 2

[0564] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0565] In modern society, the number of elderly people living alone is increasing, making support for daily life and emergency response important issues. Furthermore, providing appropriate responses according to emotions is required to reduce the sense of isolation felt by elderly people. Conventional speech recognition systems have difficulty analyzing a user's emotions and generating appropriate responses. Furthermore, their ability to provide information needed for daily life and emergency response is limited. Therefore, a comprehensive system is needed that can provide multifaceted support for the daily life of elderly people and provide appropriate responses according to their emotions.

[0566] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for transmitting information input by a user to the information processing device via a terminal, and the information processing device generating a user profile based on the information; means for recognizing a voice input by the user and converting the voice into text; means for transmitting the text to the information processing device and generating a response based on the received text and emotion analysis; means for the terminal to convert the response into speech and provide it to the user again; means for the information processing device to search for information corresponding to the user's request from a network or database and provide it to the terminal; means for inputting information about the user's health condition, transmitting it to the information processing device via the terminal, and automatically contacting an emergency contact; and means for transmitting information about a product desired by the user via the terminal to the information processing device, and the information processing device searching for the product on a related e-commerce site and completing a purchase procedure. This makes it possible to comprehensively support the daily lives of elderly people and provide appropriate responses according to their emotions.

[0567] "Information processing device" refers to a device or system that inputs, processes, stores, and outputs data. Generally, this includes servers and computers.

[0568] "User profile" refers to a user's personal information data, including basic information such as the user's name, age, address, medical history, and sentiment analysis results.

[0569] "Speech recognition" refers to the technology of analyzing a user's voice input and converting it into text data.

[0570] "Emotion analysis" refers to the technology of analyzing a user's emotional state from text data and voice data.

[0571] "Response generation" refers to the process of creating an appropriate response to the user based on the analyzed information.

[0572] A "network" refers to a system that connects multiple computers and devices and enables the exchange of data.

[0573] A "database" refers to a system for efficiently managing, searching, and storing data.

[0574] "Encryption protocol" refers to the technology used to encrypt data in order to ensure secure data communication.

[0575] "Emergency contact" refers to contact information that is set to be contacted in the event of a sudden change in the user's health condition or an emergency.

[0576] "E-commerce site" refers to a website that allows users to sell or purchase goods or services over the Internet.

[0577] This invention is a comprehensive service system that provides multifaceted support for elderly people living alone. Its key feature is its ability to recognize the user's emotions and provide appropriate responses and support based on those emotions. This system functions in coordination with three entities: a server, a terminal, and a user, and combines an emotion engine to enhance the user experience. Below, we will explain in detail each component of this system and its operation.

[0578] Hardware and Software Use

[0579] This system uses several key hardware and software components. The terminal is a device that performs speech recognition, speech synthesis, and display. Specifically, it is a smart device equipped with a microphone, speaker, and display. The system uses the Google Cloud Speech-to-Text API for speech recognition and the Google Cloud Text-to-Speech API for speech synthesis. The system also uses the IBM Watson® Tone Analyzer for user emotion analysis.

[0580] The server manages user profiles, analyzes emotions, generates responses, manages databases, and performs network communications. The server is a computer with high-performance computing capabilities and uses MySQL as its database. The HTTPS protocol is used as the communications protocol to ensure security.

[0581] System Operation

[0582] User Registration and Profiling

[0583] The user turns on the device and says, "I'd like to register my information." The device uses the Google Cloud Speech-to-Text API to convert the speech to text and analyzes that text. A user information input screen is displayed, and the user enters information such as their name, age, address, and medical history. The device converts this information into JSON format and sends it to the server using the HTTPS protocol. The server analyzes the received data and stores it in a MySQL database. Once the profile is generated, a confirmation message is displayed on the device and a voice notification is played saying, "Registration is complete."

[0584] Daily communication

[0585] In the morning, when a user says "Good morning" to the device, the device converts the speech into text and sends it to the server. The server performs text and sentiment analysis and generates an appropriate response. The generated response is sent to the device, which converts it into speech and conveys it to the user. For example, the server might generate a response like "Good morning. The weather is sunny today, and it looks like it's going to be a pleasant day."

[0586] Searching for and providing information

[0587] When a user says to the device, "Tell me the contact information for the nearest welfare center," the device converts the speech into text and sends it to the server. The server analyzes the text, searches for contact information using a database or external API, and provides information in an appropriate format based on the user's emotion. For example, the server might generate a message saying, "The contact information for the nearest welfare center is XXX-XXXX-XXXX. If you have any concerns, you can contact them immediately," and the device will notify the message aloud.

[0588] Health Management and Emergency Response

[0589] When the user tells the device that they are "feeling unwell", the device goes into emergency call mode and sends their profile data and situation to the server. The server retrieves emergency contacts and contacts the appropriate resource (e.g., emergency services). The server executes the emergency call and sends feedback to the device saying "An ambulance has been called." The device then tells the user by voice, "An ambulance has been called. Please remain calm."

[0590] Online shopping support

[0591] When a user requests "I want to buy bread" from the device, the device converts the speech into text and sends it to the server. The server analyzes the text and searches for product lists from affiliated online shopping sites. It performs sentiment analysis on the found product list and presents it to the user in an appropriate format. If the user responds "Yes," the server carries out the final purchase procedure and sends a confirmation message to the device. The device then displays and sounds a message saying "Purchase completed."

[0592] Specific examples

[0593] For example, when a user says "Good morning" to a device in the morning, the device converts the speech into text and sends it to the server. The server analyzes the received text and generates a response based on the results of emotion analysis. This series of steps allows the user to receive a response that reflects their emotions. The specific prompt text is as follows:

[0594] When you say "Good morning," you respond with:

[0595] "Good morning. The weather is sunny today and it looks like it's going to be a pleasant day."

[0596] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0597] User Registration and Profiling

[0598] Step 1:

[0599] The user starts up the device and speaks, "I'd like to register my information." Once the voice input is complete, the device receives the voice data.

[0600] Input: User voice input

[0601] Output: Audio data

[0602] What happens: The device's microphone captures the user's voice.

[0603] Step 2:

[0604] The device uses the Google Cloud Speech-to-Text API to convert speech to text.

[0605] Input: Audio data

[0606] Output: Text data

[0607] Specific operation: The audio data is sent to the Google Cloud Speech-to-Text API and returned as text data.

[0608] Step 3:

[0609] The terminal analyzes the converted text and displays a screen for entering user information.

[0610] Input: Text data

[0611] Output: User information input screen

[0612] Specific operation: The terminal analyzes the text data and generates a graphical user interface based on its content.

[0613] Step 4:

[0614] The user enters information such as name, age, address, medical history, etc. The terminal receives this input and examines the data in real time.

[0615] Input: User input information

[0616] Output: Validated input data

[0617] Specific operation: The terminal checks the data type and format of each field in real time.

[0618] Step 5:

[0619] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[0620] Input: Validated input data

[0621] Output: JSON data

[0622] Specific operation: The device formats the data into JSON format and sends it to the server over a secure communication channel.

[0623] Step 6:

[0624] The server analyzes the received data and stores it in a MySQL database.

[0625] Input: JSON data

[0626] Output: User information stored in the database

[0627] What happens: The server deserializes the data and performs an insert operation into the database.

[0628] Step 7:

[0629] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[0630] Input: User information stored in the database

[0631] Output:Confirmation message

[0632] Specific operations: The server configures the user profile and sends a message to the terminal indicating success.

[0633] Step 8:

[0634] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[0635] Input:Confirmation message

[0636] Output: Visual and audio notifications

[0637] Specific operation: The device displays the message on the screen and converts it into audio using the Google Cloud Text-to-Speech API and plays it back.

[0638] Daily communication

[0639] Step 1:

[0640] The user speaks "Good morning" to the device in the morning, and the device recognizes the voice.

[0641] Input: User voice input

[0642] Output: Audio data

[0643] Specific operation: The device's microphone captures audio data.

[0644] Step 2:

[0645] The device uses the Google Cloud Speech-to-Text API to convert speech to text.

[0646] Input: Audio data

[0647] Output: Text data

[0648] Specific operation: The device sends voice data to the Google Cloud Speech-to-Text API and receives it as text data.

[0649] Step 3:

[0650] The terminal sends the converted text to the server.

[0651] Input: Text data

[0652] Output: Text data sent to the server

[0653] Specific operation: The terminal communicates via the Internet to send text data to the server.

[0654] Step 4:

[0655] The server receives the text data and performs sentiment analysis using IBM Watson Tone Analyzer.

[0656] Input: Text data

[0657] Output: Emotion data

[0658] Specific operation: The server sends the text data to the IBM Watson Tone Analyzer and obtains the analysis results.

[0659] Step 5:

[0660] The server generates a response text based on the sentiment analysis results.

[0661] Input: Emotion data

[0662] Output: Response text

[0663] Specific operation: The server analyzes the emotional data and generates an appropriate response as text.

[0664] Step 6:

[0665] The server generates a response text and sends it to the terminal.

[0666] Input: Response text

[0667] Output: Response text sent to the terminal

[0668] Specific operation: The server communicates via the Internet to send the response text to the terminal.

[0669] Step 7:

[0670] The device converts the received response text into speech using the Google Cloud Text-to-Speech API and conveys it to the user.

[0671] Input: Response text

[0672] Output: Voice response

[0673] What it does: Your device converts the text into speech and plays it through the speaker.

[0674] Searching for and providing information

[0675] Step 1:

[0676] The user speaks to the device, saying, "Please tell me the contact information for the nearest welfare center." The device recognizes the voice.

[0677] Input: User voice input

[0678] Output: Audio data

[0679] Specific operation: The device's microphone captures audio data.

[0680] Step 2:

[0681] The device uses the Google Cloud Speech-to-Text API to convert speech to text.

[0682] Input: Audio data

[0683] Output: Text data

[0684] Specific operation: Sends audio data to the API and retrieves text data.

[0685] Step 3:

[0686] The terminal sends the converted text to the server.

[0687] Input: Text data

[0688] Output: Text data sent to the server

[0689] Specific operation: To send text data to the server, the HTTPS protocol is used for communication.

[0690] Step 4:

[0691] The server receives the text data and searches for "welfare center contact information" using an internal database or external API.

[0692] Input: Text data

[0693] Output: Contact information

[0694] What it does: Parses text data and queries databases and APIs to retrieve contact information.

[0695] Step 5:

[0696] The server performs sentiment analysis on the contact information and generates it in a format that corresponds to the user's sentiment.

[0697] Input: Contact Information

[0698] Output: Formatted response text

[0699] Specific operation: Based on the results of sentiment analysis, the contact information is formatted into appropriate text.

[0700] Step 6:

[0701] The server sends the information to the terminal.

[0702] Input: Formatted response text

[0703] Output: Response text sent to the terminal

[0704] What it does: Sends text to the device using the HTTPS protocol.

[0705] Step 7:

[0706] The information received by the device is converted into audio using the Google Cloud Text-to-Speech API, and is displayed and audio notified.

[0707] Input: Formatted response text

[0708] Output: Visual and audio notifications

[0709] Specific behavior: Converts text to speech, displays it on the display, and plays it through the speaker.

[0710] Health Management and Emergency Response

[0711] Step 1:

[0712] The user says to the device, "I feel sick." The device recognizes the voice.

[0713] Input: User voice input

[0714] Output: Audio data

[0715] Specific operation: The device's microphone captures audio data.

[0716] Step 2:

[0717] The device enters emergency call mode and transmits profile data and status to the server.

[0718] Input: Audio data

[0719] Output: Profile and status data sent to the server

[0720] Specific operation: Activate emergency mode and send necessary data to the server.

[0721] Step 3:

[0722] The server retrieves the emergency contact information from the database and contacts the appropriate resource.

[0723] Input: Profile data and situational data

[0724] Output: Emergency contact

[0725] Specific behavior: Retrieves emergency contact information from a database and contacts emergency response services.

[0726] Step 4:

[0727] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called."

[0728] Input: Emergency Contact Information

[0729] Output: Feedback message

[0730] Specific operation: Make an emergency call and send the results to the device.

[0731] Step 5:

[0732] The device will notify the user by voice, "An ambulance has been called. Please remain calm."

[0733] Input: Feedback message

[0734] Output: Audio notification

[0735] Specific behavior: Convert the message into audio and notify the user.

[0736] Online shopping support

[0737] Step 1:

[0738] The user requests the device to "buy bread." The device recognizes the voice.

[0739] Input: User voice input

[0740] Output: Audio data

[0741] Specific operation: The device's microphone captures audio data.

[0742] Step 2:

[0743] The device uses the Google Cloud Speech-to-Text API to convert speech to text.

[0744] Input: Audio data

[0745] Output: Text data

[0746] Specific operation: Sends audio data to the API and retrieves text data.

[0747] Step 3:

[0748] The terminal sends the converted text to the server.

[0749] Input: Text data

[0750] Output: Text data sent to the server

[0751] Specific operation: To send text data to the server, the HTTPS protocol is used for communication.

[0752] Step 4:

[0753] The server analyzes the text data and searches for product lists from affiliated online shopping sites.

[0754] Input: Text data

[0755] Output: Product list

[0756] Specific operation: The server parses the text data, creates a query, and sends it to the shopping site's API.

[0757] Step 5:

[0758] The server performs sentiment analysis on the product list and generates it in the appropriate format.

[0759] Input: Product list

[0760] Output: A formatted product list

[0761] Specific operation: Format product list based on sentiment analysis results.

[0762] Step 6:

[0763] The server sends the product list to the terminal.

[0764] Input: Formatted product list

[0765] Output: Product list sent to the terminal

[0766] Specific operation: The product list is sent to the terminal using the HTTPS protocol.

[0767] Step 7:

[0768] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[0769] Input: Formatted product list

[0770] Output: Display and audio queries

[0771] Specific operation: Display a product list on the screen and ask questions by voice.

[0772] Step 8:

[0773] If the user answers "yes," the terminal sends the information to the server.

[0774] Input: Voice response

[0775] Output: Response sent to the server

[0776] Specific operation: The device converts the voice response into text and sends it to the server.

[0777] Step 9:

[0778] The server completes the final purchase and sends a confirmation message to the terminal.

[0779] Input: Response data

[0780] Output:Confirmation message

[0781] Specific operation: The server executes the purchase procedure and sends the results to the terminal.

[0782] Step 10:

[0783] The terminal will notify the user by displaying and audibly saying "Purchase completed."

[0784] Input:Confirmation message

[0785] Output: Visual and audio notifications

[0786] Specific operation: A confirmation message is displayed on the screen and played back as audio.

[0787] (Application example 2)

[0788] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0789] When elderly people shop alone in physical stores, they face the challenge of finding it difficult to obtain information and advice about products, making it difficult for them to enjoy shopping with peace of mind. Furthermore, due to a lack of individualized support tailored to the elderly's emotions, they often feel isolated and anxious. Furthermore, there is a need for a system that can respond quickly in emergencies. It is necessary to solve these problems and provide an environment where elderly people can enjoy shopping with peace of mind.

[0790] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0791] In this invention, the server includes: means for transmitting information input by a user to the server through a terminal and having the server generate a user profile based on the information; means for recognizing a voice input by the user and converting the voice to text; means for transmitting the text to the server and generating a response based on the text received by the server; means for the terminal to convert the response to voice and provide it to the user again; means for the server to search the Internet or a database for information corresponding to the user's request and provide it to the terminal; means for inputting information about the user's health condition and transmitting it to the server through the terminal, which then automatically contacts an emergency contact; means for transmitting information about a product desired by the user to the server through the terminal, which then searches for the product on affiliated e-commerce sites and completes the purchase process; and a smart brick-and-mortar store support application that has an engine for analyzing user emotions and provides responses and support based on the user's emotions. This allows elderly people to enjoy shopping with peace of mind and receive the information and support they need appropriately.

[0792] A "user profile" is an information structure that includes individual user characteristics and background information, and is generated based on information entered by the user.

[0793] "Voice input" is the process by which a device receives and analyzes what a user says as sound.

[0794] "Text conversion" is the process of analyzing information input via voice and converting it into text.

[0795] A "server" is a central processing unit that stores, processes, sends and receives data over a network.

[0796] "Response generation" is the process of creating information to respond to the user based on the received text.

[0797] An "emotion analysis engine" is software or an algorithm that determines emotions from input information and provides appropriate responses or support based on the results.

[0798] The "Smart Physical Store Support Application" is an application designed to support users' shopping experiences in physical stores, and is a system that handles user input, analyzes emotions, and provides necessary information.

[0799] An "Internet search" is the process of utilizing a wide range of networks to locate and provide information requested by a user.

[0800] "Emergency contact method" is a process in which emergency information such as the user's health condition is sent to a server, and the server automatically contacts the appropriate emergency contact.

[0801] "Product information search" is the process of finding and providing information about the product desired by the user from e-commerce sites and other sources.

[0802] This invention is a support system that allows elderly people to enjoy shopping in brick-and-mortar stores with peace of mind. In particular, it has the feature of recognizing the user's emotions and providing appropriate responses and support based on those emotions.

[0803] System Configuration

[0804] The system is composed of a terminal used by the user (smart glasses, smartphone, etc.), a server that performs central processing functions, sensors for acquiring data on the user's behavior and health status, etc. Specific embodiments of the present invention will be described below in order.

[0805] User Registration and Profiling

[0806] 1. User Registration:

[0807] The user starts up the device and registers basic information (such as name, age, address, medical history, etc.) by voice or manual input. This information is sent from the device to the server, which then creates a user profile and stores it in a database.

[0808] After the profile is generated, a confirmation message is sent back to the terminal and the user is notified audibly and visually.

[0809] Daily shopping assistance

[0810] 2. Voice to text conversion:

[0811] While shopping, users input voice commands into the device, such as "Tell me about this product," which are converted into text via speech recognition software such as the Google Speech-to-Text API.

[0812] The converted text is sent to the server using the HTTPS protocol.

[0813] 3. Sentiment analysis and response generation:

[0814] The server uses a sentiment analysis engine such as Amazon Comprehend to analyze sentiment from the user's text input.

[0815] Based on the results of emotion analysis, appropriate responses and support information are generated for the user and sent from the server to the terminal.

[0816] 4. Response provision:

[0817] The response text is converted into speech at the terminal and provided to the user in speech and text.

[0818] Emergency response and health management

[0819] 5. Emergency Response:

[0820] When a user tells the terminal that they are "feeling unwell," the terminal converts this speech into text and sends it to the server.

[0821] The server consults the user profile and emergency contact database and contacts the appropriate resources (such as emergency services).

[0822] Emergency response information is communicated to the user via the terminal by voice.

[0823] Searching and providing product information

[0824] 6. Product Search:

[0825] When a user types "Tell me about this product," the server searches for detailed information about the product from affiliated e-commerce sites, performs sentiment analysis on the results, and then provides them to the user.

[0826] Examples of concrete examples and prompts

[0827] For example, when a user speaks to a terminal in a store and says, "Tell me about this product," the terminal converts the speech into text and sends it to the server. The server analyzes the text and the user's emotions and generates appropriate product information and recommendations. This process allows the user to receive a response that reflects their emotions, allowing them to enjoy shopping with peace of mind.

[0828] Prompt Sentence Examples

[0829] “When a user says, ‘Tell me about this product,’ provide a product description and offer recommendations based on the user’s emotions. For example, if the user seems anxious, highlight how the product will help them.”

[0830] By utilizing generative AI models and prompts, this system provides an environment where seniors can enjoy shopping in physical stores with peace of mind.

[0831] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0832] Step 1:

[0833] User Registration:

[0834] The user starts up the device and registers basic information (name, age, address, medical history, etc.) by voice input or manual input.

[0835] The device converts this input information into text, formats it as JSON, and sends it to the server using the HTTPS protocol.

[0836] The server parses the received JSON data, generates a user profile, and saves it in the database.

[0837] Based on the generated profile, the server creates a confirmation message and sends it to the terminal.

[0838] The device will notify the user of a confirmation message by voice and text.

[0839] Step 2:

[0840] Speech to text:

[0841] While shopping, the user issues a voice command such as "Tell me about this product."

[0842] The device converts voice input into text using the Google Speech-to-Text API.

[0843] This text data is sent to the server using the HTTPS protocol.

[0844] Step 3:

[0845] Sentiment analysis and response generation:

[0846] The server analyzes the user's emotions from the received text data using an emotion analysis engine such as Amazon Comprehend.

[0847] Based on the sentiment analysis results and the text, the server generates a prompt sentence, which is then used to generate an appropriate response text.

[0848] The response text might be generated in the form of, for example, "This product contains healthy ingredients and is ideal for the elderly."

[0849] Step 4:

[0850] Response provided:

[0851] The server generates a response text and sends it to the terminal.

[0852] The terminal converts the received response text into speech and provides it to the user in speech and text.

[0853] Step 5:

[0854] Product Search:

[0855] The user again instructs the terminal to "tell me about this product."

[0856] The device converts the speech into text and sends the text to the server.

[0857] The server uses the text to search for product information using the API of the e-commerce site.

[0858] The server retrieves the search results, simultaneously performs sentiment analysis, and provides information in a format that corresponds to the user's emotions.

[0859] The server sends this product information to the terminal, and the terminal informs the user by voice and text, "The price of this product is XXX yen and it is very popular."

[0860] Step 6:

[0861] Emergency Response:

[0862] The user tells the terminal that he is "feeling unwell."

[0863] The device converts the voice into text and sends it to the server as emergency information.

[0864] The server consults the user profile and emergency contact database and contacts the appropriate emergency contact (eg, emergency services).

[0865] The server generates progress information on the emergency response and transmits it to the terminal.

[0866] The terminal will tell the user by voice, "An ambulance has been called. Please wait without worry."

[0867] Through these processing steps, the system can quickly and accurately provide the information and support necessary for seniors to enjoy shopping safely in physical stores.

[0868] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0869] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0870] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0871] [Second embodiment]

[0872] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0873] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0874] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0875] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0876] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0877] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0878] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0879] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0880] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0881] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0882] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0883] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0884] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone. This system functions by interoperating with three entities: a server, a terminal, and a user.

[0885] Program processing details

[0886] 1. User Registration and Profiling

[0887] The user turns on the device (e.g., a smartphone or tablet) and speaks, "I would like to register my information."

[0888] The device converts the voice to text and displays a screen for entering information.

[0889] The user enters basic information such as name, age, address, and medical history.

[0890] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[0891] The server parses the received JSON data and stores the information in the database.

[0892] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[0893] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[0894] 2. Daily communication

[0895] In the morning, the user says "Good morning" to the terminal.

[0896] The device converts the speech into text and sends the text to the server.

[0897] The server receives the text and generates a response text saying "Good morning. The weather is sunny today."

[0898] The server generates a response and sends it to the terminal.

[0899] The terminal converts the received response text into speech and tells the user, "Good morning. The weather is sunny today."

[0900] 3. Searching for and providing information

[0901] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[0902] The device converts the speech into text and sends the text to the server.

[0903] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[0904] The server sends the contact information it finds to the device.

[0905] The information received by the terminal is displayed and output to the user as audio, such as "The contact information for the nearest welfare center is XXX-XXXX-XXXX."

[0906] 4. Health Management and Emergency Response

[0907] The user tells the terminal that he is "feeling unwell."

[0908] The device detects this sound and goes into emergency call mode.

[0909] The device sends profile data (such as medical history) and status to the server.

[0910] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[0911] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called."

[0912] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[0913] 5. Online shopping support

[0914] The user makes a request to the terminal saying, "I want to buy bread."

[0915] The device converts the speech into text and sends it to the server.

[0916] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[0917] The server sends a list of the found products to the terminal.

[0918] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[0919] If the user answers "yes," the terminal sends the information to the server.

[0920] The server completes the final purchase and sends a confirmation message to the terminal.

[0921] The terminal will display and audibly inform the user that "Purchase completed."

[0922] Specific examples

[0923] For example, when a user wakes up in the morning and says "Good morning" to the device, the device converts the speech into text and sends it to the server. The server generates a response, "Good morning. The weather is sunny today," and sends it back to the device. The device then converts this response into speech and conveys it to the user. This series of steps allows users to engage in everyday communication without feeling isolated.

[0924] This system is divided into detailed processing steps, and by performing specific actions at each step, it provides multifaceted support for elderly people living alone.

[0925] The processing flow will be explained below.

[0926] User Registration and Profiling

[0927] Step 1:

[0928] The user starts up the device and says, "I'd like to register my information."

[0929] Your device uses speech recognition to convert speech into text.

[0930] Step 2:

[0931] The terminal analyzes the converted text and displays a screen for entering user information.

[0932] Step 3:

[0933] The user enters basic information such as name, age, address, and medical history.

[0934] Step 4:

[0935] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[0936] Step 5:

[0937] The server parses the received JSON data and stores the information in the database.

[0938] Step 6:

[0939] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[0940] Step 7:

[0941] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[0942] Daily communication

[0943] Step 1:

[0944] In the morning, the user says "Good morning" to the terminal.

[0945] Step 2:

[0946] The device converts the speech into text and sends the text to the server.

[0947] Step 3:

[0948] The server receives the text and generates a response text saying "Good morning. The weather is sunny today."

[0949] Step 4:

[0950] The server generates a response and sends it to the terminal.

[0951] Step 5:

[0952] The terminal converts the received response text into speech and tells the user, "Good morning. The weather is sunny today."

[0953] Searching for and providing information

[0954] Step 1:

[0955] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[0956] Step 2:

[0957] The device converts the speech into text and sends the text to the server.

[0958] Step 3:

[0959] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[0960] Step 4:

[0961] The server sends the contact information it finds to the device.

[0962] Step 5:

[0963] The information received by the terminal is displayed and output to the user as audio, such as "The contact information for the nearest welfare center is XXX-XXXX-XXXX."

[0964] Health Management and Emergency Response

[0965] Step 1:

[0966] The user tells the terminal that he is "feeling unwell."

[0967] Step 2:

[0968] The device detects this sound and goes into emergency call mode.

[0969] Step 3:

[0970] The device sends profile data (such as medical history) and status to the server.

[0971] Step 4:

[0972] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[0973] Step 5:

[0974] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called."

[0975] Step 6:

[0976] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[0977] Online shopping support

[0978] Step 1:

[0979] The user makes a request to the terminal saying, "I want to buy bread."

[0980] Step 2:

[0981] The device converts the speech into text and sends it to the server.

[0982] Step 3:

[0983] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[0984] Step 4:

[0985] The server sends a list of the found products to the terminal.

[0986] Step 5:

[0987] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[0988] Step 6:

[0989] If the user answers "yes," the terminal sends the information to the server.

[0990] Step 7:

[0991] The server completes the final purchase and sends a confirmation message to the terminal.

[0992] Step 8:

[0993] The terminal will display and audibly inform the user that "Purchase completed."

[0994] By going through the above steps, this system can provide multifaceted support to elderly people living alone.

[0995] Example 1

[0996] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0997] Elderly people living alone face a wide range of challenges, including daily communication, health management, and emergency response. To efficiently and comprehensively address these challenges, an easy-to-use system is required. However, current technology provides solutions for individual problems in a fragmented manner, preventing a consistent user experience.

[0998] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0999] In this invention, the server includes means for transmitting information input by a user to the server via a terminal and for the server to generate a user profile based on the information, means for recognizing a voice input by the user and converting the voice into text, and means for transmitting the text to the server and generating a response based on the text received by the server, thereby enabling multifaceted support for elderly people in their daily activities such as communication, emergency response, health management, and online shopping.

[1000] "Users" refers to the elderly people who use the system and the people who support them.

[1001] A "terminal" is a device operated by a user, and refers to information and communication devices such as smartphones and tablets.

[1002] "Server" refers to a central computing device that processes information sent from terminals and manages it in a database.

[1003] A "user profile" refers to a data set that includes personal information such as a user's basic information and medical history.

[1004] "Voice input" refers to voice commands given by a user through a microphone.

[1005] "Speech recognition" refers to the technology that converts voice input into text form.

[1006] "Text conversion" refers to the process of processing text obtained by speech recognition as data.

[1007] "Secure Protocol" refers to a secure communications protocol such as HTTPS.

[1008] "Generate" refers to the process by which the server generates text or audio information.

[1009] "Search" refers to the operation by which a server retrieves information using the Internet or an internal database.

[1010] "Emergency contact" refers to the procedure by which the server automatically notifies pre-defined contacts in the event of an emergency.

[1011] "Shopping Site" refers to the affiliated e-commerce platform.

[1012] "Voice notification" refers to a function in which a terminal uses a voice conversion function to convey information to a user.

[1013] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone. This system functions by interoperating with three entities: a server, a terminal, and a user.

[1014] First, the user uses a smartphone or tablet as a device. These devices have speech recognition capabilities and convert the user's voice input into text using the Google Speech-to-Text API. The device also converts text into speech using the Google Text-to-Speech API.

[1015] Next, the information entered by the user is sent to the server via the terminal. For example, when a user enters basic information such as name, age, address, and medical history, this information is converted into JSON format and sent to the server using the HTTPS protocol. The server analyzes the received information, stores it in a MySQL database, and generates a user profile.

[1016] In everyday communication, when a user speaks to a device, for example, "Good morning," the device converts the speech into text and sends it to the server. The server generates a response text and sends it to the device. The device then converts the received response text into speech and conveys it to the user. For example, "Good morning. The weather is sunny today."

[1017] Also, when a user says to the device, "Please tell me the contact information for the nearest welfare center," the device converts the voice into text and sends that text to the server. The server uses an internal database or an external API (such as Google Maps API) to search for the welfare center's contact information and sends that information to the device. The device then converts the received information into voice and relays it to the user.

[1018] Furthermore, if the user tells the device that they are "feeling unwell," the device will enter emergency call mode and send their profile data and status to the server. The server will retrieve emergency contact information from a database and contact the appropriate resource (e.g., emergency services). The server will then send a confirmation message to the device saying "an ambulance has been called," which the device will then verbally convey to the user.

[1019] For online shopping support, when a user requests the device to "buy bread," the device converts the voice to text and sends it to the server. The server then searches for a product list from affiliated shopping sites and sends it to the device. The device then displays the product list to the user and asks aloud, "Is this OK?" If the user answers "yes," the device completes the final purchase procedure and a confirmation message is spoken by the device.

[1020] Specific examples

[1021] For example, if a user says "Good morning" to a device in the morning, the device converts the speech into text and sends it to the server. The server generates a response such as "Good morning. The weather is sunny today" and sends it back to the device. The device then converts this response into speech and conveys it to the user. This series of steps allows users to engage in everyday communication without feeling isolated.

[1022] Examples of prompts include "Good morning," "Please tell me the contact information of the nearest welfare center," "I'm not feeling well," "I want to buy some bread," etc. Through such prompts, it is possible to provide the multifaceted support that the elderly require in their daily lives.

[1023] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1024] Program processing flow

[1025] User Registration and Profiling

[1026] Step 1:

[1027] The user starts up the device and says, "I'd like to register my information."

[1028] Input: Audio

[1029] Output: Audio data

[1030] Specific operation: The user uses a smartphone or tablet to input voice information.

[1031] Step 2:

[1032] The device converts the speech to text.

[1033] Input: Audio data

[1034] Output: Text data

[1035] What happens: The device converts speech to text using the Google Speech-to-Text API.

[1036] Step 3:

[1037] The user enters basic information such as name, age, address, and medical history.

[1038] Input: User basic information

[1039] Output: User basic information data

[1040] Specific operation: The user enters information into an input form displayed on the device screen.

[1041] Step 4:

[1042] The device converts the information into JSON format.

[1043] Input: User basic information data

[1044] Output: JSON data

[1045] Specific operation: The terminal converts the input data into JSON format.

[1046] Step 5:

[1047] The terminal sends data to the server.

[1048] Input: JSON data

[1049] Output: Send data to the server

[1050] Specific operation: The device sends data to the server using the HTTPS protocol.

[1051] Step 6:

[1052] The server analyzes the data and stores it in a database.

[1053] Input: JSON data

[1054] Output: Store in database

[1055] Specific operation: The server analyzes the received data and stores it in a MySQL database.

[1056] Step 7:

[1057] The server generates a confirmation message and sends it to the terminal.

[1058] Input: Database update information

[1059] Output:Confirmation message

[1060] Specific operation: The server creates a message saying "Profile has been generated" and sends it to the terminal.

[1061] Step 8:

[1062] The device will notify the user by voice.

[1063] Input:Confirmation message

[1064] Output: Audio notification

[1065] Specific behavior: The device uses the Google Text-to-Speech API to convert the text to speech and notifies the user that "Registration is complete."

[1066] Daily communication

[1067] Step 1:

[1068] The user says "Good morning" to the terminal.

[1069] Input: Audio

[1070] Output: Audio data

[1071] Specific operation: The user greets the device by voice.

[1072] Step 2:

[1073] The device converts the speech to text.

[1074] Input: Audio data

[1075] Output: Text data

[1076] What happens: The device converts speech to text using the Google Speech-to-Text API.

[1077] Step 3:

[1078] The device sends the text to the server.

[1079] Input: Text data

[1080] Output: Send text data to the server

[1081] Specific operation: The device sends text data to the server via HTTP.

[1082] Step 4:

[1083] The server parses the text and generates a response.

[1084] Input: Text data

[1085] Output: Response text

[1086] Specific operation: The server analyzes the received text data and generates a response saying, "Good morning. The weather is sunny today."

[1087] Step 5:

[1088] The server sends the response text to the terminal.

[1089] Input: Response text

[1090] Output: Response sent from the server

[1091] Specific operation: The response text generated by the server is sent to the terminal.

[1092] Step 6:

[1093] The terminal converts the response into voice and conveys it to the user.

[1094] Input: Response text

[1095] Output: Audio notification

[1096] What happens: The device uses the Google Text-to-Speech API to convert text to speech and tells the user, "Good morning. The weather is sunny today."

[1097] Searching for and providing information

[1098] Step 1:

[1099] The user speaks to the device, saying, "Please tell me the contact information for the nearest welfare center."

[1100] Input: Audio

[1101] Output: Audio data

[1102] Specific behavior: The user makes a request by voice.

[1103] Step 2:

[1104] The device converts the speech to text.

[1105] Input: Audio data

[1106] Output: Text data

[1107] What happens: The device converts speech to text using the Google Speech-to-Text API.

[1108] Step 3:

[1109] The device sends the text to the server.

[1110] Input: Text data

[1111] Output: Send data to the server

[1112] Specific operation: The device sends text data to the server via HTTP.

[1113] Step 4:

[1114] The server retrieves the information.

[1115] Input: Text data

[1116] Output: Search results

[1117] Specific operation: The server searches for "welfare center contact information" using its internal database and Google Maps API.

[1118] Step 5:

[1119] The server sends the found information to the device.

[1120] Input: Search results

[1121] Output: Information sent from the server

[1122] Specific operation: The server sends the search results to the terminal.

[1123] Step 6:

[1124] The device converts the information into voice and conveys it to the user.

[1125] Input: Search results

[1126] Output: Audio notification

[1127] What it does: The device uses the Google Text-to-Speech API to convert the text into speech and tells the user, "The contact information for the nearest welfare center is XXX-XXXX-XXXX."

[1128] Health Management and Emergency Response

[1129] Step 1:

[1130] The user tells the terminal that he is "feeling unwell."

[1131] Input: Audio

[1132] Output: Audio data

[1133] Specific operation: The user vocally complains about feeling unwell.

[1134] Step 2:

[1135] The device converts the speech to text.

[1136] Input: Audio data

[1137] Output: Text data

[1138] What happens: The device converts speech to text using the Google Speech-to-Text API.

[1139] Step 3:

[1140] The device enters emergency call mode.

[1141] Input: Text data

[1142] Output: Emergency call mode activated

[1143] Specific operation: The device receives the text data and activates emergency call mode.

[1144] Step 4:

[1145] The device sends profile data and status to the server.

[1146] Input: Profile data, text data

[1147] Output: Send data to the server

[1148] Specific operation: The device sends profile data and status to the server.

[1149] Step 5:

[1150] The server retrieves the emergency contact information.

[1151] Input: Profile data

[1152] Output: Emergency contact information

[1153] Specific operation: The server retrieves emergency contact information from the database.

[1154] Step 6:

[1155] The server contacts emergency services.

[1156] Input: Emergency Contact Information

[1157] Output: Emergency contact execution

[1158] Specific Actions: The server contacts emergency services.

[1159] Step 7:

[1160] The server sends a confirmation message to the terminal.

[1161] Input: Contact status

[1162] Output:Confirmation message

[1163] Specific operation: The server sends a confirmation message to the terminal saying "An ambulance has been called."

[1164] Step 8:

[1165] The device will notify the user by voice.

[1166] Input:Confirmation message

[1167] Output: Audio notification

[1168] What happens: The device uses the Google Text-to-Speech API to convert the text into speech and notifies the user, "An ambulance has been called. Please remain calm."

[1169] Online shopping support

[1170] Step 1:

[1171] The user makes a request to the terminal saying, "I want to buy bread."

[1172] Input: Audio

[1173] Output: Audio data

[1174] Specific operation: The user expresses their purchase intention by voice.

[1175] Step 2:

[1176] The device converts the speech to text.

[1177] Input: Audio data

[1178] Output: Text data

[1179] What happens: The device converts speech to text using the Google Speech-to-Text API.

[1180] Step 3:

[1181] The device sends the text to the server.

[1182] Input: Text data

[1183] Output: Send data to the server

[1184] Specific operation: The device sends text data to the server via HTTP.

[1185] Step 4:

[1186] The server parses the text and searches for products.

[1187] Input: Text data

[1188] Output: Product list

[1189] Specific operation: The server searches for a list of bread products using the API of a shopping site it is affiliated with.

[1190] Step 5:

[1191] The server sends the product list to the terminal.

[1192] Input: Product list

[1193] Output: List sent from server

[1194] Specific operation: The server sends the list of found products to the terminal.

[1195] Step 6:

[1196] The device displays a list of products and asks aloud, "Is this okay?"

[1197] Input: Product list

[1198] Output: Display, voice prompts

[1199] Specific operation: The device displays a list of products to the user and asks aloud, "Is this okay?"

[1200] Step 7:

[1201] The user responds "Yes."

[1202] Input: Audio

[1203] Output: Audio data

[1204] What happens: The user confirms the purchase and responds verbally.

[1205] Step 8:

[1206] The terminal sends this response to the server.

[1207] Input: Audio data

[1208] Output: Text data

[1209] Specific operation: The device converts the voice data into text and sends it to the server.

[1210] Step 9:

[1211] The server completes the final purchase and sends a confirmation message to the terminal.

[1212] Input: Text data

[1213] Output:Confirmation message

[1214] Specific operation: The server performs the final purchase procedure, generates a message saying "Purchase completed" and sends it to the terminal.

[1215] Step 10:

[1216] The device will notify you by voice that the purchase is complete.

[1217] Input:Confirmation message

[1218] Output: Audio notification

[1219] What happens: The device uses the Google Text-to-Speech API to convert the text to speech and tells the user, "Your purchase is complete."

[1220] (Application example 1)

[1221] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1222] The lack of comprehensive support systems for the various problems that elderly people face in their daily lives is a challenge. In particular, systems that can quickly respond when elderly people order meals or in emergencies are needed, but the current means to achieve this are insufficient. Furthermore, interactive interfaces are also needed to reduce the burden that elderly people feel when performing these tasks alone.

[1223] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1224] In this invention, the server includes: means for transmitting information input by a user to the server through a terminal and for the server to generate a user profile based on the information; means for recognizing a voice input by the user and converting the voice to text; means for transmitting the text to the server and generating a response based on the text received by the server; means for the terminal to convert the response to voice and provide it to the user again; means for the server to search the Internet or a database for information corresponding to the user's request and provide it to the terminal; means for transmitting an order for a meal desired by the user to the server through the terminal, searching for product lists from affiliated food delivery services with the server, and completing the purchase procedure; means for the server to check the delivery status and obtain information and send it to the terminal when the user says "Tell me the delivery status" to the terminal; and means for the terminal to enter emergency response mode when the user says to the terminal that they are feeling unwell, notifying emergency contacts via the server, and arranging appropriate responses. This enables elderly people to easily meet various needs in their daily lives and respond to emergencies.

[1225] 1. A "terminal" is an electronic device that allows users to input information or give voice instructions. Examples include smartphones, tablets, and robots.

[1226] 2. A "server" is a computer system that receives data sent from a terminal via the Internet or a network, performs the necessary processing, and then sends a response back to the terminal.

[1227] 3. A "user profile" is a set of detailed data such as a user's personal information, past behavioral history, preferences, and allergy information.

[1228] 4. "Voice input" means the user speaks through a microphone, which is converted into text by the system.

[1229] 5. "Text conversion" is the process of analyzing voice input and converting it into text information.

[1230] 6. "Response" means a reply message generated by the server based on received text and profile information.

[1231] 7. "Voice output" is the process of converting the generated response text from text information back into voice and delivering it to the user.

[1232] 8. "Internet" means a worldwide collection of computer networks that enable the transmission and reception of information.

[1233] 9. A "database" is a collection of structured data that organizes large amounts of information and allows for quick search and retrieval.

[1234] 10. "Food delivery service" means a service that allows a user to order a meal they want and delivers the meal to a location specified by the user.

[1235] 11. "Emergency Response Mode" is a function that allows the system to automatically take special action and notify emergency contacts when a user reports a health problem.

[1236] 12. "Purchase procedure" refers to a series of processes for purchasing products from affiliated shopping sites based on information about the products desired by the user.

[1237] 13. "Delivery status" is information that indicates the current location of the ordered meal or product and how long it will take to arrive at the user's location.

[1238] 14. "Application" means a specific example of applying this invention to a specific field or use.

[1239] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone, and it functions by interoperating with three main entities: a terminal, a server, and a user. The system has a wide range of functions, including user profile generation, voice input recognition, text conversion, response generation, information search, food ordering, delivery status tracking, and emergency response.

[1240] 1. User profiling

[1241] Users input information through devices such as smartphones and tablets. The devices convert the input speech into text and send it to a server in JSON format using the HTTPS protocol. The server analyzes the data and creates a user profile including name, age, address, medical history, dietary preferences, and allergy information, and stores it in a database.

[1242] 2. Voice Input and Response Generation

[1243] When a user says "I'd like to order lunch" to the device, the device converts the speech into text and sends it to the server. The server analyzes the received text and generates a list of recommended menu items based on past order history and profile information. This menu list is sent back to the device, which then asks aloud, "Is this OK?" This allows the user to complete their meal order using only the voice interface.

[1244] 3. Track your delivery status

[1245] When the user speaks to the device saying, "Tell me the delivery status," the device converts the speech into text and sends it to the server. The server obtains the delivery status from the partner food delivery service and sends the real-time status to the device. The device then notifies the user by notifying them, for example, "The delivery person will arrive in 5 minutes."

[1246] 4. Emergency Response

[1247] When a user notifies the device that they are "feeling unwell," the device enters emergency response mode and sends an emergency notification along with their profile data to the server. The server notifies emergency contacts and arranges for the necessary response. The device then notifies the user by voice, saying, "An ambulance has been called. Please remain calm."

[1248] Hardware and software used

[1249] Hardware: smartphones, tablets, robots

[1250] software:

[1251] Natural Language Processing: Google Cloud Speech-to-Text

[1252] Data transmission / reception: HTTPS protocol

[1253] Database: MySQL

[1254] Server: AWS EC2

[1255] Speech synthesis: Google Cloud Text-to-Speech

[1256] Specific examples

[1257] Consider a scenario where an elderly person wants to order lunch. When the user says "I'd like to order lunch," the following dialogue takes place:

[1258] User: "I'd like to order lunch."

[1259] Device: "The recommended menu items registered are curry rice, sandwiches, and udon. Which would you like?"

[1260] User: "Curry rice"

[1261] Terminal: "I ordered curry rice."

[1262] Example prompt sentence:

[1263] "A user is trying to order lunch using a food delivery assistance assistant for seniors. Generate a series of dialogues with corresponding menu choices."

[1264] This will enable elderly people to easily meet various needs in their daily lives and respond to emergencies.

[1265] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1266] Step 1:

[1267] The user turns on a device such as a smartphone or tablet and says, "I would like to register as a user." The device converts the voice to text and displays an information entry screen. The user enters basic information such as name, address, dietary preferences, and allergy information, and the device converts this information into JSON format. This data is sent to the server using the HTTPS protocol.

[1268] Input: User voice input "I would like to register as a user", basic information

[1269] Output: JSON formatted data sent to the server

[1270] Step 2:

[1271] The server analyzes the received JSON data and generates a user profile including the user's name, age, address, medical history, dietary preferences, and allergy information. The generated user profile is stored in a database. The server confirms that the profile has been successfully generated and notifies the device that "The profile has been generated."

[1272] Input: User information in JSON format

[1273] Output: User profile stored in database, notification to terminal

[1274] Step 3:

[1275] The user speaks to the device, saying, "I'd like to order lunch." The device converts the speech into text and sends it to the server. The server analyzes the text and generates a list of recommended menu items based on past order history and user profile information. This menu list is then sent to the device.

[1276] Input: User speaks "I want to order lunch"

[1277] Output: Menu list sent to the terminal

[1278] Step 4:

[1279] The terminal displays a menu list and asks the user aloud, "Is this OK?" If the user answers "Yes," the terminal sends the information to the server, which then processes the final order. The server then sends a confirmation message to the terminal, informing the user that the order has been completed.

[1280] Input: User voice input "Yes"

[1281] Output: Order information sent to the server, confirmation message sent to the terminal

[1282] Step 5:

[1283] The user speaks to the device, saying, "Tell me the delivery status." The device converts the speech into text and sends it to the server. The server obtains the delivery status from the partner food delivery service and sends the real-time information to the device. The device then notifies the user, "The delivery person will arrive in 5 minutes."

[1284] Input: User's voice input "Tell me the delivery status"

[1285] Output: Delivery status information sent to the terminal

[1286] Step 6:

[1287] The user says "I'm feeling unwell" to the device. The device converts the speech to text and enters emergency response mode. It sends an emergency notification along with the profile data to the server. The server notifies emergency contacts and arranges for the necessary response. The device then tells the user by voice, "An ambulance has been called. Please remain calm."

[1288] Input: User's voice input "I feel sick"

[1289] Output: Notification to emergency contacts, feedback to device

[1290] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1291] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone, and is equipped with a function that recognizes the user's emotions and provides appropriate responses and support based on those emotions. This system functions in cooperation with three entities: a server, a terminal, and a user, and combines an emotion engine to improve the user experience.

[1292] Program processing details

[1293] User Registration and Profiling

[1294] The user starts up the device and says, "I'd like to register my information."

[1295] Your device uses speech recognition to convert speech into text.

[1296] The terminal analyzes the converted text and displays a screen for entering user information.

[1297] The user enters basic information such as name, age, address, and medical history.

[1298] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[1299] The server parses the received JSON data and stores the information in the database.

[1300] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[1301] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[1302] Daily communication

[1303] In the morning, the user says "Good morning" to the terminal.

[1304] The device converts the speech into text and sends the text to the server.

[1305] The server receives the text and analyzes the user's emotions using an emotion engine.

[1306] Based on the analysis results, the server generates a response text saying, "Good morning. Today's weather is sunny and it looks like it's going to be a pleasant day."

[1307] The server generates a response and sends it to the terminal.

[1308] The terminal converts the received response text into speech and tells the user, "Good morning. The weather is sunny today, and it looks like it's going to be a pleasant day."

[1309] Searching for and providing information

[1310] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[1311] The device converts the speech into text and sends the text to the server.

[1312] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[1313] The server analyzes the contact information it finds using an emotion engine and presents it in a format that corresponds to the user's emotions.

[1314] The server sends the information to the terminal.

[1315] The information received by the device is displayed and output to the user as audio, saying, "The contact number for the nearest welfare center is XXX-XXXX-XXXX. If you have any concerns, you can contact them immediately."

[1316] Health Management and Emergency Response

[1317] The user tells the terminal that he is "feeling unwell."

[1318] The device detects this sound and goes into emergency call mode.

[1319] The device sends profile data (such as medical history) and status to the server.

[1320] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[1321] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called. Please wait a moment."

[1322] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[1323] Online shopping support

[1324] The user makes a request to the terminal saying, "I want to buy bread."

[1325] The device converts the speech into text and sends it to the server.

[1326] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[1327] The server analyzes the found product list using an emotion engine and presents it in an appropriate format.

[1328] The server sends the product list to the terminal.

[1329] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[1330] If the user answers "yes," the terminal sends the information to the server.

[1331] The server completes the final purchase and sends a confirmation message to the terminal.

[1332] The terminal will display and audibly inform the user that "Purchase completed."

[1333] Specific examples

[1334] For example, if a user says "Good morning" to a device in the morning, the device converts the speech into text and sends it to the server. The server analyzes the text and the user's emotions, and generates a response that best suits the user's state of mind. This process allows the user to receive a response that reflects their emotions, enabling everyday communication while reducing feelings of isolation.

[1335] By combining this system with an emotion engine, it provides multifaceted support for elderly people living alone and offers more personalized services.

[1336] The processing flow will be explained below.

[1337] User Registration and Profiling

[1338] Step 1:

[1339] The user starts up the device and says, "I'd like to register my information."

[1340] Your device uses speech recognition to convert speech into text.

[1341] Step 2:

[1342] The terminal analyzes the converted text and displays a screen for entering user information.

[1343] Step 3:

[1344] The user enters basic information such as name, age, address, and medical history.

[1345] Step 4:

[1346] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[1347] Step 5:

[1348] The server parses the received JSON data and stores the information in the database.

[1349] Step 6:

[1350] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[1351] Step 7:

[1352] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[1353] Daily communication

[1354] Step 1:

[1355] In the morning, the user says "Good morning" to the terminal.

[1356] Step 2:

[1357] The device converts the speech into text and sends the text to the server.

[1358] Step 3:

[1359] The server receives the text and analyzes the user's emotions using an emotion engine.

[1360] Step 4:

[1361] Based on the results of the sentiment analysis, the server generates a response text such as "Good morning. The weather is sunny today and you look like you're in a good mood."

[1362] Step 5:

[1363] The server generates a response and sends it to the terminal.

[1364] Step 6:

[1365] The device converts the received response text into speech and tells the user, "Good morning. The weather is sunny today and you look like you're in a good mood."

[1366] Searching for and providing information

[1367] Step 1:

[1368] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[1369] Step 2:

[1370] The device converts the speech into text and sends the text to the server.

[1371] Step 3:

[1372] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[1373] Step 4:

[1374] The server analyzes the contact information it finds using an emotion engine and creates information in a format that corresponds to the user's emotions.

[1375] Step 5:

[1376] The server sends the information to the terminal.

[1377] Step 6:

[1378] The information received by the terminal is displayed and output to the user as audio, saying, "The contact number for the nearest welfare center is XXX-XXXX-XXXX. Please feel free to contact us at any time."

[1379] Health Management and Emergency Response

[1380] Step 1:

[1381] The user tells the terminal that he is "feeling unwell."

[1382] Step 2:

[1383] The device detects this sound and goes into emergency call mode.

[1384] Step 3:

[1385] The device sends profile data (such as medical history) and status to the server.

[1386] Step 4:

[1387] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[1388] Step 5:

[1389] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called. Please wait a moment."

[1390] Step 6:

[1391] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[1392] Online shopping support

[1393] Step 1:

[1394] The user makes a request to the terminal saying, "I want to buy bread."

[1395] Step 2:

[1396] The device converts the speech into text and sends it to the server.

[1397] Step 3:

[1398] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[1399] Step 4:

[1400] The server analyzes the found product list using an emotion engine and presents it in an appropriate format.

[1401] Step 5:

[1402] The server sends the product list to the terminal.

[1403] Step 6:

[1404] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[1405] Step 7:

[1406] If the user answers "yes," the terminal sends the information to the server.

[1407] Step 8:

[1408] The server completes the final purchase and sends a confirmation message to the terminal.

[1409] Step 9:

[1410] The terminal will display and audibly inform the user that "Purchase completed."

[1411] Through these steps, the system can provide multifaceted support for elderly people living alone and provide services that take users' emotions into consideration.

[1412] Example 2

[1413] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1414] In modern society, the number of elderly people living alone is increasing, making support for daily life and emergency response important issues. Furthermore, providing appropriate responses according to emotions is required to reduce the sense of isolation felt by elderly people. Conventional speech recognition systems have difficulty analyzing a user's emotions and generating appropriate responses. Furthermore, their ability to provide information needed for daily life and emergency response is limited. Therefore, a comprehensive system is needed that can provide multifaceted support for the daily life of elderly people and provide appropriate responses according to their emotions.

[1415] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for transmitting information input by a user to the information processing device via a terminal, and the information processing device generating a user profile based on the information; means for recognizing a voice input by the user and converting the voice into text; means for transmitting the text to the information processing device and generating a response based on the received text and emotion analysis; means for the terminal to convert the response into speech and provide it to the user again; means for the information processing device to search for information corresponding to the user's request from a network or database and provide it to the terminal; means for inputting information about the user's health condition, transmitting it to the information processing device via the terminal, and automatically contacting an emergency contact; and means for transmitting information about a product desired by the user via the terminal to the information processing device, and the information processing device searching for the product on a related e-commerce site and completing a purchase procedure. This makes it possible to comprehensively support the daily lives of elderly people and provide appropriate responses according to their emotions.

[1416] "Information processing device" refers to a device or system that inputs, processes, stores, and outputs data. Generally, this includes servers and computers.

[1417] "User profile" refers to a user's personal information data, including basic information such as the user's name, age, address, medical history, and sentiment analysis results.

[1418] "Speech recognition" refers to the technology of analyzing a user's voice input and converting it into text data.

[1419] "Emotion analysis" refers to the technology of analyzing a user's emotional state from text data and voice data.

[1420] "Response generation" refers to the process of creating an appropriate response to the user based on the analyzed information.

[1421] A "network" refers to a system that connects multiple computers and devices and enables the exchange of data.

[1422] A "database" refers to a system for efficiently managing, searching, and storing data.

[1423] "Encryption protocol" refers to the technology used to encrypt data in order to ensure secure data communication.

[1424] "Emergency contact" refers to contact information that is set to be contacted in the event of a sudden change in the user's health condition or an emergency.

[1425] "E-commerce site" refers to a website that allows users to sell or purchase goods or services over the Internet.

[1426] This invention is a comprehensive service system that provides multifaceted support for elderly people living alone. Its key feature is its ability to recognize the user's emotions and provide appropriate responses and support based on those emotions. This system functions in coordination with three entities: a server, a terminal, and a user, and combines an emotion engine to enhance the user experience. Below, we will explain in detail each component of this system and its operation.

[1427] Hardware and Software Use

[1428] This system uses several key hardware and software components. The terminal is a device that performs speech recognition, speech synthesis, and display. Specifically, it is a smart device equipped with a microphone, speaker, and display. The system uses the Google Cloud Speech-to-Text API for speech recognition and the Google Cloud Text-to-Speech API for speech synthesis. It also uses the IBM Watson Tone Analyzer for user sentiment analysis.

[1429] The server manages user profiles, analyzes emotions, generates responses, manages databases, and performs network communications. The server is a computer with high-performance computing capabilities and uses MySQL as its database. The HTTPS protocol is used as the communications protocol to ensure security.

[1430] System Operation

[1431] User Registration and Profiling

[1432] The user turns on the device and says, "I'd like to register my information." The device uses the Google Cloud Speech-to-Text API to convert the speech to text and analyzes that text. A user information input screen is displayed, and the user enters information such as their name, age, address, and medical history. The device converts this information into JSON format and sends it to the server using the HTTPS protocol. The server analyzes the received data and stores it in a MySQL database. Once the profile is generated, a confirmation message is displayed on the device and a voice notification is played saying, "Registration is complete."

[1433] Daily communication

[1434] In the morning, when a user says "Good morning" to the device, the device converts the speech into text and sends it to the server. The server performs text and sentiment analysis and generates an appropriate response. The generated response is sent to the device, which converts it into speech and conveys it to the user. For example, the server might generate a response like "Good morning. The weather is sunny today, and it looks like it's going to be a pleasant day."

[1435] Searching for and providing information

[1436] When a user says to the device, "Tell me the contact information for the nearest welfare center," the device converts the speech into text and sends it to the server. The server analyzes the text, searches for contact information using a database or external API, and provides information in an appropriate format based on the user's emotion. For example, the server might generate a message saying, "The contact information for the nearest welfare center is XXX-XXXX-XXXX. If you have any concerns, you can contact them immediately," and the device will notify the message aloud.

[1437] Health Management and Emergency Response

[1438] When the user tells the device that they are "feeling unwell", the device goes into emergency call mode and sends their profile data and situation to the server. The server retrieves emergency contacts and contacts the appropriate resource (e.g., emergency services). The server executes the emergency call and sends feedback to the device saying "An ambulance has been called." The device then tells the user by voice, "An ambulance has been called. Please remain calm."

[1439] Online shopping support

[1440] When a user requests "I want to buy bread" from the device, the device converts the speech into text and sends it to the server. The server analyzes the text and searches for product lists from affiliated online shopping sites. It performs sentiment analysis on the found product list and presents it to the user in an appropriate format. If the user responds "Yes," the server carries out the final purchase procedure and sends a confirmation message to the device. The device then displays and sounds a message saying "Purchase completed."

[1441] Specific examples

[1442] For example, when a user says "Good morning" to a device in the morning, the device converts the speech into text and sends it to the server. The server analyzes the received text and generates a response based on the results of emotion analysis. This series of steps allows the user to receive a response that reflects their emotions. The specific prompt text is as follows:

[1443] When you say "Good morning," you respond with:

[1444] "Good morning. The weather is sunny today and it looks like it's going to be a pleasant day."

[1445] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1446] User Registration and Profiling

[1447] Step 1:

[1448] The user starts up the device and speaks, "I'd like to register my information." Once the voice input is complete, the device receives the voice data.

[1449] Input: User voice input

[1450] Output: Audio data

[1451] What happens: The device's microphone captures the user's voice.

[1452] Step 2:

[1453] The device uses the Google Cloud Speech-to-Text API to convert speech to text.

[1454] Input: Audio data

[1455] Output: Text data

[1456] Specific operation: The audio data is sent to the Google Cloud Speech-to-Text API and returned as text data.

[1457] Step 3:

[1458] The terminal analyzes the converted text and displays a screen for entering user information.

[1459] Input: Text data

[1460] Output: User information input screen

[1461] Specific operation: The terminal analyzes the text data and generates a graphical user interface based on its content.

[1462] Step 4:

[1463] The user enters information such as name, age, address, medical history, etc. The terminal receives this input and examines the data in real time.

[1464] Input: User input information

[1465] Output: Validated input data

[1466] Specific operation: The terminal checks the data type and format of each field in real time.

[1467] Step 5:

[1468] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[1469] Input: Validated input data

[1470] Output: JSON data

[1471] Specific operation: The device formats the data into JSON format and sends it to the server over a secure communication channel.

[1472] Step 6:

[1473] The server analyzes the received data and stores it in a MySQL database.

[1474] Input: JSON data

[1475] Output: User information stored in the database

[1476] What happens: The server deserializes the data and performs an insert operation into the database.

[1477] Step 7:

[1478] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[1479] Input: User information stored in the database

[1480] Output:Confirmation message

[1481] Specific operations: The server configures the user profile and sends a message to the terminal indicating success.

[1482] Step 8:

[1483] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[1484] Input:Confirmation message

[1485] Output: Visual and audio notifications

[1486] Specific operation: The device displays the message on the screen and converts it into audio using the Google Cloud Text-to-Speech API and plays it back.

[1487] Daily communication

[1488] Step 1:

[1489] The user speaks "Good morning" to the device in the morning, and the device recognizes the voice.

[1490] Input: User voice input

[1491] Output: Audio data

[1492] Specific operation: The device's microphone captures audio data.

[1493] Step 2:

[1494] The device uses the Google Cloud Speech-to-Text API to convert speech to text.

[1495] Input: Audio data

[1496] Output: Text data

[1497] Specific operation: The device sends voice data to the Google Cloud Speech-to-Text API and receives it as text data.

[1498] Step 3:

[1499] The terminal sends the converted text to the server.

[1500] Input: Text data

[1501] Output: Text data sent to the server

[1502] Specific operation: The terminal communicates via the Internet to send text data to the server.

[1503] Step 4:

[1504] The server receives the text data and performs sentiment analysis using IBM Watson Tone Analyzer.

[1505] Input: Text data

[1506] Output: Emotion data

[1507] Specific operation: The server sends the text data to the IBM Watson Tone Analyzer and obtains the analysis results.

[1508] Step 5:

[1509] The server generates a response text based on the sentiment analysis results.

[1510] Input: Emotion data

[1511] Output: Response text

[1512] Specific operation: The server analyzes the emotional data and generates an appropriate response as text.

[1513] Step 6:

[1514] The server generates a response text and sends it to the terminal.

[1515] Input: Response text

[1516] Output: Response text sent to the terminal

[1517] Specific operation: The server communicates via the Internet to send the response text to the terminal.

[1518] Step 7:

[1519] The device converts the received response text into speech using the Google Cloud Text-to-Speech API and conveys it to the user.

[1520] Input: Response text

[1521] Output: Voice response

[1522] What it does: Your device converts the text into speech and plays it through the speaker.

[1523] Searching for and providing information

[1524] Step 1:

[1525] The user speaks to the device, saying, "Please tell me the contact information for the nearest welfare center." The device recognizes the voice.

[1526] Input: User voice input

[1527] Output: Audio data

[1528] Specific operation: The device's microphone captures audio data.

[1529] Step 2:

[1530] The device uses the Google Cloud Speech-to-Text API to convert speech to text.

[1531] Input: Audio data

[1532] Output: Text data

[1533] Specific operation: Sends audio data to the API and retrieves text data.

[1534] Step 3:

[1535] The terminal sends the converted text to the server.

[1536] Input: Text data

[1537] Output: Text data sent to the server

[1538] Specific operation: To send text data to the server, the HTTPS protocol is used for communication.

[1539] Step 4:

[1540] The server receives the text data and searches for "welfare center contact information" using an internal database or external API.

[1541] Input: Text data

[1542] Output: Contact information

[1543] What it does: Parses text data and queries databases and APIs to retrieve contact information.

[1544] Step 5:

[1545] The server performs sentiment analysis on the contact information and generates it in a format that corresponds to the user's sentiment.

[1546] Input: Contact Information

[1547] Output: Formatted response text

[1548] Specific operation: Based on the results of sentiment analysis, the contact information is formatted into appropriate text.

[1549] Step 6:

[1550] The server sends the information to the terminal.

[1551] Input: Formatted response text

[1552] Output: Response text sent to the terminal

[1553] What it does: Sends text to the device using the HTTPS protocol.

[1554] Step 7:

[1555] The information received by the device is converted into audio using the Google Cloud Text-to-Speech API, and is displayed and audio notified.

[1556] Input: Formatted response text

[1557] Output: Visual and audio notifications

[1558] Specific behavior: Converts text to speech, displays it on the display, and plays it through the speaker.

[1559] Health Management and Emergency Response

[1560] Step 1:

[1561] The user says to the device, "I feel sick." The device recognizes the voice.

[1562] Input: User voice input

[1563] Output: Audio data

[1564] Specific operation: The device's microphone captures audio data.

[1565] Step 2:

[1566] The device enters emergency call mode and transmits profile data and status to the server.

[1567] Input: Audio data

[1568] Output: Profile and status data sent to the server

[1569] Specific operation: Activate emergency mode and send necessary data to the server.

[1570] Step 3:

[1571] The server retrieves the emergency contact information from the database and contacts the appropriate resource.

[1572] Input: Profile data and situational data

[1573] Output: Emergency contact

[1574] Specific behavior: Retrieves emergency contact information from a database and contacts emergency response services.

[1575] Step 4:

[1576] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called."

[1577] Input: Emergency Contact Information

[1578] Output: Feedback message

[1579] Specific operation: Make an emergency call and send the results to the device.

[1580] Step 5:

[1581] The device will notify the user by voice, "An ambulance has been called. Please remain calm."

[1582] Input: Feedback message

[1583] Output: Audio notification

[1584] Specific behavior: Convert the message into audio and notify the user.

[1585] Online shopping support

[1586] Step 1:

[1587] The user requests the device to "buy bread." The device recognizes the voice.

[1588] Input: User voice input

[1589] Output: Audio data

[1590] Specific operation: The device's microphone captures audio data.

[1591] Step 2:

[1592] The device uses the Google Cloud Speech-to-Text API to convert speech to text.

[1593] Input: Audio data

[1594] Output: Text data

[1595] Specific operation: Sends audio data to the API and retrieves text data.

[1596] Step 3:

[1597] The terminal sends the converted text to the server.

[1598] Input: Text data

[1599] Output: Text data sent to the server

[1600] Specific operation: To send text data to the server, the HTTPS protocol is used for communication.

[1601] Step 4:

[1602] The server analyzes the text data and searches for product lists from affiliated online shopping sites.

[1603] Input: Text data

[1604] Output: Product list

[1605] Specific operation: The server parses the text data, creates a query, and sends it to the shopping site's API.

[1606] Step 5:

[1607] The server performs sentiment analysis on the product list and generates it in the appropriate format.

[1608] Input: Product list

[1609] Output: A formatted product list

[1610] Specific operation: Format product list based on sentiment analysis results.

[1611] Step 6:

[1612] The server sends the product list to the terminal.

[1613] Input: Formatted product list

[1614] Output: Product list sent to the terminal

[1615] Specific operation: The product list is sent to the terminal using the HTTPS protocol.

[1616] Step 7:

[1617] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[1618] Input: Formatted product list

[1619] Output: Display and audio queries

[1620] Specific operation: Display a product list on the screen and ask questions by voice.

[1621] Step 8:

[1622] If the user answers "yes," the terminal sends the information to the server.

[1623] Input: Voice response

[1624] Output: Response sent to the server

[1625] Specific operation: The device converts the voice response into text and sends it to the server.

[1626] Step 9:

[1627] The server completes the final purchase and sends a confirmation message to the terminal.

[1628] Input: Response data

[1629] Output:Confirmation message

[1630] Specific operation: The server executes the purchase procedure and sends the results to the terminal.

[1631] Step 10:

[1632] The terminal will notify the user by displaying and audibly saying "Purchase completed."

[1633] Input:Confirmation message

[1634] Output: Visual and audio notifications

[1635] Specific operation: A confirmation message is displayed on the screen and played back as audio.

[1636] (Application example 2)

[1637] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1638] When elderly people shop alone in physical stores, they face the challenge of finding it difficult to obtain information and advice about products, making it difficult for them to enjoy shopping with peace of mind. Furthermore, due to a lack of individualized support tailored to the elderly's emotions, they often feel isolated and anxious. Furthermore, there is a need for a system that can respond quickly in emergencies. It is necessary to solve these problems and provide an environment where elderly people can enjoy shopping with peace of mind.

[1639] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1640] In this invention, the server includes: means for transmitting information input by a user to the server through a terminal and having the server generate a user profile based on the information; means for recognizing a voice input by the user and converting the voice to text; means for transmitting the text to the server and generating a response based on the text received by the server; means for the terminal to convert the response to voice and provide it to the user again; means for the server to search the Internet or a database for information corresponding to the user's request and provide it to the terminal; means for inputting information about the user's health condition and transmitting it to the server through the terminal, which then automatically contacts an emergency contact; means for transmitting information about a product desired by the user to the server through the terminal, which then searches for the product on affiliated e-commerce sites and completes the purchase process; and a smart brick-and-mortar store support application that has an engine for analyzing user emotions and provides responses and support based on the user's emotions. This allows elderly people to enjoy shopping with peace of mind and receive the information and support they need appropriately.

[1641] A "user profile" is an information structure that includes individual user characteristics and background information, and is generated based on information entered by the user.

[1642] "Voice input" is the process by which a device receives and analyzes what a user says as sound.

[1643] "Text conversion" is the process of analyzing information input via voice and converting it into text.

[1644] A "server" is a central processing unit that stores, processes, sends and receives data over a network.

[1645] "Response generation" is the process of creating information to respond to the user based on the received text.

[1646] An "emotion analysis engine" is software or an algorithm that determines emotions from input information and provides appropriate responses or support based on the results.

[1647] The "Smart Physical Store Support Application" is an application designed to support users' shopping experiences in physical stores, and is a system that handles user input, analyzes emotions, and provides necessary information.

[1648] An "Internet search" is the process of utilizing a wide range of networks to locate and provide information requested by a user.

[1649] "Emergency contact method" is a process in which emergency information such as the user's health condition is sent to a server, and the server automatically contacts the appropriate emergency contact.

[1650] "Product information search" is the process of finding and providing information about the product desired by the user from e-commerce sites and other sources.

[1651] This invention is a support system that allows elderly people to enjoy shopping in brick-and-mortar stores with peace of mind. In particular, it has the feature of recognizing the user's emotions and providing appropriate responses and support based on those emotions.

[1652] System Configuration

[1653] The system is composed of a terminal used by the user (smart glasses, smartphone, etc.), a server that performs central processing functions, sensors for acquiring data on the user's behavior and health status, etc. Specific embodiments of the present invention will be described below in order.

[1654] User Registration and Profiling

[1655] 1. User Registration:

[1656] The user starts up the device and registers basic information (such as name, age, address, medical history, etc.) by voice or manual input. This information is sent from the device to the server, which then creates a user profile and stores it in a database.

[1657] After the profile is generated, a confirmation message is sent back to the terminal and the user is notified audibly and visually.

[1658] Daily shopping assistance

[1659] 2. Voice to text conversion:

[1660] While shopping, users input voice commands into the device, such as "Tell me about this product," which are converted into text via speech recognition software such as the Google Speech-to-Text API.

[1661] The converted text is sent to the server using the HTTPS protocol.

[1662] 3. Sentiment analysis and response generation:

[1663] The server uses a sentiment analysis engine such as Amazon Comprehend to analyze sentiment from the user's text input.

[1664] Based on the results of emotion analysis, appropriate responses and support information are generated for the user and sent from the server to the terminal.

[1665] 4. Response provision:

[1666] The response text is converted into speech at the terminal and provided to the user in speech and text.

[1667] Emergency response and health management

[1668] 5. Emergency Response:

[1669] When a user tells the terminal that they are "feeling unwell," the terminal converts this speech into text and sends it to the server.

[1670] The server consults the user profile and emergency contact database and contacts the appropriate resources (such as emergency services).

[1671] Emergency response information is communicated to the user via the terminal by voice.

[1672] Searching and providing product information

[1673] 6. Product Search:

[1674] When a user types "Tell me about this product," the server searches for detailed information about the product from affiliated e-commerce sites, performs sentiment analysis on the results, and then provides them to the user.

[1675] Examples of concrete examples and prompts

[1676] For example, when a user speaks to a terminal in a store and says, "Tell me about this product," the terminal converts the speech into text and sends it to the server. The server analyzes the text and the user's emotions and generates appropriate product information and recommendations. This process allows the user to receive a response that reflects their emotions, allowing them to enjoy shopping with peace of mind.

[1677] Prompt Sentence Examples

[1678] “When a user says, ‘Tell me about this product,’ provide a product description and offer recommendations based on the user’s emotions. For example, if the user seems anxious, highlight how the product will help them.”

[1679] By utilizing generative AI models and prompts, this system provides an environment where seniors can enjoy shopping in physical stores with peace of mind.

[1680] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1681] Step 1:

[1682] User Registration:

[1683] The user starts up the device and registers basic information (name, age, address, medical history, etc.) by voice input or manual input.

[1684] The device converts this input information into text, formats it as JSON, and sends it to the server using the HTTPS protocol.

[1685] The server parses the received JSON data, generates a user profile, and saves it in the database.

[1686] Based on the generated profile, the server creates a confirmation message and sends it to the terminal.

[1687] The device will notify the user of a confirmation message by voice and text.

[1688] Step 2:

[1689] Speech to text:

[1690] While shopping, the user issues a voice command such as "Tell me about this product."

[1691] The device converts voice input into text using the Google Speech-to-Text API.

[1692] This text data is sent to the server using the HTTPS protocol.

[1693] Step 3:

[1694] Sentiment analysis and response generation:

[1695] The server analyzes the user's emotions from the received text data using an emotion analysis engine such as Amazon Comprehend.

[1696] Based on the sentiment analysis results and the text, the server generates a prompt sentence, which is then used to generate an appropriate response text.

[1697] The response text might be generated in the form of, for example, "This product contains healthy ingredients and is ideal for the elderly."

[1698] Step 4:

[1699] Response provided:

[1700] The server generates a response text and sends it to the terminal.

[1701] The terminal converts the received response text into speech and provides it to the user in speech and text.

[1702] Step 5:

[1703] Product Search:

[1704] The user again instructs the terminal to "tell me about this product."

[1705] The device converts the speech into text and sends the text to the server.

[1706] The server uses the text to search for product information using the API of the e-commerce site.

[1707] The server retrieves the search results, simultaneously performs sentiment analysis, and provides information in a format that corresponds to the user's emotions.

[1708] The server sends this product information to the terminal, and the terminal informs the user by voice and text, "The price of this product is XXX yen and it is very popular."

[1709] Step 6:

[1710] Emergency Response:

[1711] The user tells the terminal that he is "feeling unwell."

[1712] The device converts the voice into text and sends it to the server as emergency information.

[1713] The server consults the user profile and emergency contact database and contacts the appropriate emergency contact (eg, emergency services).

[1714] The server generates progress information on the emergency response and transmits it to the terminal.

[1715] The terminal will tell the user by voice, "An ambulance has been called. Please wait without worry."

[1716] Through these processing steps, the system can quickly and accurately provide the information and support necessary for seniors to enjoy shopping safely in physical stores.

[1717] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1718] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1719] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1720] [Third embodiment]

[1721] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1722] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1723] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1724] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1725] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1726] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1727] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1728] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1729] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1730] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1731] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1732] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1733] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone. This system functions by interoperating with three entities: a server, a terminal, and a user.

[1734] Program processing details

[1735] 1. User Registration and Profiling

[1736] The user turns on the device (e.g., a smartphone or tablet) and speaks, "I would like to register my information."

[1737] The device converts the voice to text and displays a screen for entering information.

[1738] The user enters basic information such as name, age, address, and medical history.

[1739] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[1740] The server parses the received JSON data and stores the information in the database.

[1741] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[1742] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[1743] 2. Daily communication

[1744] In the morning, the user says "Good morning" to the terminal.

[1745] The device converts the speech into text and sends the text to the server.

[1746] The server receives the text and generates a response text saying "Good morning. The weather is sunny today."

[1747] The server generates a response and sends it to the terminal.

[1748] The terminal converts the received response text into speech and tells the user, "Good morning. The weather is sunny today."

[1749] 3. Searching for and providing information

[1750] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[1751] The device converts the speech into text and sends the text to the server.

[1752] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[1753] The server sends the contact information it finds to the device.

[1754] The information received by the terminal is displayed and output to the user as audio, such as "The contact information for the nearest welfare center is XXX-XXXX-XXXX."

[1755] 4. Health Management and Emergency Response

[1756] The user tells the terminal that he is "feeling unwell."

[1757] The device detects this sound and goes into emergency call mode.

[1758] The device sends profile data (such as medical history) and status to the server.

[1759] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[1760] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called."

[1761] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[1762] 5. Online shopping support

[1763] The user makes a request to the terminal saying, "I want to buy bread."

[1764] The device converts the speech into text and sends it to the server.

[1765] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[1766] The server sends a list of the found products to the terminal.

[1767] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[1768] If the user answers "yes," the terminal sends the information to the server.

[1769] The server completes the final purchase and sends a confirmation message to the terminal.

[1770] The terminal will display and audibly inform the user that "Purchase completed."

[1771] Specific examples

[1772] For example, when a user wakes up in the morning and says "Good morning" to the device, the device converts the speech into text and sends it to the server. The server generates a response, "Good morning. The weather is sunny today," and sends it back to the device. The device then converts this response into speech and conveys it to the user. This series of steps allows users to engage in everyday communication without feeling isolated.

[1773] This system is divided into detailed processing steps, and by performing specific actions at each step, it provides multifaceted support for elderly people living alone.

[1774] The processing flow will be explained below.

[1775] User Registration and Profiling

[1776] Step 1:

[1777] The user starts up the device and says, "I'd like to register my information."

[1778] Your device uses speech recognition to convert speech into text.

[1779] Step 2:

[1780] The terminal analyzes the converted text and displays a screen for entering user information.

[1781] Step 3:

[1782] The user enters basic information such as name, age, address, and medical history.

[1783] Step 4:

[1784] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[1785] Step 5:

[1786] The server parses the received JSON data and stores the information in the database.

[1787] Step 6:

[1788] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[1789] Step 7:

[1790] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[1791] Daily communication

[1792] Step 1:

[1793] In the morning, the user says "Good morning" to the terminal.

[1794] Step 2:

[1795] The device converts the speech into text and sends the text to the server.

[1796] Step 3:

[1797] The server receives the text and generates a response text saying "Good morning. The weather is sunny today."

[1798] Step 4:

[1799] The server generates a response and sends it to the terminal.

[1800] Step 5:

[1801] The terminal converts the received response text into speech and tells the user, "Good morning. The weather is sunny today."

[1802] Searching for and providing information

[1803] Step 1:

[1804] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[1805] Step 2:

[1806] The device converts the speech into text and sends the text to the server.

[1807] Step 3:

[1808] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[1809] Step 4:

[1810] The server sends the contact information it finds to the device.

[1811] Step 5:

[1812] The information received by the terminal is displayed and output to the user as audio, such as "The contact information for the nearest welfare center is XXX-XXXX-XXXX."

[1813] Health Management and Emergency Response

[1814] Step 1:

[1815] The user tells the terminal that he is "feeling unwell."

[1816] Step 2:

[1817] The device detects this sound and goes into emergency call mode.

[1818] Step 3:

[1819] The device sends profile data (such as medical history) and status to the server.

[1820] Step 4:

[1821] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[1822] Step 5:

[1823] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called."

[1824] Step 6:

[1825] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[1826] Online shopping support

[1827] Step 1:

[1828] The user makes a request to the terminal saying, "I want to buy bread."

[1829] Step 2:

[1830] The device converts the speech into text and sends it to the server.

[1831] Step 3:

[1832] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[1833] Step 4:

[1834] The server sends a list of the found products to the terminal.

[1835] Step 5:

[1836] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[1837] Step 6:

[1838] If the user answers "yes," the terminal sends the information to the server.

[1839] Step 7:

[1840] The server completes the final purchase and sends a confirmation message to the terminal.

[1841] Step 8:

[1842] The terminal will display and audibly inform the user that "Purchase completed."

[1843] By going through the above steps, this system can provide multifaceted support to elderly people living alone.

[1844] Example 1

[1845] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1846] Elderly people living alone face a wide range of challenges, including daily communication, health management, and emergency response. To efficiently and comprehensively address these challenges, an easy-to-use system is required. However, current technology provides solutions for individual problems in a fragmented manner, preventing a consistent user experience.

[1847] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1848] In this invention, the server includes means for transmitting information input by a user to the server via a terminal and for the server to generate a user profile based on the information, means for recognizing a voice input by the user and converting the voice into text, and means for transmitting the text to the server and generating a response based on the text received by the server, thereby enabling multifaceted support for elderly people in their daily activities such as communication, emergency response, health management, and online shopping.

[1849] "Users" refers to the elderly people who use the system and the people who support them.

[1850] A "terminal" is a device operated by a user, and refers to information and communication devices such as smartphones and tablets.

[1851] "Server" refers to a central computing device that processes information sent from terminals and manages it in a database.

[1852] A "user profile" refers to a data set that includes personal information such as a user's basic information and medical history.

[1853] "Voice input" refers to voice commands given by a user through a microphone.

[1854] "Speech recognition" refers to the technology that converts voice input into text form.

[1855] "Text conversion" refers to the process of processing text obtained by speech recognition as data.

[1856] "Secure Protocol" refers to a secure communications protocol such as HTTPS.

[1857] "Generate" refers to the process by which the server generates text or audio information.

[1858] "Search" refers to the operation by which a server retrieves information using the Internet or an internal database.

[1859] "Emergency contact" refers to the procedure by which the server automatically notifies pre-defined contacts in the event of an emergency.

[1860] "Shopping Site" refers to the affiliated e-commerce platform.

[1861] "Voice notification" refers to a function in which a terminal uses a voice conversion function to convey information to a user.

[1862] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone. This system functions by interoperating with three entities: a server, a terminal, and a user.

[1863] First, the user uses a smartphone or tablet as a device. These devices have speech recognition capabilities and convert the user's voice input into text using the Google Speech-to-Text API. The device also converts text into speech using the Google Text-to-Speech API.

[1864] Next, the information entered by the user is sent to the server via the terminal. For example, when a user enters basic information such as name, age, address, and medical history, this information is converted into JSON format and sent to the server using the HTTPS protocol. The server analyzes the received information, stores it in a MySQL database, and generates a user profile.

[1865] In everyday communication, when a user speaks to a device, for example, "Good morning," the device converts the speech into text and sends it to the server. The server generates a response text and sends it to the device. The device then converts the received response text into speech and conveys it to the user. For example, "Good morning. The weather is sunny today."

[1866] Also, when a user says to the device, "Please tell me the contact information for the nearest welfare center," the device converts the voice into text and sends that text to the server. The server uses an internal database or an external API (such as Google Maps API) to search for the welfare center's contact information and sends that information to the device. The device then converts the received information into voice and relays it to the user.

[1867] Furthermore, if the user tells the device that they are "feeling unwell," the device will enter emergency call mode and send their profile data and status to the server. The server will retrieve emergency contact information from a database and contact the appropriate resource (e.g., emergency services). The server will then send a confirmation message to the device saying "an ambulance has been called," which the device will then verbally convey to the user.

[1868] For online shopping support, when a user requests the device to "buy bread," the device converts the voice to text and sends it to the server. The server then searches for a product list from affiliated shopping sites and sends it to the device. The device then displays the product list to the user and asks aloud, "Is this OK?" If the user answers "yes," the device completes the final purchase procedure and a confirmation message is spoken by the device.

[1869] Specific examples

[1870] For example, if a user says "Good morning" to a device in the morning, the device converts the speech into text and sends it to the server. The server generates a response such as "Good morning. The weather is sunny today" and sends it back to the device. The device then converts this response into speech and conveys it to the user. This series of steps allows users to engage in everyday communication without feeling isolated.

[1871] Examples of prompts include "Good morning," "Please tell me the contact information of the nearest welfare center," "I'm not feeling well," "I want to buy some bread," etc. Through such prompts, it is possible to provide the multifaceted support that the elderly require in their daily lives.

[1872] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1873] Program processing flow

[1874] User Registration and Profiling

[1875] Step 1:

[1876] The user starts up the device and says, "I'd like to register my information."

[1877] Input: Audio

[1878] Output: Audio data

[1879] Specific operation: The user uses a smartphone or tablet to input voice information.

[1880] Step 2:

[1881] The device converts the speech to text.

[1882] Input: Audio data

[1883] Output: Text data

[1884] What happens: The device converts speech to text using the Google Speech-to-Text API.

[1885] Step 3:

[1886] The user enters basic information such as name, age, address, and medical history.

[1887] Input: User basic information

[1888] Output: User basic information data

[1889] Specific operation: The user enters information into an input form displayed on the device screen.

[1890] Step 4:

[1891] The device converts the information into JSON format.

[1892] Input: User basic information data

[1893] Output: JSON data

[1894] Specific operation: The terminal converts the input data into JSON format.

[1895] Step 5:

[1896] The terminal sends data to the server.

[1897] Input: JSON data

[1898] Output: Send data to the server

[1899] Specific operation: The device sends data to the server using the HTTPS protocol.

[1900] Step 6:

[1901] The server analyzes the data and stores it in a database.

[1902] Input: JSON data

[1903] Output: Store in database

[1904] Specific operation: The server analyzes the received data and stores it in a MySQL database.

[1905] Step 7:

[1906] The server generates a confirmation message and sends it to the terminal.

[1907] Input: Database update information

[1908] Output:Confirmation message

[1909] Specific operation: The server creates a message saying "Profile has been generated" and sends it to the terminal.

[1910] Step 8:

[1911] The device will notify the user by voice.

[1912] Input:Confirmation message

[1913] Output: Audio notification

[1914] Specific behavior: The device uses the Google Text-to-Speech API to convert the text to speech and notifies the user that "Registration is complete."

[1915] Daily communication

[1916] Step 1:

[1917] The user says "Good morning" to the terminal.

[1918] Input: Audio

[1919] Output: Audio data

[1920] Specific operation: The user greets the device by voice.

[1921] Step 2:

[1922] The device converts the speech to text.

[1923] Input: Audio data

[1924] Output: Text data

[1925] What happens: The device converts speech to text using the Google Speech-to-Text API.

[1926] Step 3:

[1927] The device sends the text to the server.

[1928] Input: Text data

[1929] Output: Send text data to the server

[1930] Specific operation: The device sends text data to the server via HTTP.

[1931] Step 4:

[1932] The server parses the text and generates a response.

[1933] Input: Text data

[1934] Output: Response text

[1935] Specific operation: The server analyzes the received text data and generates a response saying, "Good morning. The weather is sunny today."

[1936] Step 5:

[1937] The server sends the response text to the terminal.

[1938] Input: Response text

[1939] Output: Response sent from the server

[1940] Specific operation: The response text generated by the server is sent to the terminal.

[1941] Step 6:

[1942] The terminal converts the response into voice and conveys it to the user.

[1943] Input: Response text

[1944] Output: Audio notification

[1945] What happens: The device uses the Google Text-to-Speech API to convert text to speech and tells the user, "Good morning. The weather is sunny today."

[1946] Searching for and providing information

[1947] Step 1:

[1948] The user speaks to the device, saying, "Please tell me the contact information for the nearest welfare center."

[1949] Input: Audio

[1950] Output: Audio data

[1951] Specific behavior: The user makes a request by voice.

[1952] Step 2:

[1953] The device converts the speech to text.

[1954] Input: Audio data

[1955] Output: Text data

[1956] What happens: The device converts speech to text using the Google Speech-to-Text API.

[1957] Step 3:

[1958] The device sends the text to the server.

[1959] Input: Text data

[1960] Output: Send data to the server

[1961] Specific operation: The device sends text data to the server via HTTP.

[1962] Step 4:

[1963] The server retrieves the information.

[1964] Input: Text data

[1965] Output: Search results

[1966] Specific operation: The server searches for "welfare center contact information" using its internal database and Google Maps API.

[1967] Step 5:

[1968] The server sends the found information to the device.

[1969] Input: Search results

[1970] Output: Information sent from the server

[1971] Specific operation: The server sends the search results to the terminal.

[1972] Step 6:

[1973] The device converts the information into voice and conveys it to the user.

[1974] Input: Search results

[1975] Output: Audio notification

[1976] What it does: The device uses the Google Text-to-Speech API to convert the text into speech and tells the user, "The contact information for the nearest welfare center is XXX-XXXX-XXXX."

[1977] Health Management and Emergency Response

[1978] Step 1:

[1979] The user tells the terminal that he is "feeling unwell."

[1980] Input: Audio

[1981] Output: Audio data

[1982] Specific operation: The user vocally complains about feeling unwell.

[1983] Step 2:

[1984] The device converts the speech to text.

[1985] Input: Audio data

[1986] Output: Text data

[1987] What happens: The device converts speech to text using the Google Speech-to-Text API.

[1988] Step 3:

[1989] The device enters emergency call mode.

[1990] Input: Text data

[1991] Output: Emergency call mode activated

[1992] Specific operation: The device receives the text data and activates emergency call mode.

[1993] Step 4:

[1994] The device sends profile data and status to the server.

[1995] Input: Profile data, text data

[1996] Output: Send data to the server

[1997] Specific operation: The device sends profile data and status to the server.

[1998] Step 5:

[1999] The server retrieves the emergency contact information.

[2000] Input: Profile data

[2001] Output: Emergency contact information

[2002] Specific operation: The server retrieves emergency contact information from the database.

[2003] Step 6:

[2004] The server contacts emergency services.

[2005] Input: Emergency Contact Information

[2006] Output: Emergency contact execution

[2007] Specific Actions: The server contacts emergency services.

[2008] Step 7:

[2009] The server sends a confirmation message to the terminal.

[2010] Input: Contact status

[2011] Output:Confirmation message

[2012] Specific operation: The server sends a confirmation message to the terminal saying "An ambulance has been called."

[2013] Step 8:

[2014] The device will notify the user by voice.

[2015] Input:Confirmation message

[2016] Output: Audio notification

[2017] What happens: The device uses the Google Text-to-Speech API to convert the text into speech and notifies the user, "An ambulance has been called. Please remain calm."

[2018] Online shopping support

[2019] Step 1:

[2020] The user makes a request to the terminal saying, "I want to buy bread."

[2021] Input: Audio

[2022] Output: Audio data

[2023] Specific operation: The user expresses their purchase intention by voice.

[2024] Step 2:

[2025] The device converts the speech to text.

[2026] Input: Audio data

[2027] Output: Text data

[2028] What happens: The device converts speech to text using the Google Speech-to-Text API.

[2029] Step 3:

[2030] The device sends the text to the server.

[2031] Input: Text data

[2032] Output: Send data to the server

[2033] Specific operation: The device sends text data to the server via HTTP.

[2034] Step 4:

[2035] The server parses the text and searches for products.

[2036] Input: Text data

[2037] Output: Product list

[2038] Specific operation: The server searches for a list of bread products using the API of a shopping site it is affiliated with.

[2039] Step 5:

[2040] The server sends the product list to the terminal.

[2041] Input: Product list

[2042] Output: List sent from server

[2043] Specific operation: The server sends the list of found products to the terminal.

[2044] Step 6:

[2045] The device displays a list of products and asks aloud, "Is this okay?"

[2046] Input: Product list

[2047] Output: Display, voice prompts

[2048] Specific operation: The device displays a list of products to the user and asks aloud, "Is this okay?"

[2049] Step 7:

[2050] The user responds "Yes."

[2051] Input: Audio

[2052] Output: Audio data

[2053] What happens: The user confirms the purchase and responds verbally.

[2054] Step 8:

[2055] The terminal sends this response to the server.

[2056] Input: Audio data

[2057] Output: Text data

[2058] Specific operation: The device converts the voice data into text and sends it to the server.

[2059] Step 9:

[2060] The server completes the final purchase and sends a confirmation message to the terminal.

[2061] Input: Text data

[2062] Output:Confirmation message

[2063] Specific operation: The server performs the final purchase procedure, generates a message saying "Purchase completed" and sends it to the terminal.

[2064] Step 10:

[2065] The device will notify you by voice that the purchase is complete.

[2066] Input:Confirmation message

[2067] Output: Audio notification

[2068] What happens: The device uses the Google Text-to-Speech API to convert the text to speech and tells the user, "Your purchase is complete."

[2069] (Application example 1)

[2070] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2071] The lack of comprehensive support systems for the various problems that elderly people face in their daily lives is a challenge. In particular, systems that can quickly respond when elderly people order meals or in emergencies are needed, but the current means to achieve this are insufficient. Furthermore, interactive interfaces are also needed to reduce the burden that elderly people feel when performing these tasks alone.

[2072] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2073] In this invention, the server includes: means for transmitting information input by a user to the server through a terminal and for the server to generate a user profile based on the information; means for recognizing a voice input by the user and converting the voice to text; means for transmitting the text to the server and generating a response based on the text received by the server; means for the terminal to convert the response to voice and provide it to the user again; means for the server to search the Internet or a database for information corresponding to the user's request and provide it to the terminal; means for transmitting an order for a meal desired by the user to the server through the terminal, searching for product lists from affiliated food delivery services with the server, and completing the purchase procedure; means for the server to check the delivery status and obtain information and send it to the terminal when the user says "Tell me the delivery status" to the terminal; and means for the terminal to enter emergency response mode when the user says to the terminal that they are feeling unwell, notifying emergency contacts via the server, and arranging appropriate responses. This enables elderly people to easily meet various needs in their daily lives and respond to emergencies.

[2074] 1. A "terminal" is an electronic device that allows users to input information or give voice instructions. Examples include smartphones, tablets, and robots.

[2075] 2. A "server" is a computer system that receives data sent from a terminal via the Internet or a network, performs the necessary processing, and then sends a response back to the terminal.

[2076] 3. A "user profile" is a set of detailed data such as a user's personal information, past behavioral history, preferences, and allergy information.

[2077] 4. "Voice input" means the user speaks through a microphone, which is converted into text by the system.

[2078] 5. "Text conversion" is the process of analyzing voice input and converting it into text information.

[2079] 6. "Response" means a reply message generated by the server based on received text and profile information.

[2080] 7. "Voice output" is the process of converting the generated response text from text information back into voice and delivering it to the user.

[2081] 8. "Internet" means a worldwide collection of computer networks that enable the transmission and reception of information.

[2082] 9. A "database" is a collection of structured data that organizes large amounts of information and allows for quick search and retrieval.

[2083] 10. "Food delivery service" means a service that allows a user to order a meal they want and delivers the meal to a location specified by the user.

[2084] 11. "Emergency Response Mode" is a function that allows the system to automatically take special action and notify emergency contacts when a user reports a health problem.

[2085] 12. "Purchase procedure" refers to a series of processes for purchasing products from affiliated shopping sites based on information about the products desired by the user.

[2086] 13. "Delivery status" is information that indicates the current location of the ordered meal or product and how long it will take to arrive at the user's location.

[2087] 14. "Application" means a specific example of applying this invention to a specific field or use.

[2088] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone, and it functions by interoperating with three main entities: a terminal, a server, and a user. The system has a wide range of functions, including user profile generation, voice input recognition, text conversion, response generation, information search, food ordering, delivery status tracking, and emergency response.

[2089] 1. User profiling

[2090] Users input information through devices such as smartphones and tablets. The devices convert the input speech into text and send it to a server in JSON format using the HTTPS protocol. The server analyzes the data and creates a user profile including name, age, address, medical history, dietary preferences, and allergy information, and stores it in a database.

[2091] 2. Voice Input and Response Generation

[2092] When a user says "I'd like to order lunch" to the device, the device converts the speech into text and sends it to the server. The server analyzes the received text and generates a list of recommended menu items based on past order history and profile information. This menu list is sent back to the device, which then asks aloud, "Is this OK?" This allows the user to complete their meal order using only the voice interface.

[2093] 3. Track your delivery status

[2094] When the user speaks to the device saying, "Tell me the delivery status," the device converts the speech into text and sends it to the server. The server obtains the delivery status from the partner food delivery service and sends the real-time status to the device. The device then notifies the user by notifying them, for example, "The delivery person will arrive in 5 minutes."

[2095] 4. Emergency Response

[2096] When a user notifies the device that they are "feeling unwell," the device enters emergency response mode and sends an emergency notification along with their profile data to the server. The server notifies emergency contacts and arranges for the necessary response. The device then notifies the user by voice, saying, "An ambulance has been called. Please remain calm."

[2097] Hardware and software used

[2098] Hardware: smartphones, tablets, robots

[2099] software:

[2100] Natural Language Processing: Google Cloud Speech-to-Text

[2101] Data transmission / reception: HTTPS protocol

[2102] Database: MySQL

[2103] Server: AWS EC2

[2104] Speech synthesis: Google Cloud Text-to-Speech

[2105] Specific examples

[2106] Consider a scenario where an elderly person wants to order lunch. When the user says "I'd like to order lunch," the following dialogue takes place:

[2107] User: "I'd like to order lunch."

[2108] Device: "The recommended menu items registered are curry rice, sandwiches, and udon. Which would you like?"

[2109] User: "Curry rice"

[2110] Terminal: "I ordered curry rice."

[2111] Example prompt sentence:

[2112] "A user is trying to order lunch using a food delivery assistance assistant for seniors. Generate a series of dialogues with corresponding menu choices."

[2113] This will enable elderly people to easily meet various needs in their daily lives and respond to emergencies.

[2114] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2115] Step 1:

[2116] The user turns on a device such as a smartphone or tablet and says, "I would like to register as a user." The device converts the voice to text and displays an information entry screen. The user enters basic information such as name, address, dietary preferences, and allergy information, and the device converts this information into JSON format. This data is sent to the server using the HTTPS protocol.

[2117] Input: User voice input "I would like to register as a user", basic information

[2118] Output: JSON formatted data sent to the server

[2119] Step 2:

[2120] The server analyzes the received JSON data and generates a user profile including the user's name, age, address, medical history, dietary preferences, and allergy information. The generated user profile is stored in a database. The server confirms that the profile has been successfully generated and notifies the device that "The profile has been generated."

[2121] Input: User information in JSON format

[2122] Output: User profile stored in database, notification to terminal

[2123] Step 3:

[2124] The user speaks to the device, saying, "I'd like to order lunch." The device converts the speech into text and sends it to the server. The server analyzes the text and generates a list of recommended menu items based on past order history and user profile information. This menu list is then sent to the device.

[2125] Input: User speaks "I want to order lunch"

[2126] Output: Menu list sent to the terminal

[2127] Step 4:

[2128] The terminal displays a menu list and asks the user aloud, "Is this OK?" If the user answers "Yes," the terminal sends the information to the server, which then processes the final order. The server then sends a confirmation message to the terminal, informing the user that the order has been completed.

[2129] Input: User voice input "Yes"

[2130] Output: Order information sent to the server, confirmation message sent to the terminal

[2131] Step 5:

[2132] The user speaks to the device, saying, "Tell me the delivery status." The device converts the speech into text and sends it to the server. The server obtains the delivery status from the partner food delivery service and sends the real-time information to the device. The device then notifies the user, "The delivery person will arrive in 5 minutes."

[2133] Input: User's voice input "Tell me the delivery status"

[2134] Output: Delivery status information sent to the terminal

[2135] Step 6:

[2136] The user says "I'm feeling unwell" to the device. The device converts the speech to text and enters emergency response mode. It sends an emergency notification along with the profile data to the server. The server notifies emergency contacts and arranges for the necessary response. The device then tells the user by voice, "An ambulance has been called. Please remain calm."

[2137] Input: User's voice input "I feel sick"

[2138] Output: Notification to emergency contacts, feedback to device

[2139] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2140] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone, and is equipped with a function that recognizes the user's emotions and provides appropriate responses and support based on those emotions. This system functions in cooperation with three entities: a server, a terminal, and a user, and combines an emotion engine to improve the user experience.

[2141] Program processing details

[2142] User Registration and Profiling

[2143] The user starts up the device and says, "I'd like to register my information."

[2144] Your device uses speech recognition to convert speech into text.

[2145] The terminal analyzes the converted text and displays a screen for entering user information.

[2146] The user enters basic information such as name, age, address, and medical history.

[2147] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[2148] The server parses the received JSON data and stores the information in the database.

[2149] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[2150] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[2151] Daily communication

[2152] In the morning, the user says "Good morning" to the terminal.

[2153] The device converts the speech into text and sends the text to the server.

[2154] The server receives the text and analyzes the user's emotions using an emotion engine.

[2155] Based on the analysis results, the server generates a response text saying, "Good morning. Today's weather is sunny and it looks like it's going to be a pleasant day."

[2156] The server generates a response and sends it to the terminal.

[2157] The terminal converts the received response text into speech and tells the user, "Good morning. The weather is sunny today, and it looks like it's going to be a pleasant day."

[2158] Searching for and providing information

[2159] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[2160] The device converts the speech into text and sends the text to the server.

[2161] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[2162] The server analyzes the contact information it finds using an emotion engine and presents it in a format that corresponds to the user's emotions.

[2163] The server sends the information to the terminal.

[2164] The information received by the device is displayed and output to the user as audio, saying, "The contact number for the nearest welfare center is XXX-XXXX-XXXX. If you have any concerns, you can contact them immediately."

[2165] Health Management and Emergency Response

[2166] The user tells the terminal that he is "feeling unwell."

[2167] The device detects this sound and goes into emergency call mode.

[2168] The device sends profile data (such as medical history) and status to the server.

[2169] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[2170] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called. Please wait a moment."

[2171] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[2172] Online shopping support

[2173] The user makes a request to the terminal saying, "I want to buy bread."

[2174] The device converts the speech into text and sends it to the server.

[2175] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[2176] The server analyzes the found product list using an emotion engine and presents it in an appropriate format.

[2177] The server sends the product list to the terminal.

[2178] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[2179] If the user answers "yes," the terminal sends the information to the server.

[2180] The server completes the final purchase and sends a confirmation message to the terminal.

[2181] The terminal will display and audibly inform the user that "Purchase completed."

[2182] Specific examples

[2183] For example, if a user says "Good morning" to a device in the morning, the device converts the speech into text and sends it to the server. The server analyzes the text and the user's emotions, and generates a response that best suits the user's state of mind. This process allows the user to receive a response that reflects their emotions, enabling everyday communication while reducing feelings of isolation.

[2184] By combining this system with an emotion engine, it provides multifaceted support for elderly people living alone and offers more personalized services.

[2185] The processing flow will be explained below.

[2186] User Registration and Profiling

[2187] Step 1:

[2188] The user starts up the device and says, "I'd like to register my information."

[2189] Your device uses speech recognition to convert speech into text.

[2190] Step 2:

[2191] The terminal analyzes the converted text and displays a screen for entering user information.

[2192] Step 3:

[2193] The user enters basic information such as name, age, address, and medical history.

[2194] Step 4:

[2195] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[2196] Step 5:

[2197] The server parses the received JSON data and stores the information in the database.

[2198] Step 6:

[2199] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[2200] Step 7:

[2201] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[2202] Daily communication

[2203] Step 1:

[2204] In the morning, the user says "Good morning" to the terminal.

[2205] Step 2:

[2206] The device converts the speech into text and sends the text to the server.

[2207] Step 3:

[2208] The server receives the text and analyzes the user's emotions using an emotion engine.

[2209] Step 4:

[2210] Based on the results of the sentiment analysis, the server generates a response text such as "Good morning. The weather is sunny today and you look like you're in a good mood."

[2211] Step 5:

[2212] The server generates a response and sends it to the terminal.

[2213] Step 6:

[2214] The device converts the received response text into speech and tells the user, "Good morning. The weather is sunny today and you look like you're in a good mood."

[2215] Searching for and providing information

[2216] Step 1:

[2217] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[2218] Step 2:

[2219] The device converts the speech into text and sends the text to the server.

[2220] Step 3:

[2221] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[2222] Step 4:

[2223] The server analyzes the contact information it finds using an emotion engine and creates information in a format that corresponds to the user's emotions.

[2224] Step 5:

[2225] The server sends the information to the terminal.

[2226] Step 6:

[2227] The information received by the terminal is displayed and output to the user as audio, saying, "The contact number for the nearest welfare center is XXX-XXXX-XXXX. Please feel free to contact us at any time."

[2228] Health Management and Emergency Response

[2229] Step 1:

[2230] The user tells the terminal that he is "feeling unwell."

[2231] Step 2:

[2232] The device detects this sound and goes into emergency call mode.

[2233] Step 3:

[2234] The device sends profile data (such as medical history) and status to the server.

[2235] Step 4:

[2236] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[2237] Step 5:

[2238] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called. Please wait a moment."

[2239] Step 6:

[2240] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[2241] Online shopping support

[2242] Step 1:

[2243] The user makes a request to the terminal saying, "I want to buy bread."

[2244] Step 2:

[2245] The device converts the speech into text and sends it to the server.

[2246] Step 3:

[2247] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[2248] Step 4:

[2249] The server analyzes the found product list using an emotion engine and presents it in an appropriate format.

[2250] Step 5:

[2251] The server sends the product list to the terminal.

[2252] Step 6:

[2253] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[2254] Step 7:

[2255] If the user answers "yes," the terminal sends the information to the server.

[2256] Step 8:

[2257] The server completes the final purchase and sends a confirmation message to the terminal.

[2258] Step 9:

[2259] The terminal will display and audibly inform the user that "Purchase completed."

[2260] Through these steps, the system can provide multifaceted support for elderly people living alone and provide services that take users' emotions into consideration.

[2261] Example 2

[2262] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2263] In modern society, the number of elderly people living alone is increasing, making support for daily life and emergency response important issues. Furthermore, providing appropriate responses according to emotions is required to reduce the sense of isolation felt by elderly people. Conventional speech recognition systems have difficulty analyzing a user's emotions and generating appropriate responses. Furthermore, their ability to provide information needed for daily life and emergency response is limited. Therefore, a comprehensive system is needed that can provide multifaceted support for the daily life of elderly people and provide appropriate responses according to their emotions.

[2264] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for transmitting information input by a user to the information processing device via a terminal, and the information processing device generating a user profile based on the information; means for recognizing a voice input by the user and converting the voice into text; means for transmitting the text to the information processing device and generating a response based on the received text and emotion analysis; means for the terminal to convert the response into speech and provide it to the user again; means for the information processing device to search for information corresponding to the user's request from a network or database and provide it to the terminal; means for inputting information about the user's health condition, transmitting it to the information processing device via the terminal, and automatically contacting an emergency contact; and means for transmitting information about a product desired by the user via the terminal to the information processing device, and the information processing device searching for the product on a related e-commerce site and completing a purchase procedure. This makes it possible to comprehensively support the daily lives of elderly people and provide appropriate responses according to their emotions.

[2265] "Information processing device" refers to a device or system that inputs, processes, stores, and outputs data. Generally, this includes servers and computers.

[2266] "User profile" refers to a user's personal information data, including basic information such as the user's name, age, address, medical history, and sentiment analysis results.

[2267] "Speech recognition" refers to the technology of analyzing a user's voice input and converting it into text data.

[2268] "Emotion analysis" refers to the technology of analyzing a user's emotional state from text data and voice data.

[2269] "Response generation" refers to the process of creating an appropriate response to the user based on the analyzed information.

[2270] A "network" refers to a system that connects multiple computers and devices and enables the exchange of data.

[2271] A "database" refers to a system for efficiently managing, searching, and storing data.

[2272] "Encryption protocol" refers to the technology used to encrypt data in order to ensure secure data communication.

[2273] "Emergency contact" refers to contact information that is set to be contacted in the event of a sudden change in the user's health condition or an emergency.

[2274] "E-commerce site" refers to a website that allows users to sell or purchase goods or services over the Internet.

[2275] This invention is a comprehensive service system that provides multifaceted support for elderly people living alone. Its key feature is its ability to recognize the user's emotions and provide appropriate responses and support based on those emotions. This system functions in coordination with three entities: a server, a terminal, and a user, and combines an emotion engine to enhance the user experience. Below, we will explain in detail each component of this system and its operation.

[2276] Hardware and Software Use

[2277] This system uses several key hardware and software components. The terminal is a device that performs speech recognition, speech synthesis, and display. Specifically, it is a smart device equipped with a microphone, speaker, and display. The system uses the Google Cloud Speech-to-Text API for speech recognition and the Google Cloud Text-to-Speech API for speech synthesis. It also uses the IBM Watson Tone Analyzer for user sentiment analysis.

[2278] The server manages user profiles, analyzes emotions, generates responses, manages databases, and performs network communications. The server is a computer with high-performance computing capabilities and uses MySQL as its database. The HTTPS protocol is used as the communications protocol to ensure security.

[2279] System Operation

[2280] User Registration and Profiling

[2281] The user turns on the device and says, "I'd like to register my information." The device uses the Google Cloud Speech-to-Text API to convert the speech to text and analyzes that text. A user information input screen is displayed, and the user enters information such as their name, age, address, and medical history. The device converts this information into JSON format and sends it to the server using the HTTPS protocol. The server analyzes the received data and stores it in a MySQL database. Once the profile is generated, a confirmation message is displayed on the device and a voice notification is played saying, "Registration is complete."

[2282] Daily communication

[2283] In the morning, when a user says "Good morning" to the device, the device converts the speech into text and sends it to the server. The server performs text and sentiment analysis and generates an appropriate response. The generated response is sent to the device, which converts it into speech and conveys it to the user. For example, the server might generate a response like "Good morning. The weather is sunny today, and it looks like it's going to be a pleasant day."

[2284] Searching for and providing information

[2285] When a user says to the device, "Tell me the contact information for the nearest welfare center," the device converts the speech into text and sends it to the server. The server analyzes the text, searches for contact information using a database or external API, and provides information in an appropriate format based on the user's emotion. For example, the server might generate a message saying, "The contact information for the nearest welfare center is XXX-XXXX-XXXX. If you have any concerns, you can contact them immediately," and the device will notify the message aloud.

[2286] Health Management and Emergency Response

[2287] When the user tells the device that they are "feeling unwell", the device goes into emergency call mode and sends their profile data and situation to the server. The server retrieves emergency contacts and contacts the appropriate resource (e.g., emergency services). The server executes the emergency call and sends feedback to the device saying "An ambulance has been called." The device then tells the user by voice, "An ambulance has been called. Please remain calm."

[2288] Online shopping support

[2289] When a user requests "I want to buy bread" from the device, the device converts the speech into text and sends it to the server. The server analyzes the text and searches for product lists from affiliated online shopping sites. It performs sentiment analysis on the found product list and presents it to the user in an appropriate format. If the user responds "Yes," the server carries out the final purchase procedure and sends a confirmation message to the device. The device then displays and sounds a message saying "Purchase completed."

[2290] Specific examples

[2291] For example, when a user says "Good morning" to a device in the morning, the device converts the speech into text and sends it to the server. The server analyzes the received text and generates a response based on the results of emotion analysis. This series of steps allows the user to receive a response that reflects their emotions. The specific prompt text is as follows:

[2292] When you say "Good morning," you respond with:

[2293] "Good morning. The weather is sunny today and it looks like it's going to be a pleasant day."

[2294] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2295] User Registration and Profiling

[2296] Step 1:

[2297] The user starts up the device and speaks, "I'd like to register my information." Once the voice input is complete, the device receives the voice data.

[2298] Input: User voice input

[2299] Output: Audio data

[2300] What happens: The device's microphone captures the user's voice.

[2301] Step 2:

[2302] The device uses the Google Cloud Speech-to-Text API to convert speech to text.

[2303] Input: Audio data

[2304] Output: Text data

[2305] Specific operation: The audio data is sent to the Google Cloud Speech-to-Text API and returned as text data.

[2306] Step 3:

[2307] The terminal analyzes the converted text and displays a screen for entering user information.

[2308] Input: Text data

[2309] Output: User information input screen

[2310] Specific operation: The terminal analyzes the text data and generates a graphical user interface based on its content.

[2311] Step 4:

[2312] The user enters information such as name, age, address, medical history, etc. The terminal receives this input and examines the data in real time.

[2313] Input: User input information

[2314] Output: Validated input data

[2315] Specific operation: The terminal checks the data type and format of each field in real time.

[2316] Step 5:

[2317] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[2318] Input: Validated input data

[2319] Output: JSON data

[2320] Specific operation: The device formats the data into JSON format and sends it to the server over a secure communication channel.

[2321] Step 6:

[2322] The server analyzes the received data and stores it in a MySQL database.

[2323] Input: JSON data

[2324] Output: User information stored in the database

[2325] What happens: The server deserializes the data and performs an insert operation into the database.

[2326] Step 7:

[2327] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[2328] Input: User information stored in the database

[2329] Output:Confirmation message

[2330] Specific operations: The server configures the user profile and sends a message to the terminal indicating success.

[2331] Step 8:

[2332] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[2333] Input:Confirmation message

[2334] Output: Visual and audio notifications

[2335] Specific operation: The device displays the message on the screen and converts it into audio using the Google Cloud Text-to-Speech API and plays it back.

[2336] Daily communication

[2337] Step 1:

[2338] The user speaks "Good morning" to the device in the morning, and the device recognizes the voice.

[2339] Input: User voice input

[2340] Output: Audio data

[2341] Specific operation: The device's microphone captures audio data.

[2342] Step 2:

[2343] The device uses the Google Cloud Speech-to-Text API to convert speech to text.

[2344] Input: Audio data

[2345] Output: Text data

[2346] Specific operation: The device sends voice data to the Google Cloud Speech-to-Text API and receives it as text data.

[2347] Step 3:

[2348] The terminal sends the converted text to the server.

[2349] Input: Text data

[2350] Output: Text data sent to the server

[2351] Specific operation: The terminal communicates via the Internet to send text data to the server.

[2352] Step 4:

[2353] The server receives the text data and performs sentiment analysis using IBM Watson Tone Analyzer.

[2354] Input: Text data

[2355] Output: Emotion data

[2356] Specific operation: The server sends the text data to the IBM Watson Tone Analyzer and obtains the analysis results.

[2357] Step 5:

[2358] The server generates a response text based on the sentiment analysis results.

[2359] Input: Emotion data

[2360] Output: Response text

[2361] Specific operation: The server analyzes the emotional data and generates an appropriate response as text.

[2362] Step 6:

[2363] The server generates a response text and sends it to the terminal.

[2364] Input: Response text

[2365] Output: Response text sent to the terminal

[2366] Specific operation: The server communicates via the Internet to send the response text to the terminal.

[2367] Step 7:

[2368] The device converts the received response text into speech using the Google Cloud Text-to-Speech API and conveys it to the user.

[2369] Input: Response text

[2370] Output: Voice response

[2371] What it does: Your device converts the text into speech and plays it through the speaker.

[2372] Searching for and providing information

[2373] Step 1:

[2374] The user speaks to the device, saying, "Please tell me the contact information for the nearest welfare center." The device recognizes the voice.

[2375] Input: User voice input

[2376] Output: Audio data

[2377] Specific operation: The device's microphone captures audio data.

[2378] Step 2:

[2379] The device uses the Google Cloud Speech-to-Text API to convert speech to text.

[2380] Input: Audio data

[2381] Output: Text data

[2382] Specific operation: Sends audio data to the API and retrieves text data.

[2383] Step 3:

[2384] The terminal sends the converted text to the server.

[2385] Input: Text data

[2386] Output: Text data sent to the server

[2387] Specific operation: To send text data to the server, the HTTPS protocol is used for communication.

[2388] Step 4:

[2389] The server receives the text data and searches for "welfare center contact information" using an internal database or external API.

[2390] Input: Text data

[2391] Output: Contact information

[2392] What it does: Parses text data and queries databases and APIs to retrieve contact information.

[2393] Step 5:

[2394] The server performs sentiment analysis on the contact information and generates it in a format that corresponds to the user's sentiment.

[2395] Input: Contact Information

[2396] Output: Formatted response text

[2397] Specific operation: Based on the results of sentiment analysis, the contact information is formatted into appropriate text.

[2398] Step 6:

[2399] The server sends the information to the terminal.

[2400] Input: Formatted response text

[2401] Output: Response text sent to the terminal

[2402] What it does: Sends text to the device using the HTTPS protocol.

[2403] Step 7:

[2404] The information received by the device is converted into audio using the Google Cloud Text-to-Speech API, and is displayed and audio notified.

[2405] Input: Formatted response text

[2406] Output: Visual and audio notifications

[2407] Specific behavior: Converts text to speech, displays it on the display, and plays it through the speaker.

[2408] Health Management and Emergency Response

[2409] Step 1:

[2410] The user says to the device, "I feel sick." The device recognizes the voice.

[2411] Input: User voice input

[2412] Output: Audio data

[2413] Specific operation: The device's microphone captures audio data.

[2414] Step 2:

[2415] The device enters emergency call mode and transmits profile data and status to the server.

[2416] Input: Audio data

[2417] Output: Profile and status data sent to the server

[2418] Specific operation: Activate emergency mode and send necessary data to the server.

[2419] Step 3:

[2420] The server retrieves the emergency contact information from the database and contacts the appropriate resource.

[2421] Input: Profile data and situational data

[2422] Output: Emergency contact

[2423] Specific behavior: Retrieves emergency contact information from a database and contacts emergency response services.

[2424] Step 4:

[2425] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called."

[2426] Input: Emergency Contact Information

[2427] Output: Feedback message

[2428] Specific operation: Make an emergency call and send the results to the device.

[2429] Step 5:

[2430] The device will notify the user by voice, "An ambulance has been called. Please remain calm."

[2431] Input: Feedback message

[2432] Output: Audio notification

[2433] Specific behavior: Convert the message into audio and notify the user.

[2434] Online shopping support

[2435] Step 1:

[2436] The user requests the device to "buy bread." The device recognizes the voice.

[2437] Input: User voice input

[2438] Output: Audio data

[2439] Specific operation: The device's microphone captures audio data.

[2440] Step 2:

[2441] The device uses the Google Cloud Speech-to-Text API to convert speech to text.

[2442] Input: Audio data

[2443] Output: Text data

[2444] Specific operation: Sends audio data to the API and retrieves text data.

[2445] Step 3:

[2446] The terminal sends the converted text to the server.

[2447] Input: Text data

[2448] Output: Text data sent to the server

[2449] Specific operation: To send text data to the server, the HTTPS protocol is used for communication.

[2450] Step 4:

[2451] The server analyzes the text data and searches for product lists from affiliated online shopping sites.

[2452] Input: Text data

[2453] Output: Product list

[2454] Specific operation: The server parses the text data, creates a query, and sends it to the shopping site's API.

[2455] Step 5:

[2456] The server performs sentiment analysis on the product list and generates it in the appropriate format.

[2457] Input: Product list

[2458] Output: A formatted product list

[2459] Specific operation: Format product list based on sentiment analysis results.

[2460] Step 6:

[2461] The server sends the product list to the terminal.

[2462] Input: Formatted product list

[2463] Output: Product list sent to the terminal

[2464] Specific operation: The product list is sent to the terminal using the HTTPS protocol.

[2465] Step 7:

[2466] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[2467] Input: Formatted product list

[2468] Output: Display and audio queries

[2469] Specific operation: Display a product list on the screen and ask questions by voice.

[2470] Step 8:

[2471] If the user answers "yes," the terminal sends the information to the server.

[2472] Input: Voice response

[2473] Output: Response sent to the server

[2474] Specific operation: The device converts the voice response into text and sends it to the server.

[2475] Step 9:

[2476] The server completes the final purchase and sends a confirmation message to the terminal.

[2477] Input: Response data

[2478] Output:Confirmation message

[2479] Specific operation: The server executes the purchase procedure and sends the results to the terminal.

[2480] Step 10:

[2481] The terminal will notify the user by displaying and audibly saying "Purchase completed."

[2482] Input:Confirmation message

[2483] Output: Visual and audio notifications

[2484] Specific operation: A confirmation message is displayed on the screen and played back as audio.

[2485] (Application example 2)

[2486] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2487] When elderly people shop alone in physical stores, they face the challenge of finding it difficult to obtain information and advice about products, making it difficult for them to enjoy shopping with peace of mind. Furthermore, due to a lack of individualized support tailored to the elderly's emotions, they often feel isolated and anxious. Furthermore, there is a need for a system that can respond quickly in emergencies. It is necessary to solve these problems and provide an environment where elderly people can enjoy shopping with peace of mind.

[2488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2489] In this invention, the server includes: means for transmitting information input by a user to the server through a terminal and having the server generate a user profile based on the information; means for recognizing a voice input by the user and converting the voice to text; means for transmitting the text to the server and generating a response based on the text received by the server; means for the terminal to convert the response to voice and provide it to the user again; means for the server to search the Internet or a database for information corresponding to the user's request and provide it to the terminal; means for inputting information about the user's health condition and transmitting it to the server through the terminal, which then automatically contacts an emergency contact; means for transmitting information about a product desired by the user to the server through the terminal, which then searches for the product on affiliated e-commerce sites and completes the purchase process; and a smart brick-and-mortar store support application that has an engine for analyzing user emotions and provides responses and support based on the user's emotions. This allows elderly people to enjoy shopping with peace of mind and receive the information and support they need appropriately.

[2490] A "user profile" is an information structure that includes individual user characteristics and background information, and is generated based on information entered by the user.

[2491] "Voice input" is the process by which a device receives and analyzes what a user says as sound.

[2492] "Text conversion" is the process of analyzing information input via voice and converting it into text.

[2493] A "server" is a central processing unit that stores, processes, sends and receives data over a network.

[2494] "Response generation" is the process of creating information to respond to the user based on the received text.

[2495] An "emotion analysis engine" is software or an algorithm that determines emotions from input information and provides appropriate responses or support based on the results.

[2496] The "Smart Physical Store Support Application" is an application designed to support users' shopping experiences in physical stores, and is a system that handles user input, analyzes emotions, and provides necessary information.

[2497] An "Internet search" is the process of utilizing a wide range of networks to locate and provide information requested by a user.

[2498] "Emergency contact method" is a process in which emergency information such as the user's health condition is sent to a server, and the server automatically contacts the appropriate emergency contact.

[2499] "Product information search" is the process of finding and providing information about the product desired by the user from e-commerce sites and other sources.

[2500] This invention is a support system that allows elderly people to enjoy shopping in brick-and-mortar stores with peace of mind. In particular, it has the feature of recognizing the user's emotions and providing appropriate responses and support based on those emotions.

[2501] System Configuration

[2502] The system is composed of a terminal used by the user (smart glasses, smartphone, etc.), a server that performs central processing functions, sensors for acquiring data on the user's behavior and health status, etc. Specific embodiments of the present invention will be described below in order.

[2503] User Registration and Profiling

[2504] 1. User Registration:

[2505] The user starts up the device and registers basic information (such as name, age, address, medical history, etc.) by voice or manual input. This information is sent from the device to the server, which then creates a user profile and stores it in a database.

[2506] After the profile is generated, a confirmation message is sent back to the terminal and the user is notified audibly and visually.

[2507] Daily shopping assistance

[2508] 2. Voice to text conversion:

[2509] While shopping, users input voice commands into the device, such as "Tell me about this product," which are converted into text via speech recognition software such as the Google Speech-to-Text API.

[2510] The converted text is sent to the server using the HTTPS protocol.

[2511] 3. Sentiment analysis and response generation:

[2512] The server uses a sentiment analysis engine such as Amazon Comprehend to analyze sentiment from the user's text input.

[2513] Based on the results of emotion analysis, appropriate responses and support information are generated for the user and sent from the server to the terminal.

[2514] 4. Response provision:

[2515] The response text is converted into speech at the terminal and provided to the user in speech and text.

[2516] Emergency response and health management

[2517] 5. Emergency Response:

[2518] When a user tells the terminal that they are "feeling unwell," the terminal converts this speech into text and sends it to the server.

[2519] The server consults the user profile and emergency contact database and contacts the appropriate resources (such as emergency services).

[2520] Emergency response information is communicated to the user via the terminal by voice.

[2521] Searching and providing product information

[2522] 6. Product Search:

[2523] When a user types "Tell me about this product," the server searches for detailed information about the product from affiliated e-commerce sites, performs sentiment analysis on the results, and then provides them to the user.

[2524] Examples of concrete examples and prompts

[2525] For example, when a user speaks to a terminal in a store and says, "Tell me about this product," the terminal converts the speech into text and sends it to the server. The server analyzes the text and the user's emotions and generates appropriate product information and recommendations. This process allows the user to receive a response that reflects their emotions, allowing them to enjoy shopping with peace of mind.

[2526] Prompt Sentence Examples

[2527] “When a user says, ‘Tell me about this product,’ provide a product description and offer recommendations based on the user’s emotions. For example, if the user seems anxious, highlight how the product will help them.”

[2528] By utilizing generative AI models and prompts, this system provides an environment where seniors can enjoy shopping in physical stores with peace of mind.

[2529] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2530] Step 1:

[2531] User Registration:

[2532] The user starts up the device and registers basic information (name, age, address, medical history, etc.) by voice input or manual input.

[2533] The device converts this input information into text, formats it as JSON, and sends it to the server using the HTTPS protocol.

[2534] The server parses the received JSON data, generates a user profile, and saves it in the database.

[2535] Based on the generated profile, the server creates a confirmation message and sends it to the terminal.

[2536] The device will notify the user of a confirmation message by voice and text.

[2537] Step 2:

[2538] Speech to text:

[2539] While shopping, the user issues a voice command such as "Tell me about this product."

[2540] The device converts voice input into text using the Google Speech-to-Text API.

[2541] This text data is sent to the server using the HTTPS protocol.

[2542] Step 3:

[2543] Sentiment analysis and response generation:

[2544] The server analyzes the user's emotions from the received text data using an emotion analysis engine such as Amazon Comprehend.

[2545] Based on the sentiment analysis results and the text, the server generates a prompt sentence, which is then used to generate an appropriate response text.

[2546] The response text might be generated in the form of, for example, "This product contains healthy ingredients and is ideal for the elderly."

[2547] Step 4:

[2548] Response provided:

[2549] The server generates a response text and sends it to the terminal.

[2550] The terminal converts the received response text into speech and provides it to the user in speech and text.

[2551] Step 5:

[2552] Product Search:

[2553] The user again instructs the terminal to "tell me about this product."

[2554] The device converts the speech into text and sends the text to the server.

[2555] The server uses the text to search for product information using the API of the e-commerce site.

[2556] The server retrieves the search results, simultaneously performs sentiment analysis, and provides information in a format that corresponds to the user's emotions.

[2557] The server sends this product information to the terminal, and the terminal informs the user by voice and text, "The price of this product is XXX yen and it is very popular."

[2558] Step 6:

[2559] Emergency Response:

[2560] The user tells the terminal that he is "feeling unwell."

[2561] The device converts the voice into text and sends it to the server as emergency information.

[2562] The server consults the user profile and emergency contact database and contacts the appropriate emergency contact (eg, emergency services).

[2563] The server generates progress information on the emergency response and transmits it to the terminal.

[2564] The terminal will tell the user by voice, "An ambulance has been called. Please wait without worry."

[2565] Through these processing steps, the system can quickly and accurately provide the information and support necessary for seniors to enjoy shopping safely in physical stores.

[2566] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2567] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2568] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2569] [Fourth embodiment]

[2570] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2571] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2572] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2573] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2574] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2575] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2576] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2577] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2578] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2579] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2580] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2581] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2582] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2583] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone. This system functions by interoperating with three entities: a server, a terminal, and a user.

[2584] Program processing details

[2585] 1. User Registration and Profiling

[2586] The user turns on the device (e.g., a smartphone or tablet) and speaks, "I would like to register my information."

[2587] The device converts the voice to text and displays a screen for entering information.

[2588] The user enters basic information such as name, age, address, and medical history.

[2589] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[2590] The server parses the received JSON data and stores the information in the database.

[2591] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[2592] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[2593] 2. Daily communication

[2594] In the morning, the user says "Good morning" to the terminal.

[2595] The device converts the speech into text and sends the text to the server.

[2596] The server receives the text and generates a response text saying "Good morning. The weather is sunny today."

[2597] The server generates a response and sends it to the terminal.

[2598] The terminal converts the received response text into speech and tells the user, "Good morning. The weather is sunny today."

[2599] 3. Searching for and providing information

[2600] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[2601] The device converts the speech into text and sends the text to the server.

[2602] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[2603] The server sends the contact information it finds to the device.

[2604] The information received by the terminal is displayed and output to the user as audio, such as "The contact information for the nearest welfare center is XXX-XXXX-XXXX."

[2605] 4. Health Management and Emergency Response

[2606] The user tells the terminal that he is "feeling unwell."

[2607] The device detects this sound and goes into emergency call mode.

[2608] The device sends profile data (such as medical history) and status to the server.

[2609] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[2610] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called."

[2611] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[2612] 5. Online shopping support

[2613] The user makes a request to the terminal saying, "I want to buy bread."

[2614] The device converts the speech into text and sends it to the server.

[2615] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[2616] The server sends a list of the found products to the terminal.

[2617] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[2618] If the user answers "yes," the terminal sends the information to the server.

[2619] The server completes the final purchase and sends a confirmation message to the terminal.

[2620] The terminal will display and audibly inform the user that "Purchase completed."

[2621] Specific examples

[2622] For example, when a user wakes up in the morning and says "Good morning" to the device, the device converts the speech into text and sends it to the server. The server generates a response, "Good morning. The weather is sunny today," and sends it back to the device. The device then converts this response into speech and conveys it to the user. This series of steps allows users to engage in everyday communication without feeling isolated.

[2623] This system is divided into detailed processing steps, and by performing specific actions at each step, it provides multifaceted support for elderly people living alone.

[2624] The processing flow will be explained below.

[2625] User Registration and Profiling

[2626] Step 1:

[2627] The user starts up the device and says, "I'd like to register my information."

[2628] Your device uses speech recognition to convert speech into text.

[2629] Step 2:

[2630] The terminal analyzes the converted text and displays a screen for entering user information.

[2631] Step 3:

[2632] The user enters basic information such as name, age, address, and medical history.

[2633] Step 4:

[2634] The terminal converts this information into JSON format and sends it to the server using the HTTPS protocol.

[2635] Step 5:

[2636] The server parses the received JSON data and stores the information in the database.

[2637] Step 6:

[2638] The server generates the user profile and sends a confirmation message to the terminal saying "Profile generated."

[2639] Step 7:

[2640] The device will display a confirmation message to the user and also provide an audio notification that "Registration is complete."

[2641] Daily communication

[2642] Step 1:

[2643] In the morning, the user says "Good morning" to the terminal.

[2644] Step 2:

[2645] The device converts the speech into text and sends the text to the server.

[2646] Step 3:

[2647] The server receives the text and generates a response text saying "Good morning. The weather is sunny today."

[2648] Step 4:

[2649] The server generates a response and sends it to the terminal.

[2650] Step 5:

[2651] The terminal converts the received response text into speech and tells the user, "Good morning. The weather is sunny today."

[2652] Searching for and providing information

[2653] Step 1:

[2654] The user tells the terminal, "Please tell me the contact information for the nearest welfare center."

[2655] Step 2:

[2656] The device converts the speech into text and sends the text to the server.

[2657] Step 3:

[2658] The server analyzes the received text and searches for "welfare center contact information" using an internal database or external API.

[2659] Step 4:

[2660] The server sends the contact information it finds to the device.

[2661] Step 5:

[2662] The information received by the terminal is displayed and output to the user as audio, such as "The contact information for the nearest welfare center is XXX-XXXX-XXXX."

[2663] Health Management and Emergency Response

[2664] Step 1:

[2665] The user tells the terminal that he is "feeling unwell."

[2666] Step 2:

[2667] The device detects this sound and goes into emergency call mode.

[2668] Step 3:

[2669] The device sends profile data (such as medical history) and status to the server.

[2670] Step 4:

[2671] The server retrieves emergency contact information from a database and contacts the appropriate resources (e.g., emergency services).

[2672] Step 5:

[2673] The server makes the emergency call and sends feedback to the device saying, "An ambulance has been called."

[2674] Step 6:

[2675] The device will tell the user by voice, "An ambulance has been called. Please remain calm."

[2676] Online shopping support

[2677] Step 1:

[2678] The user makes a request to the terminal saying, "I want to buy bread."

[2679] Step 2:

[2680] The device converts the speech into text and sends it to the server.

[2681] Step 3:

[2682] The server analyzes the text and searches for bread product listings from affiliated online shopping sites.

[2683] Step 4:

[2684] The server sends a list of the found products to the terminal.

[2685] Step 5:

[2686] The terminal displays a list of products to the user and asks aloud, "Is this okay?"

[2687] Step 6:

[2688] If the user answers "yes," the terminal sends the information to the server.

[2689] Step 7:

[2690] The server completes the final purchase and sends a confirmation message to the terminal.

[2691] Step 8:

[2692] The terminal will display and audibly inform the user that "Purchase completed."

[2693] By going through the above steps, this system can provide multifaceted support to elderly people living alone.

[2694] Example 1

[2695] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2696] Elderly people living alone face a wide range of challenges, including daily communication, health management, and emergency response. To efficiently and comprehensively address these challenges, an easy-to-use system is required. However, current technology provides solutions for individual problems in a fragmented manner, preventing a consistent user experience.

[2697] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2698] In this invention, the server includes means for transmitting information input by a user to the server via a terminal and for the server to generate a user profile based on the information, means for recognizing a voice input by the user and converting the voice into text, and means for transmitting the text to the server and generating a response based on the text received by the server, thereby enabling multifaceted support for elderly people in their daily activities such as communication, emergency response, health management, and online shopping.

[2699] "Users" refers to the elderly people who use the system and the people who support them.

[2700] A "terminal" is a device operated by a user, and refers to information and communication devices such as smartphones and tablets.

[2701] "Server" refers to a central computing device that processes information sent from terminals and manages it in a database.

[2702] A "user profile" refers to a data set that includes personal information such as a user's basic information and medical history.

[2703] "Voice input" refers to voice commands given by a user through a microphone.

[2704] "Speech recognition" refers to the technology that converts voice input into text form.

[2705] "Text conversion" refers to the process of processing text obtained by speech recognition as data.

[2706] "Secure Protocol" refers to a secure communications protocol such as HTTPS.

[2707] "Generate" refers to the process by which the server generates text or audio information.

[2708] "Search" refers to the operation by which a server retrieves information using the Internet or an internal database.

[2709] "Emergency contact" refers to the procedure by which the server automatically notifies pre-defined contacts in the event of an emergency.

[2710] "Shopping Site" refers to the affiliated e-commerce platform.

[2711] "Voice notification" refers to a function in which a terminal uses a voice conversion function to convey information to a user.

[2712] This invention is a comprehensive service system for providing multifaceted support to elderly people living alone. This system functions by interoperating with three entities: a server, a terminal, and a user.

[2713] First, the user uses a smartphone or tablet as a device. These devices have speech recognition capabilities and convert the user's voice input into text using the Google Speech-to-Text API. The device also converts text into speech using the Google Text-to-Speech API.

[2714] Next, the information entered by the user is sent to the server via the terminal. For example, when a user enters basic information such as name, age, address, and medical history, this information is converted into JSON format and sent to the server using the HTTPS protocol. The server analyzes the received information, stores it in a MySQL database, and generates a user profile.

[2715] In everyday communication, when a user speaks to a device, for example, "Good morning," the device converts the speech into text and sends it to the server. The server generates a response text and sends it to the device. The device then converts the received response text into speech and conveys it to the user. For example, "Good morning. The weather is sunny today."

[2716] Also, when a user says to the device, "Please tell me the contact information for the nearest welfare center," the device converts the voice into text and sends that text to the server. The server uses an internal database or an external API (such as Google Maps API) to search for the welfare center's contact information and sends that information to the device. The device then converts the received information into voice and relays it to the user.

[2717] Furthermore, if the user tells the device that they are "feeling unwell," the device will enter emergency call mode and send their profile data and status to the server. The server will retrieve emergency contact information from a database and contact the appropriate resource (e.g., emergency services). The server will then send a confirmation message to the device saying "an ambulance has been called," which the device will then verbally convey to the user.

[2718] For online shopping support, when a user requests the device to "buy bread," the device converts the voice to text and sends it to the server. The server then searches for a product list from affiliated shopping sites and sends it to the device. The device then displays the product list to the user and asks aloud, "Is this OK?" If the user answers "yes," the device completes the final purchase procedure and a confirmation message is spoken by the device.

[2719] Specific examples

[2720] For example, if a user says "Good morning" to a device in the morning, the device converts the speech into text and sends it to the server. The server generates a response such as "Good morning. The weather is sunny today" and sends it back to the device. The device then converts this response into speech and conveys it to the user. This series of steps allows users to engage in everyday communication without feeling isolated.

[2721] Examples of prompts include "Good morning," "Please tell me the contact information of the nearest welfare center," "I'm not feeling well," "I want to buy some bread," etc. Through such prompts, it is possible to provide the multifaceted support that the elderly require in their daily lives.

[2722] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2723] Program processing flow

[2724] User Registration and Profiling

[2725] Step 1:

[2726] The user starts up the device and says, "I'd like to register my information."

[2727] Input: Audio

[2728] Output: Audio data

[2729] Specific operation: The user uses a smartphone or tablet to input voice information.

[2730] Step 2:

[2731] The device converts the speech to text.

[2732] Input: Audio data

[2733] Output: Text data

[2734] What happens: The device converts speech to text using the Google Speech-to-Text API.

[2735] Step 3:

[2736] The user enters basic information such as name, age, address, and medical history.

[2737] Input: User basic information

[2738] Output: User basic information data

[2739] Specific operation: The user enters information into an input form displayed on the device screen.

[2740] Step 4:

[2741] The device converts the information into JSON format.

[2742] Input: User basic information data

[2743] Output: JSON data

[2744] Specific operation: The terminal converts the input data into JSON format.

[2745] Step 5:

[2746] The terminal sends data to the server.

[2747] Input: JSON data

[2748] Output: Send data to the server

[2749] Specific operation: The device sends data to the server using the HTTPS protocol.

[2750] Step 6:

[2751] The server analyzes the data and stores it in a database.

[2752] Input: JSON data

[2753] Output: Store in database

[2754] Specific operation: The server analyzes the received data and stores it in a MySQL database.

[2755] Step 7:

[2756] The server generates a confirmation message and sends it to the terminal.

[2757] Input: Database update information

[2758] Output:Confirmation message

[2759] Specific operation: The server creates a message saying "Profile has been generated" and sends it to the terminal.

[2760] Step 8:

[2761] The device will notify the user by voice.

[2762] Input:Confirmation message

[2763] Output: Audio notification

[2764] Specific behavior: The device uses the Google Text-to-Speech API to convert the text to speech and notifies the user that "Registration is complete."

[2765] Daily communication

[2766] Step 1:

[2767] The user says "Good morning" to the terminal.

[2768] Input: Audio

[2769] Output: Audio data

[2770] Specific operation: The user greets the device by voice.

[2771] Step 2:

[2772] The device converts the speech to text.

[2773] Input: Audio data

[2774] Output: Text data

[2775] What happens: The device converts speech to text using the Google Speech-to-Text API.

[2776] Step 3:

[2777] The device sends the text to the server.

[2778] Input: Text data

[2779] Output: Send text data to the server

[2780] Specific operation: The device sends text data to the server via HTTP.

[2781] Step 4:

[2782] The server parses the text and generates a response.

[2783] Input: Text data

[2784] Output: Response text

[2785] Specific operation: The server analyzes the received text data and generates a response saying, "Good morning. The weather is sunny today."

[2786] Step 5:

[2787] The server sends the response text to the terminal.

[2788] Input: Response text

[2789] Output: Response sent from the server

[2790] Specific operation: The response text generated by the server is sent to the terminal.

[2791] Step 6:

[2792] The terminal converts the response into voice and conveys it to the user.

[2793] Input: Response text

[2794] Output: Audio notification

[2795] What happens: The device uses the Google Text-to-Speech API to convert text to speech and tells the user, "Good morning. The weather is sunny today."

[2796] Searching for and providing information

[2797] Step 1:

[2798] The user speaks to the device, saying, "Please tell me the contact information for the nearest welfare center."

[2799] Input: Audio

[2800] Output: Audio data

[2801] Specific behavior: The user makes a request by voice.

[2802] Step 2:

[2803] The device converts the speech to text.

[2804] Input: Audio data

[2805] Output: Text data

[2806] What happens: The device converts speech to text using the Google Speech-to-Text API.

[2807] Step 3:

[2808] The device sends the text to the server.

[2809] Input: Text data

[2810] Output: Send data to the server

[2811] Specific operation: The device sends text data to the server via HTTP.

[2812] Step 4:

[2813] The server retrieves the information.

[2814] Input: Text data

[2815] Output: Search results

[2816] Specific operation: The server searches for "welfare center contact information" using its internal database and Google Maps API.

[2817] Step 5:

[2818] The server sends the found information to the device.

[2819] Input: Search results

[2820] Output: Information sent from the server

[2821] Specific operation: The server sends the search results to the terminal.

[2822] Step 6:

[2823] The device converts the information into voice and conveys it to the user.

[2824] Input: Search results

[2825] Output: Audio notification

[2826] What it does: The device uses the Google Text-to-Speech API to convert the text into speech and tells the user, "The contact information for the nearest welfare center is XXX-XXXX-XXXX."

[2827] Health Management and Emergency Response

[2828] Step 1:

[2829] The user tells the terminal that he is "feeling unwell."

[2830] Input: Audio

[2831] Output: Audio data

[2832] Specific operation: The user vocally complains about feeling unwell.

[2833] Step 2:

[2834] The device converts the speech to text.

[2835] Input: Audio data

[2836] Output: Text data

[2837] What happens: The device converts speech to text using the Google Speech-to-Text API.

[2838] Step 3:

[2839] The device enters emergency call mode.

[2840] Input: Text data

[2841] Output: Emergency call mode activated

[2842] Specific operation: The device receives the text data and activates emergency call mode.

[2843] Step 4:

[2844] The device sends profile data and status to the server.

[2845] Input: Profile data, text data

[2846] Output: Send data to the server

[2847] Specific operation: The device sends profile data and status to the server.

[2848] Step 5:

[2849] The server retrieves the emergency contact information.

[2850] Input: Profile data

[2851] Output: Emergency contact information

[2852] Specific operation: The server retrieves emergency contact information from the database.

[2853] Step 6:

[2854] The server contacts emergency services.

[2855] Input: Emergency Contact Information

[2856] Output: Emergency contact execution

[2857] Specific Actions: The server contacts emergency services.

[2858] Step 7:

[2859] The server sends a confirmation message to the terminal.

[2860] Input: Contact status

[2861] Output:Confirmation message

[2862] Specific operation: The server sends a confirmation message to the terminal saying "An ambulance has been called."

[2863] Step 8:

[2864] The device will notify the user by voice.

[2865] Input:Confirmation message

[2866] Output: Audio notification

[2867] What happens: The device uses the Google Text-to-Speech API to convert the text into speech and notifies the user, "An ambulance has been called. Please remain calm."

[2868] Online shopping support

[2869] Step 1:

[2870] The user makes a request to the terminal saying, "I want to buy bread."

[2871] Input: Audio

[2872] Output: Audio data

[2873] Specific operation: The user expresses their purchase intention by voice.

[2874] Step 2:

[2875] The device converts the speech to text.

[2876] Input: Audio data

[2877] Output: Text data

[2878] What happens: The device converts speech to text using the Google Speech-to-Text API.

[2879] Step 3:

[2880] The device sends the text to the server.

[2881] Input: Text data

[2882] Output: Send data to the server

[2883] Specific operation: The device sends text data to the server via HTTP.

[2884] Step 4:

[2885] The server parses the text and searches for products.

[2886] Input: Text data

[2887] Output: Product list

[2888] Specific operation: The server searches for a list of bread products using the API of a shopping site it is affiliated with.

[2889] Step 5:

[2890] The server sends the product list to the terminal.

[2891] Input: Product list

[2892] Output: List sent from server

[2893] Specific operation: The server sends the list of found products to the terminal.

[2894] Step 6:

[2895] The device displays a list of products and asks aloud, "Is this okay?"

[2896] Input: Product list

[2897] Output: Display, voice prompts

[2898] Specific operation: The device displays a list of products to the user and asks aloud, "Is this okay?"

[2899] Step 7:

[2900] The user responds "Yes."

[2901] Input: Audio

[2902] Output: Audio data

[2903] What happens: The user confirms the purchase and responds verbally.

[2904] Step 8:

[2905] The terminal sends this response to the server.

[2906] Input: Audio data

[2907] Output: Text data

[2908] Specific operation: The device converts the voice data into text and sends it to the server.

[2909] Step 9:

[2910] The server completes the final purchase and sends a confirmation message to the terminal.

[2911] Input: Text data

[2912] Output:Confirmation message

[2913] Specific operation: The server performs the final purchase procedure, generates a message saying "Purchase completed" and sends it to the terminal.

[2914] Step 10:

[2915] The device will notify you by voice that the purchase is complete.

[2916] Input:Confirmation message

[2917] Output: Audio notification

[2918] What happens: The device uses the Google Text-to-Speech API to convert the text to speech and tells the user, "Your purchase is complete."

[2919] (Application example 1)

[2920] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2921] The lack of comprehensive support systems for the various problems that elderly people face in their daily lives is a challenge. In particular, systems that can quickly respond when elderly people order meals or in emergencies are needed, but the current means to achieve this are insufficient. Furthermore, interactive interfaces are also needed to reduce the burden that elderly people feel when performing these tasks alone.

[2922] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2923] In this invention, the server includes: means for transmitting information input by a user to the server through a terminal and for the server to generate a user profile based on the information; means for recognizing a voice input by the user and converting the voice to text; means for transmitting the text to the server and generating a response based on the text received by the server; means for the terminal to convert the response to voice and provide it to the user again; means for the server to search the Internet or a database for information corresponding to the user's request and provide it to the terminal; means for transmitting an order for a meal desired by the user to the server through the terminal, searching for product lists from affiliated food delivery services with the server, and completing the purchase procedure; means for the server to check the delivery status and obtain information and send it to the...

Claims

1. means for transmitting information input by a user to a server via a terminal, and for the server to generate a user profile based on the information; means for recognizing speech input by a user and converting the speech to text; means for transmitting the text to a server and generating a response based on the text received by the server; a means for converting the response into voice by the terminal and providing it to the user again; A means for the server to search the Internet or a database for information according to the user's request and provide it to the terminal; A means for inputting information on the user's health condition, transmitting it to a server via a terminal, and having the server automatically contact an emergency contact; A means for transmitting information about a product desired by a user to a server via a terminal, and for the server to search for the product on an affiliated shopping site and carry out a purchase procedure; A system including:

2. The system of claim 1 , wherein the user profile is generated by including basic information such as name, age, address, and medical history.

3. 2. The system according to claim 1, wherein the terminal uses the HTTPS protocol in converting speech into text and transmitting the text to the server.

Images