system

Abstract

Provide a system. means for receiving voice input from a user; means for transmitting the received voice data to a server; means for audibly and visually displaying notifications sent from the server; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The problems that this invention aims to solve are social isolation, increased health risks, and inadequate emergency response measures faced by the elderly. In particular, there is a lack of voice recognition devices that are easy for the elderly to use, and support for daily life and health management is insufficient. This leads to a decline in the quality of life of the elderly and the progression of frailty. Another problem is the digital divide, which limits opportunities for the elderly to use the latest technology. [Means for solving the problem]

[0005] The present invention utilizes voice recognition technology to provide a voice recognition IoT device that can be easily used by the elderly. Specifically, the system includes a means for receiving voice input from a user, a means for transmitting the received voice data to a server, and a means for displaying notifications transmitted from the server via voice and on a screen. Furthermore, the system supports the user's health management by including a means for analyzing the user's voice data and detecting changes in their physical condition or lifestyle, a means for generating notifications based on the detected changes, and a means for transmitting the generated notifications to a terminal. The system also includes a means for detecting emergency keyword voices and a means for notifying the server of the detected emergency voices, allowing the elderly to receive prompt assistance. This can improve the quality of life of the elderly and reduce health risks.

[0006] "Users" refers to the elderly people who use the system and their users.

[0007] "Voice input" refers to voice information input by a user speaking into a device.

[0008] "Terminal" refers to the voice recognition IoT device used by the user, which receives voice input and displays notifications, etc.

[0009] "Server" refers to the central processing unit that receives and analyzes data sent from the device and sends notifications and alerts.

[0010] "Voice data" refers to digital data that records what a user says.

[0011] "Notification" refers to a message that conveys information or warnings to the user through audio or visual means.

[0012] "Changes in physical condition or lifestyle" refers to abnormalities or unusual patterns occurring in the user's health condition or daily behavior.

[0013] "Analysis" refers to the act of analyzing audio data and extracting meanings and anomalies from it.

[0014] "Emergency keyword voice" refers to specific phrases or words that a user may utter in an emergency, such as "help me."

[0015] "Health risks" refer to factors that threaten the health of older people, such as frailty, disease progression, and accidents.

[0016] The "digital divide" primarily refers to generational and economic disparities in access to the internet and digital technology.

[0017] "Frailty" refers to a decline in physical function and a state of weakness due to aging.

[0018] "Voice recognition technology" refers to the technology of converting voice into digital data, analyzing its content, and understanding its meaning. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The system of this invention is a voice recognition IoT device that supports the lives of elderly people and aims to make it easy for users (elderly people) to use. This system mainly consists of three components: a terminal, a server, and a user. Each component and its function will be explained in detail below.

[0041] Device Features

[0042] The terminal is a device used by the elderly in their daily lives, and is mainly used to receive voice input, send voice data, and receive and display notifications. The terminal receives voice input from the user through a microphone and converts it into digital data using voice recognition technology. This digital data is then sent to a server via the Internet.

[0043] Server Features

[0044] The server is a central processing unit that receives and analyzes the voice data sent from the device. The server analyzes the data using machine learning algorithms and natural language processing (NLP) technology to detect changes in physical condition and lifestyle habits.

[0045] For example, if a user says, "I've had a stomachache recently," the server converts this voice data into text and extracts the keyword "stomachache." Based on this information, the server then assesses the user's health risk and generates an appropriate notification. The generated notification is then sent back to the device, where it is displayed to the user both audibly and on the screen.

[0046] User Actions

[0047] Users operate the system by giving voice commands to the device. For example, if they say "Turn on the TV," the device receives this voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and sends it back to the device. The device then controls the TV based on that command.

[0048] In addition, if the user utters a keyword such as "help" in an emergency, the device detects this and immediately sends an emergency notification to the server, which then notifies pre-registered emergency contacts (family members or medical institutions).

[0049] Specific examples

[0050] 1. Daily use:

[0051] When a user says, "Tell me the weather today," the device converts the speech into text and requests weather information from the server. The server retrieves the information from the weather database and notifies the user by voice and on the screen, saying, "It's sunny today."

[0052] 2. Health management:

[0053] When a user says, "I've been getting tired easily lately," the server analyzes this information, and if it determines there is an abnormality, it notifies the user, "You seem tired, so make sure you get plenty of rest."

[0054] 3. Emergency Response:

[0055] When a user shouts "help," the device detects this as an emergency keyword and immediately sends a notification to the server, which then automatically notifies the linked emergency contacts.

[0056] As described above, the system of the present invention is designed to be easy for the elderly to use, and aims to improve the quality of life and ensure safety for the elderly by providing a wide range of support, from assistance with daily life to responding to emergencies.

[0057] The processing flow will be explained below.

[0058] Daily use

[0059] When saying "What's the weather today?"

[0060] Step 1:

[0061] The user utters "What's the weather like today?"

[0062] Step 2:

[0063] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[0064] Step 3:

[0065] The terminal transmits the converted text data "Tell me the weather today" to the server.

[0066] Step 4:

[0067] The server analyzes the received text data and sends a request to retrieve weather information to an internal weather database or an external API.

[0068] Step 5:

[0069] The server sends the acquired weather information (e.g., sunny) to the terminal in text format.

[0070] Step 6:

[0071] The device converts the received text data into speech using a speech synthesis engine, and notifies the user by voice, saying, "It's sunny today." The device also displays the weather information on the screen.

[0072] health care

[0073] When saying "I get tired easily these days"

[0074] Step 1:

[0075] The user vocally says, "I've been feeling tired a lot recently."

[0076] Step 2:

[0077] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[0078] Step 3:

[0079] The terminal transmits the converted text data "I've been feeling tired lately" to the server.

[0080] Step 4:

[0081] The server analyzes the received text data and extracts the keyword "easily tired."

[0082] Step 5:

[0083] The server evaluates the user's health condition based on these keywords and generates a warning message, such as "You appear tired, so please try to get plenty of rest."

[0084] Step 6:

[0085] The server sends the generated warning message to the terminal.

[0086] Step 7:

[0087] The device converts the received warning message into speech using a speech synthesis engine, and notifies the user by voice, saying, "You seem tired, so please make sure to get plenty of rest." The device also displays a message on the screen.

[0088] Emergency response

[0089] Saying "Help me"

[0090] Step 1:

[0091] The user utters the words "help me."

[0092] Step 2:

[0093] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[0094] Step 3:

[0095] The device analyzes the converted text data "help" and detects it as an emergency keyword.

[0096] Step 4:

[0097] The device sends an emergency notification to the server, which also includes the user's location information.

[0098] Step 5:

[0099] Based on the received emergency notification, the server automatically notifies pre-registered emergency contacts (e.g., family members or medical institutions).

[0100] Step 6:

[0101] The emergency contact receives the notification from the server and initiates appropriate action.

[0102] As described above, through specific processing at each step, the system of the present invention can support the lives of elderly people and respond quickly in emergencies.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] To effectively support the daily lives of the elderly, it is necessary to provide systems that are easy for users to operate. In particular, it is necessary to realize a system that uses voice recognition technology to support the daily lives of the elderly and can respond quickly in emergencies.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes a means for receiving voice input from the user, a means for converting the received voice data into digital data, a means for transmitting the converted voice data to the server, and a means for displaying the notification transmitted from the server by voice and on a screen. This makes it easy for elderly people to operate and enables a wide range of applications from assistance with daily life to emergency response.

[0108] "Users" refer to elderly people who use the system.

[0109] "Voice input" refers to the voice that the user speaks through a microphone, and includes instructions and requests for information to the system.

[0110] "Voice Data" means data that has been converted into digital form from a User's voice.

[0111] A "server" is a central processing unit that analyzes received voice data and generates responses or notifications.

[0112] A "notification" is an information message generated by a server and sent to a terminal, and is conveyed to the user by sound or on-screen display.

[0113] "Digital data" refers to data that has been converted from analog audio signals into a format that can be processed by a computer.

[0114] "Speech recognition technology" refers to the technology that converts voice into text data.

[0115] A "machine learning algorithm" is a technology that learns patterns from large amounts of data and makes predictions and classifications.

[0116] "Natural language processing (NLP)" is the technology that enables computers to understand, interpret, and generate human language.

[0117] "Changes in physical condition and lifestyle" refers to changes in the user's health condition and behavioral patterns in their daily lives.

[0118] "Emergency keyword voice" refers to voices calling for help or specific words indicating an emergency.

[0119] An "emergency notification" is a notification generated by the server and sent to emergency contacts when an emergency keyword voice is detected.

[0120] "Emergency contacts" are pre-registered contacts such as family members or medical institutions to be contacted in an emergency.

[0121] A "device" is a device that a user directly controls, receives voice input, and displays notifications.

[0122] The system of this invention is a voice recognition IoT device designed to support the lives of the elderly. It is designed so that users (elderly people) can easily use the system. The system mainly consists of three components: a terminal, a server, and a user.

[0123] Device Features

[0124] Terminals are devices used by seniors in their daily lives. They have the following functions:

[0125] Receiving voice input: Receives voice input from the user through the microphone. For example, when a user says "Tell me the weather today," the device's microphone receives this voice.

[0126] Digital conversion and transmission of voice data: The received voice is converted into digital data and sent to a server. Specifically, the voice is converted into text using a voice recognition engine (e.g., Google® Speech-to-Text), and the converted data is sent to a server via the Internet.

[0127] Server Features

[0128] The server is a central processing unit that receives and analyzes the voice data sent from the terminal. It performs the following processes:

[0129] Voice data analysis: Using machine learning algorithms and natural language processing (NLP) techniques, voice data is converted into text and intent and keywords are extracted. For example, if a user says, "Tell me the weather today," the server will extract "weather" as a keyword and query a weather database to obtain weather information.

[0130] Health assessment: Analyzes changes in the user's physical condition and lifestyle to assess risk. For example, if a user says, "I've been feeling tired lately," the server will detect the keyword "fatigue" and generate a notification saying, "You appear tired, so please try to get plenty of rest."

[0131] Creating notifications: Based on the analysis results, appropriate notifications are created and sent back to the device.

[0132] User Actions

[0133] The user operates the system by issuing voice commands to the terminal. For example, if the user says "Turn on the TV," the terminal receives this voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and returns a command to the terminal. The terminal then operates the TV based on that command. A specific example of how this works is a sequence in which a voice recognition engine analyzes the command and operates an infrared remote control.

[0134] In addition, in the event of an emergency, if the user utters "help," the device will detect this as an emergency keyword and immediately send a notification to the server. The server will then automatically notify pre-set emergency contacts (family members or medical institutions). For example, if you say "help," the server will receive that information and send an emergency notification via message or call to the registered emergency contacts.

[0135] Specific examples

[0136] 1. Daily use:

[0137] The user says, "What's the weather like today?"

[0138] The terminal converts the voice into digital data and sends it to the server.

[0139] The server retrieves information such as "It's sunny today" from a weather database and sends a notification to the device.

[0140] The device will notify the user by voice and on the screen that "It's sunny today."

[0141] 2. Health management:

[0142] The user says, "I've been feeling tired a lot lately."

[0143] The server analyzes the voice and detects the keyword "fatigue."

[0144] The server generates a notification saying, "You appear tired, so please make sure to get plenty of rest," and sends it to the device.

[0145] The device will notify the user.

[0146] 3. Emergency Response:

[0147] The user says "help."

[0148] The device detects the emergency keyword and immediately sends a notification to the server.

[0149] The server will then notify pre-defined emergency contacts.

[0150] In this way, the system of the present invention is designed to be easy for the elderly to operate, and by covering a wide range of functions from assistance with daily life to responding to emergencies, it is expected to improve the quality of life of the elderly.

[0151] Prompt Sentence Examples

[0152] "Design a voice-activated IoT device to assist the elderly in their daily lives. Explain its specific functions, including the data flow between the device and the server, and provide examples."

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Step 1:

[0155] The user inputs voice

[0156] The user speaks instructions and questions into the device.

[0157] Specific action: The user speaks to the device, "Tell me what the weather is like today."

[0158] Input: User's voice

[0159] Output: Audio data (analog signal)

[0160] Step 2:

[0161] The device converts the audio into digital data

[0162] The terminal uses a speech recognition engine to convert the received voice into digital data.

[0163] Specific operation: The device's microphone picks up an analog voice signal and inputs this signal into a voice recognition engine (e.g., Google Speech-to-Text).

[0164] Input: Analog audio signal

[0165] Output: Text data

[0166] Step 3:

[0167] The device sends text data to the server.

[0168] The terminal transmits the converted text data to a server via the Internet.

[0169] Specific operation: The text data generated by the device is sent to the specified API endpoint on the server.

[0170] Input: Text data

[0171] Output: Network transmission

[0172] Step 4:

[0173] The server parses the text data

[0174] The server analyzes the received text data using machine learning algorithms and natural language processing techniques.

[0175] Specific operation: After the server receives the text data, it passes the data to an NLP module (e.g., SpaCy or BERT) to extract intent and keywords.

[0176] Input: Text data

[0177] Output: Analysis results (keywords and intent)

[0178] Step 5:

[0179] The server obtains the necessary information

[0180] Based on the analysis results, the server retrieves the necessary information from external databases and APIs.

[0181] Specific operation: For example, if the analysis result contains the keyword "weather," the server will query the weather database to obtain the latest weather information.

[0182] Input: Analysis results (keywords and intent)

[0183] Output: Information from an external database

[0184] Step 6:

[0185] The server generates the notification

[0186] Based on the information obtained, the server generates the appropriate notifications.

[0187] Specific operation: Create a notification message saying "It's sunny today" based on weather information.

[0188] Input: Information retrieved from an external database

[0189] Output: Notification message

[0190] Step 7:

[0191] The server sends a notification to the device

[0192] The generated notification is sent from the server to the terminal.

[0193] Specific operation: The server sends a notification message to the terminal via the network.

[0194] Input: Notification message

[0195] Output: Network transmission

[0196] Step 8:

[0197] Your device will notify you with audio and visual notifications

[0198] The device will present the received notification to the user via audio and visual notification.

[0199] Specific operation: The device converts the notification message into voice using a speech synthesis engine (e.g., Google Text-to-Speech) and plays it over the speaker. The notification message is also displayed on the screen.

[0200] Input: Notification message

[0201] Output: Audio notification and screen display

[0202] The above are the main processing steps of this system. The specific operations and data processing performed at each step have been explained in detail.

[0203] (Application example 1)

[0204] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0205] In factory work sites, many workers rely on manual and visual confirmation to execute complex instructions, leaving room for improvement in terms of efficiency and safety. Rapid response in emergencies is also required, but due to the nature of the work, it is often difficult to immediately detect and respond to emergencies. Therefore, there is a need for a system that improves the efficiency of factory work, ensures safety, and enables rapid response in emergencies.

[0206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0207] In this invention, the server includes means for receiving voice input from a user, means for transmitting the received voice data to the server, means for displaying the notification transmitted from the server by voice and on a screen, means for controlling the robot based on the received voice data, and means for detecting the voice data from the user in an emergency and generating and transmitting a corresponding notification, thereby improving work efficiency in the factory and enabling a rapid response in an emergency.

[0208] A "user" is a human user who operates a system.

[0209] "Voice input" refers to instructions or information provided by a user to a system through voice.

[0210] A "server" is a central processing unit that receives and analyzes the voice data and generates appropriate notifications and control signals.

[0211] "Notifications" are messages generated by the server to inform users of analysis results or important information.

[0212] A "robot" is a mechanical device that performs physical tasks based on instructions received from an automated control system connected to the system.

[0213] "Emergency" refers to a series of conditions or events that cannot be handled through normal operations or tasks and require immediate action.

[0214] "Keyword voice" is a specific word or phrase that is preset to indicate an emergency or a specific command.

[0215] "Analysis" is the process by which the server analyzes the received voice data, interprets the meaning, and determines the appropriate action.

[0216] "Control" refers to the system's ability to operate robots and other devices based on voice commands.

[0217] An "emergency notification" is a special message generated by the server to notify you of an emergency problem.

[0218] This invention is a system that uses a voice recognition IoT device to improve efficiency and safety at factory work sites. The detailed configuration of this system and its operating principle will be described below.

[0219] System configuration

[0220] The system consists of three main components:

[0221] 1. Terminal

[0222] 2. Server

[0223] 3. Users

[0224] Device Features

[0225] The device receives voice input from the user and converts the voice into digital data, which is then sent to a server via the Internet. It also receives notifications sent from the server and notifies the user via voice and on a screen.

[0226] Server Features

[0227] The server receives and analyzes the voice data sent from the device. This analysis uses machine learning algorithms and natural language processing (NLP) technology. The server analyzes the voice instructions, extracts appropriate commands, and generates work instructions and emergency response notifications. These notifications are then sent back to the device and displayed to the user via voice and on the screen.

[0228] Robot Control

[0229] The robot performs specific tasks based on instructions from the server. This control includes linking with IoT devices, and the robot operates according to received voice instructions. This allows users to give voice instructions to the robot and efficiently carry out tasks.

[0230] User Actions

[0231] Users can give voice commands to the device. For example, if they say "start the machine," the voice is converted into digital data on the device, which is then analyzed by the server and sent to the robot. Also, if an emergency keyword such as "help" is spoken, the system will immediately initiate an emergency response and notify emergency contacts.

[0232] Hardware and software used

[0233] The following hardware and software are used to realize this system.

[0234] Microphone and speaker: A high-sensitivity microphone receives audio input and a speaker provides audio output.

[0235] Robot control device: Uses control hardware for IOT devices.

[0236] Natural Language Processing and Speech Recognition Module: Python module 'speech_recognition' is used to convert voice data to text, and 'text_to_speech' is used to output speech. In addition, 'iot_device_control' module is used to control IOT devices.

[0237] Specific examples

[0238] 1. Daily tasks:

[0239] "Start the machine": The terminal sends the voice to the server, which analyzes it and sends instructions to the robot to start working.

[0240] "Stop the machine": Similarly, the voice command from the terminal is analyzed by the server and a command to stop is sent to the robot.

[0241] 2. Emergency Response:

[0242] "Help me": This voice command is detected by the device and sent to the server, which then generates an emergency notification and sends it to your emergency contacts.

[0243] Prompt Sentence Examples

[0244] Create a voice assistant application for a factory robot that will assist me. Features include:

[0245] 1. Receive user instructions through voice recognition.

[0246] 2. Operate IOT devices based on instructions (start, stop).

[0247] 3. Send emergency notifications with the command "Help" in case of an emergency.

[0248] This system can significantly improve work efficiency and safety within the factory.

[0249] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0250] Step 1:

[0251] Audio reception and conversion

[0252] The device receives voice input from the user. This voice data (input) is received by the microphone and sent to the speech recognition module. This module converts the voice data into text and generates digital data (output). Specifically, the 'speech_recognition' module is used to analyze the voice and convert it into a string of characters.

[0253] Step 2:

[0254] Data transmission

[0255] The device sends the text data generated in step 1 to the server. This data transmission (input) is performed via the Internet. Specifically, an HTTP request is used to send the data to the server's API.

[0256] Step 3:

[0257] Data analysis

[0258] The server analyzes the received text data (input) and understands the user's instructions. This analysis uses machine learning algorithms and natural language processing (NLP). The analysis results (output) are generated as instruction commands. Specifically, the server uses an NLP module to extract keywords and determine the appropriate action.

[0259] Step 4:

[0260] Notification Generation and Delivery

[0261] The server generates an appropriate notification based on the analysis results of step 3 and sends it to the device. This notification (input) is generated as data for voice and screen display. Specific operations include using the 'text_to_speech' module, which converts text data into voice data.

[0262] Step 5:

[0263] Robot Control

[0264] Based on the analysis results of step 3, the server generates a signal to control the IOT device and sends it to the robot. This control signal (input) is sent using a protocol for controlling IOT devices. Specifically, the 'iot_device_control' module generates a command to operate the robot and sends it to the device.

[0265] Step 6:

[0266] Receiving and displaying notifications

[0267] The device receives notifications sent from the server and notifies the user via voice and on the screen. This notification (input) includes audio data played on the device and text data displayed on the screen. Specifically, the 'text_to_speech' module generates audio and renders data for display on the screen.

[0268] Step 7:

[0269] Emergency response

[0270] When a user utters an emergency keyword, the device detects it and immediately sends an emergency notification to the server. This emergency notification (input) is analyzed by the server and a notification (output) is generated to be sent to emergency contacts. Specifically, when a specific keyword is detected, a function to send an email or SMS to emergency contacts is triggered.

[0271] In this way, each processing step begins with the user's voice input, and ultimately leads to robot control and emergency response, aiming to improve work efficiency and safety within the factory.

[0272] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0273] The system of the present invention provides a voice recognition IoT device combined with an emotion engine to more actively support the lives of the elderly. The system is designed to be easy for users (elderly people) to use and consists of three components: a terminal, a server, and a user. Each component and its function are explained in detail below.

[0274] Device Features

[0275] The terminal is a device used by elderly people in their daily lives, and is mainly used to receive voice input, send voice data, and receive and display notifications. The terminal receives voice input from the user through a microphone and converts it into digital data using a voice recognition engine and an emotion engine, while also identifying the user's emotions. This digital data and emotion data are sent to a server via the Internet.

[0276] Server Features

[0277] The server is a central processing unit that receives and analyzes the voice and emotion data sent from the device. The server uses machine learning algorithms and natural language processing (NLP) techniques to analyze the data and detect changes in physical condition and lifestyle habits. Furthermore, by analyzing the emotion data, the server can assess the user's stress and anxiety levels.

[0278] For example, if a user says, "I've been feeling tired lately," but speaks in an anxious tone, the server analyzes the user's emotions along with the voice data and generates an appropriate notification. In this case, the server generates a notification that advises the user to relax and, if necessary, instructs the user to consult a medical professional. The generated notification is then sent back to the device and notified to the user by voice and on-screen display.

[0279] User Actions

[0280] Users operate the system by giving voice commands to the device. For example, if they say "Turn on the TV," the device receives this voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and sends it back to the device. The device then controls the TV based on that command.

[0281] In addition, if the user utters a keyword such as "help" in an emergency, the device detects this and immediately sends an emergency notification to the server, which then notifies pre-registered emergency contacts (family members or medical institutions).

[0282] Specific examples

[0283] 1. Daily use:

[0284] When a user says, "Tell me the weather today," the device converts the voice into text and sends it to the server. The server obtains the weather information and sends it to the device. The device then announces, "It's sunny today," by voice and also displays it on the screen.

[0285] 2. Health management:

[0286] If a user says, "I get tired easily recently," but their voice is unsteady, the server will recognize the keywords "tired easily" and "anxiety" and generate a notification offering advice on how to relax. The device will then notify the user by voice and on-screen display.

[0287] 3. Emergency Response:

[0288] When a user shouts "help," the device detects this as an emergency keyword and sends an emergency notification to the server, which then notifies emergency contacts.

[0289] 4. Emotional Continuity Management:

[0290] The server stores emotional data extracted from daily conversations and periodically analyzes the user's emotional patterns. Based on the results of this analysis, the system provides individually customized relaxation suggestions and health-related advice during times of high stress.

[0291] As described above, the system of the present invention uses an emotion engine to support the lives of the elderly in a more personalized way, enabling early detection of health risks and rapid response in emergencies.

[0292] The processing flow will be explained below.

[0293] health care

[0294] When you say "I've been feeling tired lately" and it's accompanied by emotion recognition

[0295] Step 1:

[0296] The user vocally says, "I've been feeling tired a lot recently."

[0297] Step 2:

[0298] The device receives the user's voice through a microphone and converts it into text data using a voice recognition engine, while also analyzing the tone and pitch of the user's voice using an emotion engine to generate emotion data.

[0299] Step 3:

[0300] The device sends the converted text data "I've been feeling tired lately" and the analyzed emotional data (e.g., anxiety) to the server.

[0301] Step 4:

[0302] The server analyzes the received text data, extracts health-related keywords such as "getting tired easily," and analyzes the emotion data to determine that the user is feeling anxious.

[0303] Step 5:

[0304] Based on this information, the server will perform a risk assessment of the user's health status and emotional data, and generate an appropriate notification message, such as "You seem to be feeling tired and anxious recently. Please get plenty of rest and consult a medical professional if necessary."

[0305] Step 6:

[0306] The server sends the generated notification message to the terminal.

[0307] Step 7:

[0308] The device converts the received notification message into speech using a speech synthesis engine, and notifies the user aloud, saying, "You seem to have become tired and anxious recently. Please get plenty of rest and consult a medical professional if necessary." The same message is also displayed on the screen.

[0309] Emergency response

[0310] When you say "Help me"

[0311] Step 1:

[0312] The user utters the words "help me."

[0313] Step 2:

[0314] The device receives the user's voice through a microphone, converts it into text data using a speech recognition engine, and analyzes the urgency of the user's voice using an emotion engine.

[0315] Step 3:

[0316] The device sends an emergency notification to the server, including the converted text data "Help me" and the analyzed emotion data (e.g., extreme stress).

[0317] Step 4:

[0318] The server analyzes the received emergency notification and detects the emergency keyword "help."

[0319] Step 5:

[0320] The server will promptly notify pre-registered emergency contacts (e.g., family members or medical institutions), including the user's location information.

[0321] Step 6:

[0322] The emergency contact receives the notification from the server and initiates appropriate action.

[0323] Daily use

[0324] When saying "What's the weather today?"

[0325] Step 1:

[0326] The user utters "What's the weather like today?"

[0327] Step 2:

[0328] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[0329] Step 3:

[0330] The terminal transmits the converted text data "Tell me the weather today" to the server.

[0331] Step 4:

[0332] The server analyzes the received text data and sends a request to retrieve weather information to an internal weather database or an external API.

[0333] Step 5:

[0334] The server sends the acquired weather information (e.g., sunny) to the terminal in text format.

[0335] Step 6:

[0336] The device converts the received weather information into voice using a voice synthesis engine, notifies the user by voice, such as "It's sunny today," and also displays the weather information on the screen.

[0337] Emotional Continuity Management

[0338] Analysis of daily conversation data

[0339] Step 1:

[0340] The terminal receives voice input from the user on a daily basis, converts it into text data using a voice recognition engine, and simultaneously generates emotion data using an emotion engine.

[0341] Step 2:

[0342] The terminal transmits the converted text data and emotion data to the server.

[0343] Step 3:

[0344] The server stores the voice data and emotion data accumulated daily in a database.

[0345] Step 4:

[0346] The server periodically analyzes the accumulated data and evaluates the user's emotional patterns.

[0347] Step 5:

[0348] Based on the analysis results, the server generates notification messages to provide personalized health support to the user. For example, if the user's long-term stress level is high, the server generates a message recommending relaxation techniques or counseling.

[0349] Step 6:

[0350] The server sends the generated notification message to the terminal.

[0351] Step 7:

[0352] The device converts the received notification message into voice using a speech synthesis engine and notifies the user. The device also displays the same message on the screen.

[0353] By combining this system with an emotion engine, it will further personalize support for the elderly, enabling more effective health management and faster emergency response.

[0354] Example 2

[0355] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0356] In recent years, there has been a growing need for life support systems for the elderly, but conventional systems simply receive voice input and notify users, without taking into account their emotions or changes in their physical condition. This makes it difficult to provide personalized support to users, and there are problems with being unable to respond quickly and appropriately, especially in emergencies.

[0357] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0358] In this invention, the server includes means for analyzing the user's voice data and emotion data to detect changes in physical condition and lifestyle habits, means for generating a notification based on the detected changes, and means for transmitting the generated notification to the terminal. This enables personalized notifications that take into account the user's emotions and changes in physical condition, enabling a quick and appropriate response in an emergency.

[0359] "Voice input" means spoken information such as instructions or questions spoken by a user.

[0360] "Device" means any equipment or hardware used to receive, analyze, or transmit audio input.

[0361] "Text data" refers to character string data obtained by analyzing voice input.

[0362] "Emotional Data" means data used to identify a user's emotional state through voice analysis.

[0363] "Server" refers to a central processing unit that receives voice and emotion data, analyzes it, and generates notifications.

[0364] "Notification" means information or instructions provided to the user, whether through audio or visual display.

[0365] An "emergency" is defined as the moment a user reports a dangerous situation or health-related issue.

[0366] "Terminal" refers to a device that a user operates and uses to input voice information.

[0367] "Personalization" means providing information and assistance in a way that is adapted based on the user's individual situation and emotional state.

[0368] A "machine learning algorithm" refers to a computational method for learning from data and making predictions and analyses.

[0369] The system of the present invention provides an IoT device that combines an emotion engine and voice recognition technology to more actively support the lives of users (elderly people). This system is designed to be easy for users to use and consists of three components: a terminal, a server, and a user. The following describes in detail the modes for implementing the present invention.

[0370] Device Features

[0371] A device is a device that users use in their daily lives, and it mainly receives voice input, processes voice data, and receives and displays notifications. Specifically, a device has the following functions:

[0372] The user's voice is received by a microphone and converted into digital data using a voice recognition engine.

[0373] Analyze user emotions from voice using an emotion engine.

[0374] The converted voice data and emotion data are transmitted to a server.

[0375] Notify the user of notifications sent from the server via audio and visual.

[0376] Server Features

[0377] The server is a central processing unit that receives and analyzes the voice and emotion data sent from the device. Specifically, the server has the following functions:

[0378] The received data is analyzed using machine learning algorithms and natural language processing (NLP) to detect changes in the user's physical condition and lifestyle.

[0379] Evaluate users' stress and anxiety levels based on emotional data.

[0380] Based on the analysis results, an appropriate notification is generated and sent to the device.

[0381] User Actions

[0382] Users operate the system by issuing voice commands to the device. For example, if a command is given to "turn on the TV," the device receives the voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and sends it back to the device, which then operates the TV based on that command. Also, if a user utters "help" in an emergency, the device detects this and immediately sends an emergency notification to the server. The server receives this emergency notification and notifies pre-registered emergency contacts.

[0383] Specific use cases

[0384] Below are some specific examples of how this system can be used.

[0385] 1. Daily use

[0386] When a user says, "Tell me the weather today," the device converts the voice into text and sends it to the server. The server obtains the weather information and sends it to the device. The device then announces, "It's sunny today," by voice and also displays it on the screen.

[0387] 2. Health management

[0388] If a user says, "I get tired easily recently," but their voice is unsteady, the device recognizes this and sends it to the server. The server then recognizes the keywords "tired easily" and "anxiety" and generates a notification offering advice on how to relax. The device then notifies the user with the advice via voice and on the screen.

[0389] 3. Emergency Response

[0390] When a user shouts "help," the device detects this as an emergency keyword and sends an emergency notification to the server, which then notifies emergency contacts.

[0391] 4. Emotional Continuity Management

[0392] The server stores emotional data extracted from daily conversations and periodically analyzes the user's emotional patterns. Based on the results of this analysis, the system provides individually customized relaxation suggestions and health-related advice during times of high stress.

[0393] Example prompts for generative AI models

[0394] Below are some example prompts to input to the generative AI model:

[0395] Example prompt 1:

[0396] If a user says "I've been feeling tired lately" to a device by voice and is emotionally unstable, explain how the server generates appropriate advice and notifies the device.

[0397] Example prompt 2:

[0398] Explain how the device and server work together to notify emergency contacts when a user calls for help in an emergency.

[0399] As described above, the system of the present invention uses an emotion engine to support the lives of elderly people in a personalized way, enabling early detection of health risks and rapid response in emergencies.

[0400] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0401] Step 1:

[0402] The user provides voice input.

[0403] Specific operation: The user speaks to the device, saying, "Tell me the weather today."

[0404] Input: User's voice

[0405] Output: Audio signal

[0406] Step 2:

[0407] The device collects and processes the voice data.

[0408] Specific operation: The device uses a built-in microphone to collect voice signals, which are then converted into text data by a voice recognition engine. At the same time, the emotion engine analyzes the user's emotion data from the voice.

[0409] Input: Audio signal

[0410] Output: Text data, emotion data

[0411] Step 3:

[0412] The device transmits emotion data and voice data to the server.

[0413] Specific operation: The device assembles the generated text data "Tell me today's weather" and emotion data into a packet and sends it to a server via the Internet.

[0414] Input: Text data, emotion data

[0415] Output: Data packets

[0416] Step 4:

[0417] The server receives and processes the data.

[0418] How it works: The server decodes data packets received via the internet, extracts text and emotion data, and uses NLP techniques to analyze the user's intent and evaluates voice data to understand their emotions.

[0419] Input: Data packet

[0420] Output: User's intention, emotion evaluation results

[0421] Step 5:

[0422] The server generates the analysis results and generates the necessary notifications.

[0423] Specific operation: Based on the user's intent to "check the weather," the server retrieves the latest weather information and creates a notification saying "It's sunny today." If the emotion is "anxiety," the server also generates a notification about relaxation methods, if necessary.

[0424] Input: User's intention, emotion evaluation results

[0425] Output: Notification data

[0426] Step 6:

[0427] The server transmits the notification data to the terminal.

[0428] Specific operation: The server sends notification data "It's sunny today" to the terminal via the Internet.

[0429] Input: Notification data

[0430] Output: Data packets

[0431] Step 7:

[0432] The terminal provides a notification to the user.

[0433] Specific behavior: The device analyzes the received notification data, notifies the user by voice, saying "It's sunny today," and displays weather information on the screen. If additional notifications based on emotions are needed, they will also be displayed.

[0434] Input: Notification data

[0435] Output: Audio notification, screen display

[0436] Through the above processing steps, the system provides personalized responses to the user's needs and realizes advanced support based on changes in emotions and physical condition.

[0437] (Application example 2)

[0438] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0439] When elderly people use self-driving vehicles, there is a problem that there is insufficient support to grasp changes in their physical condition and emotions in real time and to ensure safe and comfortable use. In addition, simply recognizing voice commands tends to delay response in emergency situations involving elderly people. Therefore, it is necessary to establish a system that analyzes the emotional state of elderly drivers and provides appropriate advice and warnings.

[0440] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0441] In this invention, the server includes means for receiving voice input, means for analyzing the received voice data and emotion data, and means for generating appropriate notifications based on the analysis results, thereby improving safety and comfort for elderly drivers.

[0442] A "means for receiving audio input" is a device that captures the user's spoken voice and converts it into digital data.

[0443] The "means for transmitting voice data to a server" is a network communication function for sending the received voice data to a server via the Internet.

[0444] "Means for displaying notifications by voice and on screen" refers to a function that displays notifications sent from the server to the user by voice or on a display.

[0445] "Means for identifying emotional data" refers to a function that analyzes a user's emotional state from the user's voice or other input.

[0446] The "means for transmitting emotion data to a server" is a network communication function for transmitting the identified emotion data to a server.

[0447] The "means for generating advice or warning based on the analysis results" is a function for generating appropriate advice or warning based on the analysis results obtained by the server.

[0448] "Means for notifying advice and warnings" refers to a function that notifies the user of generated advice and warnings by voice or display.

[0449] "Means for detecting changes in physical condition and lifestyle habits" refers to a function that detects changes in the user's physical condition and lifestyle habits by analyzing voice data and emotional data.

[0450] "Means for detecting emergency keyword voice" is a function that recognizes emergency keywords uttered by the user in real time.

[0451] The "means for notifying the server of voice in an emergency" is a function that immediately transmits voice data detected in an emergency to the server.

[0452] "Means for generating and notifying appropriate measures in an emergency" is a function that creates appropriate measures in an emergency from the results of analysis on the server and notifies the user of them.

[0453] The system of the present invention is a system for assisting elderly drivers in an autonomous vehicle, and includes the following components: a voice recognition device, an emotion analysis engine, a server, and a user interface.

[0454] Hardware and Software Configuration

[0455] 1. Voice Recognition Device

[0456] A microphone is installed inside the vehicle to receive the user's voice input.

[0457] Convert speech to text data using a speech recognition engine (e.g., speech_recognition library).

[0458] 2. Sentiment Analysis Engine

[0459] Use an emotion analysis engine (e.g., emotion_recognition library) to identify emotions from user voice data.

[0460] 3. Server

[0461] Voice data and emotion data are received and analyzed.

[0462] Data is analyzed using machine learning algorithms and natural language processing (NLP) techniques.

[0463] It detects changes in the user's physical condition and lifestyle and generates appropriate advice and warnings.

[0464] 4. User Interface

[0465] The system has the function of notifying the user of the analysis results by voice and on-screen display, allowing the user to receive instructions in real time while in the vehicle.

[0466] Program processing

[0467] The server receives the voice data transmitted from the voice recognition device.

[0468] The emotion analysis engine identifies emotions from the voice data and sends them to the server.

[0469] The server analyzes the received voice and emotional data to detect changes in the user's physical condition and lifestyle.

[0470] Based on the detected changes, the server generates appropriate advice and warnings and notifies the user via the user interface.

[0471] Specific examples

[0472] For example, if a user says "I'm tired" while driving, the voice recognition device converts this speech into text data, and the emotion analysis engine identifies the emotional state "tired." The server analyzes this data and confirms that the user is tired. The server then generates advice such as "We recommend you stop driving and take a break," and notifies the user by voice and on-screen display.

[0473] Prompt Sentence Examples

[0474] You can use the following prompts to leverage specific generative AI models:

[0475] Build a voice recognition app for driving assistance for the elderly. The app recognizes voice input, analyzes the user's emotions, and provides appropriate advice and warnings while driving. For example, if a user says "I'm tired," the app will recognize it, analyze the emotion data, and notify the user, "We recommend that you stop driving and take a break." The libraries used are speech_recognition and emotion_recognition.

[0476] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0477] Step 1:

[0478] The device receives the user's voice input through a microphone, which is converted into digital data by a speech recognition engine.

[0479] Step 2:

[0480] The device sends the converted voice data to an emotion analysis engine, which identifies the user's emotion from the voice data. In this process, an emotion analysis model is used to analyze the voice tone and content to generate emotion data.

[0481] Step 3:

[0482] The terminal transmits the generated voice data and emotion data to the server, and the transmitted data includes the text content of the voice and the emotion data.

[0483] Step 4:

[0484] The server analyzes the received voice and emotion data using machine learning algorithms and natural language processing (NLP) techniques, allowing it to detect changes in the user's physical condition and lifestyle.

[0485] Step 5:

[0486] The server generates appropriate advice and warnings based on the analysis results. For example, if the user's stress level is high, it will generate a notification such as "We recommend that you stop driving and take a break."

[0487] Step 6:

[0488] The server sends the generated advice or warning to the terminal, and the notification includes the specific advice and the reason for it.

[0489] Step 7:

[0490] The device will notify the user of the notifications it receives through audio and visual displays, which are updated in real time to provide the user with information to take appropriate action.

[0491] Step 8:

[0492] Users can take appropriate action based on the notification, such as stopping driving and taking a break if they feel tired or stressed.

[0493] The above processing steps provide appropriate advice and warnings according to changes in the user's physical condition and emotions, thereby improving the safety and comfort of elderly drivers.

[0494] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0495] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0496] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0497] [Second embodiment]

[0498] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0499] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0500] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0501] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0502] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0503] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0504] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0505] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0506] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0507] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0508] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0509] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0510] The system of this invention is a voice recognition IoT device that supports the lives of elderly people and aims to make it easy for users (elderly people) to use. This system mainly consists of three components: a terminal, a server, and a user. Each component and its function will be explained in detail below.

[0511] Device Features

[0512] The terminal is a device used by the elderly in their daily lives, and is mainly used to receive voice input, send voice data, and receive and display notifications. The terminal receives voice input from the user through a microphone and converts it into digital data using voice recognition technology. This digital data is then sent to a server via the Internet.

[0513] Server Features

[0514] The server is a central processing unit that receives and analyzes the voice data sent from the device. The server analyzes the data using machine learning algorithms and natural language processing (NLP) technology to detect changes in physical condition and lifestyle habits.

[0515] For example, if a user says, "I've had a stomachache recently," the server converts this voice data into text and extracts the keyword "stomachache." Based on this information, the server then assesses the user's health risk and generates an appropriate notification. The generated notification is then sent back to the device, where it is displayed to the user both audibly and on the screen.

[0516] User Actions

[0517] Users operate the system by giving voice commands to the device. For example, if they say "Turn on the TV," the device receives this voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and sends it back to the device. The device then controls the TV based on that command.

[0518] In addition, if the user utters a keyword such as "help" in an emergency, the device detects this and immediately sends an emergency notification to the server, which then notifies pre-registered emergency contacts (family members or medical institutions).

[0519] Specific examples

[0520] 1. Daily use:

[0521] When a user says, "Tell me the weather today," the device converts the speech into text and requests weather information from the server. The server retrieves the information from the weather database and notifies the user by voice and on the screen, saying, "It's sunny today."

[0522] 2. Health management:

[0523] When a user says, "I've been getting tired easily lately," the server analyzes this information, and if it determines there is an abnormality, it notifies the user, "You seem tired, so make sure you get plenty of rest."

[0524] 3. Emergency Response:

[0525] When a user shouts "help," the device detects this as an emergency keyword and immediately sends a notification to the server, which then automatically notifies the linked emergency contacts.

[0526] As described above, the system of the present invention is designed to be easy for the elderly to use, and aims to improve the quality of life and ensure safety for the elderly by providing a wide range of support, from assistance with daily life to responding to emergencies.

[0527] The processing flow will be explained below.

[0528] Daily use

[0529] When saying "What's the weather today?"

[0530] Step 1:

[0531] The user utters "What's the weather like today?"

[0532] Step 2:

[0533] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[0534] Step 3:

[0535] The terminal transmits the converted text data "Tell me the weather today" to the server.

[0536] Step 4:

[0537] The server analyzes the received text data and sends a request to retrieve weather information to an internal weather database or an external API.

[0538] Step 5:

[0539] The server sends the acquired weather information (e.g., sunny) to the terminal in text format.

[0540] Step 6:

[0541] The device converts the received text data into speech using a speech synthesis engine, and notifies the user by voice, saying, "It's sunny today." The device also displays the weather information on the screen.

[0542] health care

[0543] When saying "I get tired easily these days"

[0544] Step 1:

[0545] The user vocally says, "I've been feeling tired a lot recently."

[0546] Step 2:

[0547] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[0548] Step 3:

[0549] The terminal transmits the converted text data "I've been feeling tired lately" to the server.

[0550] Step 4:

[0551] The server analyzes the received text data and extracts the keyword "easily tired."

[0552] Step 5:

[0553] The server evaluates the user's health condition based on these keywords and generates a warning message, such as "You appear tired, so please try to get plenty of rest."

[0554] Step 6:

[0555] The server sends the generated warning message to the terminal.

[0556] Step 7:

[0557] The device converts the received warning message into speech using a speech synthesis engine, and notifies the user by voice, saying, "You seem tired, so please make sure to get plenty of rest." The device also displays a message on the screen.

[0558] Emergency response

[0559] Saying "Help me"

[0560] Step 1:

[0561] The user utters the words "help me."

[0562] Step 2:

[0563] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[0564] Step 3:

[0565] The device analyzes the converted text data "help" and detects it as an emergency keyword.

[0566] Step 4:

[0567] The device sends an emergency notification to the server, which also includes the user's location information.

[0568] Step 5:

[0569] Based on the received emergency notification, the server automatically notifies pre-registered emergency contacts (e.g., family members or medical institutions).

[0570] Step 6:

[0571] The emergency contact receives the notification from the server and initiates appropriate action.

[0572] As described above, through specific processing at each step, the system of the present invention can support the lives of elderly people and respond quickly in emergencies.

[0573] Example 1

[0574] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0575] To effectively support the daily lives of the elderly, it is necessary to provide systems that are easy for users to operate. In particular, it is necessary to realize a system that uses voice recognition technology to support the daily lives of the elderly and can respond quickly in emergencies.

[0576] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0577] In this invention, the server includes a means for receiving voice input from the user, a means for converting the received voice data into digital data, a means for transmitting the converted voice data to the server, and a means for displaying the notification transmitted from the server by voice and on a screen. This makes it easy for elderly people to operate and enables a wide range of applications from assistance with daily life to emergency response.

[0578] "Users" refer to elderly people who use the system.

[0579] "Voice input" refers to the voice that the user speaks through a microphone, and includes instructions and requests for information to the system.

[0580] "Voice Data" means data that has been converted into digital form from a User's voice.

[0581] A "server" is a central processing unit that analyzes received voice data and generates responses or notifications.

[0582] A "notification" is an information message generated by a server and sent to a terminal, and is conveyed to the user by sound or on-screen display.

[0583] "Digital data" refers to data that has been converted from analog audio signals into a format that can be processed by a computer.

[0584] "Speech recognition technology" refers to the technology that converts voice into text data.

[0585] A "machine learning algorithm" is a technology that learns patterns from large amounts of data and makes predictions and classifications.

[0586] "Natural language processing (NLP)" is the technology that enables computers to understand, interpret, and generate human language.

[0587] "Changes in physical condition and lifestyle" refers to changes in the user's health condition and behavioral patterns in their daily lives.

[0588] "Emergency keyword voice" refers to voices calling for help or specific words indicating an emergency.

[0589] An "emergency notification" is a notification generated by the server and sent to emergency contacts when an emergency keyword voice is detected.

[0590] "Emergency contacts" are pre-registered contacts such as family members or medical institutions to be contacted in an emergency.

[0591] A "device" is a device that a user directly controls, receives voice input, and displays notifications.

[0592] The system of this invention is a voice recognition IoT device designed to support the lives of the elderly. It is designed so that users (elderly people) can easily use the system. The system mainly consists of three components: a terminal, a server, and a user.

[0593] Device Features

[0594] Terminals are devices used by seniors in their daily lives. They have the following functions:

[0595] Receiving voice input: Receives voice input from the user through the microphone. For example, when a user says "Tell me the weather today," the device's microphone receives this voice.

[0596] Digital conversion and transmission of voice data: The received voice is converted into digital data and sent to the server. Specifically, the voice is converted into text using a voice recognition engine (e.g., Google Speech-to-Text), and the converted data is sent to the server via the Internet.

[0597] Server Features

[0598] The server is a central processing unit that receives and analyzes the voice data sent from the terminal. It performs the following processes:

[0599] Voice data analysis: Using machine learning algorithms and natural language processing (NLP) techniques, voice data is converted into text and intent and keywords are extracted. For example, if a user says, "Tell me the weather today," the server will extract "weather" as a keyword and query a weather database to obtain weather information.

[0600] Health assessment: Analyzes changes in the user's physical condition and lifestyle to assess risk. For example, if a user says, "I've been feeling tired lately," the server will detect the keyword "fatigue" and generate a notification saying, "You appear tired, so please try to get plenty of rest."

[0601] Creating notifications: Based on the analysis results, appropriate notifications are created and sent back to the device.

[0602] User Actions

[0603] The user operates the system by issuing voice commands to the terminal. For example, if the user says "Turn on the TV," the terminal receives this voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and returns a command to the terminal. The terminal then operates the TV based on that command. A specific example of how this works is a sequence in which a voice recognition engine analyzes the command and operates an infrared remote control.

[0604] In addition, in the event of an emergency, if the user utters "help," the device will detect this as an emergency keyword and immediately send a notification to the server. The server will then automatically notify pre-set emergency contacts (family members or medical institutions). For example, if you say "help," the server will receive that information and send an emergency notification via message or call to the registered emergency contacts.

[0605] Specific examples

[0606] 1. Daily use:

[0607] The user says, "What's the weather like today?"

[0608] The terminal converts the voice into digital data and sends it to the server.

[0609] The server retrieves information such as "It's sunny today" from a weather database and sends a notification to the device.

[0610] The device will notify the user by voice and on the screen that "It's sunny today."

[0611] 2. Health management:

[0612] The user says, "I've been feeling tired a lot lately."

[0613] The server analyzes the voice and detects the keyword "fatigue."

[0614] The server generates a notification saying, "You appear tired, so please make sure to get plenty of rest," and sends it to the device.

[0615] The device will notify the user.

[0616] 3. Emergency Response:

[0617] The user says "help."

[0618] The device detects the emergency keyword and immediately sends a notification to the server.

[0619] The server will then notify pre-defined emergency contacts.

[0620] In this way, the system of the present invention is designed to be easy for the elderly to operate, and by covering a wide range of functions from assistance with daily life to responding to emergencies, it is expected to improve the quality of life of the elderly.

[0621] Prompt Sentence Examples

[0622] "Design a voice-activated IoT device to assist the elderly in their daily lives. Explain its specific functions, including the data flow between the device and the server, and provide examples."

[0623] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0624] Step 1:

[0625] The user inputs voice

[0626] The user speaks instructions and questions into the device.

[0627] Specific action: The user speaks to the device, "Tell me what the weather is like today."

[0628] Input: User's voice

[0629] Output: Audio data (analog signal)

[0630] Step 2:

[0631] The device converts the audio into digital data

[0632] The terminal uses a speech recognition engine to convert the received voice into digital data.

[0633] Specific operation: The device's microphone picks up an analog voice signal and inputs this signal into a voice recognition engine (e.g., Google Speech-to-Text).

[0634] Input: Analog audio signal

[0635] Output: Text data

[0636] Step 3:

[0637] The device sends text data to the server.

[0638] The terminal transmits the converted text data to a server via the Internet.

[0639] Specific operation: The text data generated by the device is sent to the specified API endpoint on the server.

[0640] Input: Text data

[0641] Output: Network transmission

[0642] Step 4:

[0643] The server parses the text data

[0644] The server analyzes the received text data using machine learning algorithms and natural language processing techniques.

[0645] Specific operation: After the server receives the text data, it passes the data to an NLP module (e.g., SpaCy or BERT) to extract intent and keywords.

[0646] Input: Text data

[0647] Output: Analysis results (keywords and intent)

[0648] Step 5:

[0649] The server obtains the necessary information

[0650] Based on the analysis results, the server retrieves the necessary information from external databases and APIs.

[0651] Specific operation: For example, if the analysis result contains the keyword "weather," the server will query the weather database to obtain the latest weather information.

[0652] Input: Analysis results (keywords and intent)

[0653] Output: Information from an external database

[0654] Step 6:

[0655] The server generates the notification

[0656] Based on the information obtained, the server generates the appropriate notifications.

[0657] Specific operation: Create a notification message saying "It's sunny today" based on weather information.

[0658] Input: Information retrieved from an external database

[0659] Output: Notification message

[0660] Step 7:

[0661] The server sends a notification to the device

[0662] The generated notification is sent from the server to the terminal.

[0663] Specific operation: The server sends a notification message to the terminal via the network.

[0664] Input: Notification message

[0665] Output: Network transmission

[0666] Step 8:

[0667] Your device will notify you with audio and visual notifications

[0668] The device will present the received notification to the user via audio and visual notification.

[0669] Specific operation: The device converts the notification message into voice using a speech synthesis engine (e.g., Google Text-to-Speech) and plays it over the speaker. The notification message is also displayed on the screen.

[0670] Input: Notification message

[0671] Output: Audio notification and screen display

[0672] The above are the main processing steps of this system. The specific operations and data processing performed at each step have been explained in detail.

[0673] (Application example 1)

[0674] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0675] In factory work sites, many workers rely on manual and visual confirmation to execute complex instructions, leaving room for improvement in terms of efficiency and safety. Rapid response in emergencies is also required, but due to the nature of the work, it is often difficult to immediately detect and respond to emergencies. Therefore, there is a need for a system that improves the efficiency of factory work, ensures safety, and enables rapid response in emergencies.

[0676] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0677] In this invention, the server includes means for receiving voice input from a user, means for transmitting the received voice data to the server, means for displaying the notification transmitted from the server by voice and on a screen, means for controlling the robot based on the received voice data, and means for detecting the voice data from the user in an emergency and generating and transmitting a corresponding notification, thereby improving work efficiency in the factory and enabling a rapid response in an emergency.

[0678] A "user" is a human user who operates a system.

[0679] "Voice input" refers to instructions or information provided by a user to a system through voice.

[0680] A "server" is a central processing unit that receives and analyzes the voice data and generates appropriate notifications and control signals.

[0681] "Notifications" are messages generated by the server to inform users of analysis results or important information.

[0682] A "robot" is a mechanical device that performs physical tasks based on instructions received from an automated control system connected to the system.

[0683] "Emergency" refers to a series of conditions or events that cannot be handled through normal operations or tasks and require immediate action.

[0684] "Keyword voice" is a specific word or phrase that is preset to indicate an emergency or a specific command.

[0685] "Analysis" is the process by which the server analyzes the received voice data, interprets the meaning, and determines the appropriate action.

[0686] "Control" refers to the system's ability to operate robots and other devices based on voice commands.

[0687] An "emergency notification" is a special message generated by the server to notify you of an emergency problem.

[0688] This invention is a system that uses a voice recognition IoT device to improve efficiency and safety at factory work sites. The detailed configuration of this system and its operating principle will be described below.

[0689] System configuration

[0690] The system consists of three main components:

[0691] 1. Terminal

[0692] 2. Server

[0693] 3. Users

[0694] Device Features

[0695] The device receives voice input from the user and converts the voice into digital data, which is then sent to a server via the Internet. It also receives notifications sent from the server and notifies the user via voice and on a screen.

[0696] Server Features

[0697] The server receives and analyzes the voice data sent from the device. This analysis uses machine learning algorithms and natural language processing (NLP) technology. The server analyzes the voice instructions, extracts appropriate commands, and generates work instructions and emergency response notifications. These notifications are then sent back to the device and displayed to the user via voice and on the screen.

[0698] Robot Control

[0699] The robot performs specific tasks based on instructions from the server. This control includes linking with IoT devices, and the robot operates according to received voice instructions. This allows users to give voice instructions to the robot and efficiently carry out tasks.

[0700] User Actions

[0701] Users can give voice commands to the device. For example, if they say "start the machine," the voice is converted into digital data on the device, which is then analyzed by the server and sent to the robot. Also, if an emergency keyword such as "help" is spoken, the system will immediately initiate an emergency response and notify emergency contacts.

[0702] Hardware and software used

[0703] The following hardware and software are used to realize this system.

[0704] Microphone and speaker: A high-sensitivity microphone receives audio input and a speaker provides audio output.

[0705] Robot control device: Uses control hardware for IOT devices.

[0706] Natural Language Processing and Speech Recognition Module: Python module 'speech_recognition' is used to convert voice data to text, and 'text_to_speech' is used to output speech. In addition, 'iot_device_control' module is used to control IOT devices.

[0707] Specific examples

[0708] 1. Daily tasks:

[0709] "Start the machine": The terminal sends the voice to the server, which analyzes it and sends instructions to the robot to start working.

[0710] "Stop the machine": Similarly, the voice command from the terminal is analyzed by the server and a command to stop is sent to the robot.

[0711] 2. Emergency Response:

[0712] "Help me": This voice command is detected by the device and sent to the server, which then generates an emergency notification and sends it to your emergency contacts.

[0713] Prompt Sentence Examples

[0714] Create a voice assistant application for a factory robot that will assist me. Features include:

[0715] 1. Receive user instructions through voice recognition.

[0716] 2. Operate IOT devices based on instructions (start, stop).

[0717] 3. Send emergency notifications with the command "Help" in case of an emergency.

[0718] This system can significantly improve work efficiency and safety within the factory.

[0719] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0720] Step 1:

[0721] Audio reception and conversion

[0722] The device receives voice input from the user. This voice data (input) is received by the microphone and sent to the speech recognition module. This module converts the voice data into text and generates digital data (output). Specifically, the 'speech_recognition' module is used to analyze the voice and convert it into a string of characters.

[0723] Step 2:

[0724] Data transmission

[0725] The device sends the text data generated in step 1 to the server. This data transmission (input) is performed via the Internet. Specifically, an HTTP request is used to send the data to the server's API.

[0726] Step 3:

[0727] Data analysis

[0728] The server analyzes the received text data (input) and understands the user's instructions. This analysis uses machine learning algorithms and natural language processing (NLP). The analysis results (output) are generated as instruction commands. Specifically, the server uses an NLP module to extract keywords and determine the appropriate action.

[0729] Step 4:

[0730] Notification Generation and Delivery

[0731] The server generates an appropriate notification based on the analysis results of step 3 and sends it to the device. This notification (input) is generated as data for voice and screen display. Specific operations include using the 'text_to_speech' module, which converts text data into voice data.

[0732] Step 5:

[0733] Robot Control

[0734] Based on the analysis results of step 3, the server generates a signal to control the IOT device and sends it to the robot. This control signal (input) is sent using a protocol for controlling IOT devices. Specifically, the 'iot_device_control' module generates a command to operate the robot and sends it to the device.

[0735] Step 6:

[0736] Receiving and displaying notifications

[0737] The device receives notifications sent from the server and notifies the user via voice and on the screen. This notification (input) includes audio data played on the device and text data displayed on the screen. Specifically, the 'text_to_speech' module generates audio and renders data for display on the screen.

[0738] Step 7:

[0739] Emergency response

[0740] When a user utters an emergency keyword, the device detects it and immediately sends an emergency notification to the server. This emergency notification (input) is analyzed by the server and a notification (output) is generated to be sent to emergency contacts. Specifically, when a specific keyword is detected, a function to send an email or SMS to emergency contacts is triggered.

[0741] In this way, each processing step begins with the user's voice input, and ultimately leads to robot control and emergency response, aiming to improve work efficiency and safety within the factory.

[0742] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0743] The system of the present invention provides a voice recognition IoT device combined with an emotion engine to more actively support the lives of the elderly. The system is designed to be easy for users (elderly people) to use and consists of three components: a terminal, a server, and a user. Each component and its function are explained in detail below.

[0744] Device Features

[0745] The terminal is a device used by elderly people in their daily lives, and is mainly used to receive voice input, send voice data, and receive and display notifications. The terminal receives voice input from the user through a microphone and converts it into digital data using a voice recognition engine and an emotion engine, while also identifying the user's emotions. This digital data and emotion data are sent to a server via the Internet.

[0746] Server Features

[0747] The server is a central processing unit that receives and analyzes the voice and emotion data sent from the device. The server uses machine learning algorithms and natural language processing (NLP) techniques to analyze the data and detect changes in physical condition and lifestyle habits. Furthermore, by analyzing the emotion data, the server can assess the user's stress and anxiety levels.

[0748] For example, if a user says, "I've been feeling tired lately," but speaks in an anxious tone, the server analyzes the user's emotions along with the voice data and generates an appropriate notification. In this case, the server generates a notification that advises the user to relax and, if necessary, instructs the user to consult a medical professional. The generated notification is then sent back to the device and notified to the user by voice and on-screen display.

[0749] User Actions

[0750] Users operate the system by giving voice commands to the device. For example, if they say "Turn on the TV," the device receives this voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and sends it back to the device. The device then controls the TV based on that command.

[0751] In addition, if the user utters a keyword such as "help" in an emergency, the device detects this and immediately sends an emergency notification to the server, which then notifies pre-registered emergency contacts (family members or medical institutions).

[0752] Specific examples

[0753] 1. Daily use:

[0754] When a user says, "Tell me the weather today," the device converts the voice into text and sends it to the server. The server obtains the weather information and sends it to the device. The device then announces, "It's sunny today," by voice and also displays it on the screen.

[0755] 2. Health management:

[0756] If a user says, "I get tired easily recently," but their voice is unsteady, the server will recognize the keywords "tired easily" and "anxiety" and generate a notification offering advice on how to relax. The device will then notify the user by voice and on-screen display.

[0757] 3. Emergency Response:

[0758] When a user shouts "help," the device detects this as an emergency keyword and sends an emergency notification to the server, which then notifies emergency contacts.

[0759] 4. Emotional Continuity Management:

[0760] The server stores emotional data extracted from daily conversations and periodically analyzes the user's emotional patterns. Based on the results of this analysis, the system provides individually customized relaxation suggestions and health-related advice during times of high stress.

[0761] As described above, the system of the present invention uses an emotion engine to support the lives of the elderly in a more personalized way, enabling early detection of health risks and rapid response in emergencies.

[0762] The processing flow will be explained below.

[0763] health care

[0764] When you say "I've been feeling tired lately" and it's accompanied by emotion recognition

[0765] Step 1:

[0766] The user vocally says, "I've been feeling tired a lot recently."

[0767] Step 2:

[0768] The device receives the user's voice through a microphone and converts it into text data using a voice recognition engine, while also analyzing the tone and pitch of the user's voice using an emotion engine to generate emotion data.

[0769] Step 3:

[0770] The device sends the converted text data "I've been feeling tired lately" and the analyzed emotional data (e.g., anxiety) to the server.

[0771] Step 4:

[0772] The server analyzes the received text data, extracts health-related keywords such as "getting tired easily," and analyzes the emotion data to determine that the user is feeling anxious.

[0773] Step 5:

[0774] Based on this information, the server will perform a risk assessment of the user's health status and emotional data, and generate an appropriate notification message, such as "You seem to be feeling tired and anxious recently. Please get plenty of rest and consult a medical professional if necessary."

[0775] Step 6:

[0776] The server sends the generated notification message to the terminal.

[0777] Step 7:

[0778] The device converts the received notification message into speech using a speech synthesis engine, and notifies the user aloud, saying, "You seem to have become tired and anxious recently. Please get plenty of rest and consult a medical professional if necessary." The same message is also displayed on the screen.

[0779] Emergency response

[0780] When you say "Help me"

[0781] Step 1:

[0782] The user utters the words "help me."

[0783] Step 2:

[0784] The device receives the user's voice through a microphone, converts it into text data using a speech recognition engine, and analyzes the urgency of the user's voice using an emotion engine.

[0785] Step 3:

[0786] The device sends an emergency notification to the server, including the converted text data "Help me" and the analyzed emotion data (e.g., extreme stress).

[0787] Step 4:

[0788] The server analyzes the received emergency notification and detects the emergency keyword "help."

[0789] Step 5:

[0790] The server will promptly notify pre-registered emergency contacts (e.g., family members or medical institutions), including the user's location information.

[0791] Step 6:

[0792] The emergency contact receives the notification from the server and initiates appropriate action.

[0793] Daily use

[0794] When saying "What's the weather today?"

[0795] Step 1:

[0796] The user utters "What's the weather like today?"

[0797] Step 2:

[0798] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[0799] Step 3:

[0800] The terminal transmits the converted text data "Tell me the weather today" to the server.

[0801] Step 4:

[0802] The server analyzes the received text data and sends a request to retrieve weather information to an internal weather database or an external API.

[0803] Step 5:

[0804] The server sends the acquired weather information (e.g., sunny) to the terminal in text format.

[0805] Step 6:

[0806] The device converts the received weather information into voice using a voice synthesis engine, notifies the user by voice, such as "It's sunny today," and also displays the weather information on the screen.

[0807] Emotional Continuity Management

[0808] Analysis of daily conversation data

[0809] Step 1:

[0810] The terminal receives voice input from the user on a daily basis, converts it into text data using a voice recognition engine, and simultaneously generates emotion data using an emotion engine.

[0811] Step 2:

[0812] The terminal transmits the converted text data and emotion data to the server.

[0813] Step 3:

[0814] The server stores the voice data and emotion data accumulated daily in a database.

[0815] Step 4:

[0816] The server periodically analyzes the accumulated data and evaluates the user's emotional patterns.

[0817] Step 5:

[0818] Based on the analysis results, the server generates notification messages to provide personalized health support to the user. For example, if the user's long-term stress level is high, the server generates a message recommending relaxation techniques or counseling.

[0819] Step 6:

[0820] The server sends the generated notification message to the terminal.

[0821] Step 7:

[0822] The device converts the received notification message into voice using a speech synthesis engine and notifies the user. The device also displays the same message on the screen.

[0823] By combining this system with an emotion engine, it will further personalize support for the elderly, enabling more effective health management and faster emergency response.

[0824] Example 2

[0825] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0826] In recent years, there has been a growing need for life support systems for the elderly, but conventional systems simply receive voice input and notify users, without taking into account their emotions or changes in their physical condition. This makes it difficult to provide personalized support to users, and there are problems with being unable to respond quickly and appropriately, especially in emergencies.

[0827] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0828] In this invention, the server includes means for analyzing the user's voice data and emotion data to detect changes in physical condition and lifestyle habits, means for generating a notification based on the detected changes, and means for transmitting the generated notification to the terminal. This enables personalized notifications that take into account the user's emotions and changes in physical condition, enabling a quick and appropriate response in an emergency.

[0829] "Voice input" means spoken information such as instructions or questions spoken by a user.

[0830] "Device" means any equipment or hardware used to receive, analyze, or transmit audio input.

[0831] "Text data" refers to character string data obtained by analyzing voice input.

[0832] "Emotional Data" means data used to identify a user's emotional state through voice analysis.

[0833] "Server" refers to a central processing unit that receives voice and emotion data, analyzes it, and generates notifications.

[0834] "Notification" means information or instructions provided to the user, whether through audio or visual display.

[0835] An "emergency" is defined as the moment a user reports a dangerous situation or health-related issue.

[0836] "Terminal" refers to a device that a user operates and uses to input voice information.

[0837] "Personalization" means providing information and assistance in a way that is adapted based on the user's individual situation and emotional state.

[0838] A "machine learning algorithm" refers to a computational method for learning from data and making predictions and analyses.

[0839] The system of the present invention provides an IoT device that combines an emotion engine and voice recognition technology to more actively support the lives of users (elderly people). This system is designed to be easy for users to use and consists of three components: a terminal, a server, and a user. The following describes in detail the modes for implementing the present invention.

[0840] Device Features

[0841] A device is a device that users use in their daily lives, and it mainly receives voice input, processes voice data, and receives and displays notifications. Specifically, a device has the following functions:

[0842] The user's voice is received by a microphone and converted into digital data using a voice recognition engine.

[0843] Analyze user emotions from voice using an emotion engine.

[0844] The converted voice data and emotion data are transmitted to a server.

[0845] Notify the user of notifications sent from the server via audio and visual.

[0846] Server Features

[0847] The server is a central processing unit that receives and analyzes the voice and emotion data sent from the device. Specifically, the server has the following functions:

[0848] The received data is analyzed using machine learning algorithms and natural language processing (NLP) to detect changes in the user's physical condition and lifestyle.

[0849] Evaluate users' stress and anxiety levels based on emotional data.

[0850] Based on the analysis results, an appropriate notification is generated and sent to the device.

[0851] User Actions

[0852] Users operate the system by issuing voice commands to the device. For example, if a command is given to "turn on the TV," the device receives the voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and sends it back to the device, which then operates the TV based on that command. Also, if a user utters "help" in an emergency, the device detects this and immediately sends an emergency notification to the server. The server receives this emergency notification and notifies pre-registered emergency contacts.

[0853] Specific use cases

[0854] Below are some specific examples of how this system can be used.

[0855] 1. Daily use

[0856] When a user says, "Tell me the weather today," the device converts the voice into text and sends it to the server. The server obtains the weather information and sends it to the device. The device then announces, "It's sunny today," by voice and also displays it on the screen.

[0857] 2. Health management

[0858] If a user says, "I get tired easily recently," but their voice is unsteady, the device recognizes this and sends it to the server. The server then recognizes the keywords "tired easily" and "anxiety" and generates a notification offering advice on how to relax. The device then notifies the user with the advice via voice and on the screen.

[0859] 3. Emergency Response

[0860] When a user shouts "help," the device detects this as an emergency keyword and sends an emergency notification to the server, which then notifies emergency contacts.

[0861] 4. Emotional Continuity Management

[0862] The server stores emotional data extracted from daily conversations and periodically analyzes the user's emotional patterns. Based on the results of this analysis, the system provides individually customized relaxation suggestions and health-related advice during times of high stress.

[0863] Example prompts for generative AI models

[0864] Below are some example prompts to input to the generative AI model:

[0865] Example prompt 1:

[0866] If a user says "I've been feeling tired lately" to a device by voice and is emotionally unstable, explain how the server generates appropriate advice and notifies the device.

[0867] Example prompt 2:

[0868] Explain how the device and server work together to notify emergency contacts when a user calls for help in an emergency.

[0869] As described above, the system of the present invention uses an emotion engine to support the lives of elderly people in a personalized way, enabling early detection of health risks and rapid response in emergencies.

[0870] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0871] Step 1:

[0872] The user provides voice input.

[0873] Specific operation: The user speaks to the device, saying, "Tell me the weather today."

[0874] Input: User's voice

[0875] Output: Audio signal

[0876] Step 2:

[0877] The device collects and processes the voice data.

[0878] Specific operation: The device uses a built-in microphone to collect voice signals, which are then converted into text data by a voice recognition engine. At the same time, the emotion engine analyzes the user's emotion data from the voice.

[0879] Input: Audio signal

[0880] Output: Text data, emotion data

[0881] Step 3:

[0882] The device transmits emotion data and voice data to the server.

[0883] Specific operation: The device assembles the generated text data "Tell me today's weather" and emotion data into a packet and sends it to a server via the Internet.

[0884] Input: Text data, emotion data

[0885] Output: Data packets

[0886] Step 4:

[0887] The server receives and processes the data.

[0888] How it works: The server decodes data packets received via the internet, extracts text and emotion data, and uses NLP techniques to analyze the user's intent and evaluates voice data to understand their emotions.

[0889] Input: Data packet

[0890] Output: User's intention, emotion evaluation results

[0891] Step 5:

[0892] The server generates the analysis results and generates the necessary notifications.

[0893] Specific operation: Based on the user's intent to "check the weather," the server retrieves the latest weather information and creates a notification saying "It's sunny today." If the emotion is "anxiety," the server also generates a notification about relaxation methods, if necessary.

[0894] Input: User's intention, emotion evaluation results

[0895] Output: Notification data

[0896] Step 6:

[0897] The server transmits the notification data to the terminal.

[0898] Specific operation: The server sends notification data "It's sunny today" to the terminal via the Internet.

[0899] Input: Notification data

[0900] Output: Data packets

[0901] Step 7:

[0902] The terminal provides a notification to the user.

[0903] Specific behavior: The device analyzes the received notification data, notifies the user by voice, saying "It's sunny today," and displays weather information on the screen. If additional notifications based on emotions are needed, they will also be displayed.

[0904] Input: Notification data

[0905] Output: Audio notification, screen display

[0906] Through the above processing steps, the system provides personalized responses to the user's needs and realizes advanced support based on changes in emotions and physical condition.

[0907] (Application example 2)

[0908] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0909] When elderly people use self-driving vehicles, there is a problem that there is insufficient support to grasp changes in their physical condition and emotions in real time and to ensure safe and comfortable use. In addition, simply recognizing voice commands tends to delay response in emergency situations involving elderly people. Therefore, it is necessary to establish a system that analyzes the emotional state of elderly drivers and provides appropriate advice and warnings.

[0910] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0911] In this invention, the server includes means for receiving voice input, means for analyzing the received voice data and emotion data, and means for generating appropriate notifications based on the analysis results, thereby improving safety and comfort for elderly drivers.

[0912] A "means for receiving audio input" is a device that captures the user's spoken voice and converts it into digital data.

[0913] The "means for transmitting voice data to a server" is a network communication function for sending the received voice data to a server via the Internet.

[0914] "Means for displaying notifications by voice and on screen" refers to a function that displays notifications sent from the server to the user by voice or on a display.

[0915] "Means for identifying emotional data" refers to a function that analyzes a user's emotional state from the user's voice or other input.

[0916] The "means for transmitting emotion data to a server" is a network communication function for transmitting the identified emotion data to a server.

[0917] The "means for generating advice or warning based on the analysis results" is a function for generating appropriate advice or warning based on the analysis results obtained by the server.

[0918] "Means for notifying advice and warnings" refers to a function that notifies the user of generated advice and warnings by voice or display.

[0919] "Means for detecting changes in physical condition and lifestyle habits" refers to a function that detects changes in the user's physical condition and lifestyle habits by analyzing voice data and emotional data.

[0920] "Means for detecting emergency keyword voice" is a function that recognizes emergency keywords uttered by the user in real time.

[0921] The "means for notifying the server of voice in an emergency" is a function that immediately transmits voice data detected in an emergency to the server.

[0922] "Means for generating and notifying appropriate measures in an emergency" is a function that creates appropriate measures in an emergency from the results of analysis on the server and notifies the user of them.

[0923] The system of the present invention is a system for assisting elderly drivers in an autonomous vehicle, and includes the following components: a voice recognition device, an emotion analysis engine, a server, and a user interface.

[0924] Hardware and Software Configuration

[0925] 1. Voice Recognition Device

[0926] A microphone is installed inside the vehicle to receive the user's voice input.

[0927] Convert speech to text data using a speech recognition engine (e.g., speech_recognition library).

[0928] 2. Sentiment Analysis Engine

[0929] Use an emotion analysis engine (e.g., emotion_recognition library) to identify emotions from user voice data.

[0930] 3. Server

[0931] Voice data and emotion data are received and analyzed.

[0932] Data is analyzed using machine learning algorithms and natural language processing (NLP) techniques.

[0933] It detects changes in the user's physical condition and lifestyle and generates appropriate advice and warnings.

[0934] 4. User Interface

[0935] The system has the function of notifying the user of the analysis results by voice and on-screen display, allowing the user to receive instructions in real time while in the vehicle.

[0936] Program processing

[0937] The server receives the voice data transmitted from the voice recognition device.

[0938] The emotion analysis engine identifies emotions from the voice data and sends them to the server.

[0939] The server analyzes the received voice and emotional data to detect changes in the user's physical condition and lifestyle.

[0940] Based on the detected changes, the server generates appropriate advice and warnings and notifies the user via the user interface.

[0941] Specific examples

[0942] For example, if a user says "I'm tired" while driving, the voice recognition device converts this speech into text data, and the emotion analysis engine identifies the emotional state "tired." The server analyzes this data and confirms that the user is tired. The server then generates advice such as "We recommend you stop driving and take a break," and notifies the user by voice and on-screen display.

[0943] Prompt Sentence Examples

[0944] You can use the following prompts to leverage specific generative AI models:

[0945] Build a voice recognition app for driving assistance for the elderly. The app recognizes voice input, analyzes the user's emotions, and provides appropriate advice and warnings while driving. For example, if a user says "I'm tired," the app will recognize it, analyze the emotion data, and notify the user, "We recommend that you stop driving and take a break." The libraries used are speech_recognition and emotion_recognition.

[0946] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0947] Step 1:

[0948] The device receives the user's voice input through a microphone, which is converted into digital data by a speech recognition engine.

[0949] Step 2:

[0950] The device sends the converted voice data to an emotion analysis engine, which identifies the user's emotion from the voice data. In this process, an emotion analysis model is used to analyze the voice tone and content to generate emotion data.

[0951] Step 3:

[0952] The terminal transmits the generated voice data and emotion data to the server, and the transmitted data includes the text content of the voice and the emotion data.

[0953] Step 4:

[0954] The server analyzes the received voice and emotion data using machine learning algorithms and natural language processing (NLP) techniques, allowing it to detect changes in the user's physical condition and lifestyle.

[0955] Step 5:

[0956] The server generates appropriate advice and warnings based on the analysis results. For example, if the user's stress level is high, it will generate a notification such as "We recommend that you stop driving and take a break."

[0957] Step 6:

[0958] The server sends the generated advice or warning to the terminal, and the notification includes the specific advice and the reason for it.

[0959] Step 7:

[0960] The device will notify the user of the notifications it receives through audio and visual displays, which are updated in real time to provide the user with information to take appropriate action.

[0961] Step 8:

[0962] Users can take appropriate action based on the notification, such as stopping driving and taking a break if they feel tired or stressed.

[0963] The above processing steps provide appropriate advice and warnings according to changes in the user's physical condition and emotions, thereby improving the safety and comfort of elderly drivers.

[0964] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0965] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0966] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0967] [Third embodiment]

[0968] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0969] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0970] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0971] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0972] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0973] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0974] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0975] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0976] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0977] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0978] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0979] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0980] The system of this invention is a voice recognition IoT device that supports the lives of elderly people and aims to make it easy for users (elderly people) to use. This system mainly consists of three components: a terminal, a server, and a user. Each component and its function will be explained in detail below.

[0981] Device Features

[0982] The terminal is a device used by the elderly in their daily lives, and is mainly used to receive voice input, send voice data, and receive and display notifications. The terminal receives voice input from the user through a microphone and converts it into digital data using voice recognition technology. This digital data is then sent to a server via the Internet.

[0983] Server Features

[0984] The server is a central processing unit that receives and analyzes the voice data sent from the device. The server analyzes the data using machine learning algorithms and natural language processing (NLP) technology to detect changes in physical condition and lifestyle habits.

[0985] For example, if a user says, "I've had a stomachache recently," the server converts this voice data into text and extracts the keyword "stomachache." Based on this information, the server then assesses the user's health risk and generates an appropriate notification. The generated notification is then sent back to the device, where it is displayed to the user both audibly and on the screen.

[0986] User Actions

[0987] Users operate the system by giving voice commands to the device. For example, if they say "Turn on the TV," the device receives this voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and sends it back to the device. The device then controls the TV based on that command.

[0988] In addition, if the user utters a keyword such as "help" in an emergency, the device detects this and immediately sends an emergency notification to the server, which then notifies pre-registered emergency contacts (family members or medical institutions).

[0989] Specific examples

[0990] 1. Daily use:

[0991] When a user says, "Tell me the weather today," the device converts the speech into text and requests weather information from the server. The server retrieves the information from the weather database and notifies the user by voice and on the screen, saying, "It's sunny today."

[0992] 2. Health management:

[0993] When a user says, "I've been getting tired easily lately," the server analyzes this information, and if it determines there is an abnormality, it notifies the user, "You seem tired, so make sure you get plenty of rest."

[0994] 3. Emergency Response:

[0995] When a user shouts "help," the device detects this as an emergency keyword and immediately sends a notification to the server, which then automatically notifies the linked emergency contacts.

[0996] As described above, the system of the present invention is designed to be easy for the elderly to use, and aims to improve the quality of life and ensure safety for the elderly by providing a wide range of support, from assistance with daily life to responding to emergencies.

[0997] The processing flow will be explained below.

[0998] Daily use

[0999] When saying "What's the weather today?"

[1000] Step 1:

[1001] The user utters "What's the weather like today?"

[1002] Step 2:

[1003] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[1004] Step 3:

[1005] The terminal transmits the converted text data "Tell me the weather today" to the server.

[1006] Step 4:

[1007] The server analyzes the received text data and sends a request to retrieve weather information to an internal weather database or an external API.

[1008] Step 5:

[1009] The server sends the acquired weather information (e.g., sunny) to the terminal in text format.

[1010] Step 6:

[1011] The device converts the received text data into speech using a speech synthesis engine, and notifies the user by voice, saying, "It's sunny today." The device also displays the weather information on the screen.

[1012] health care

[1013] When saying "I get tired easily these days"

[1014] Step 1:

[1015] The user vocally says, "I've been feeling tired a lot recently."

[1016] Step 2:

[1017] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[1018] Step 3:

[1019] The terminal transmits the converted text data "I've been feeling tired lately" to the server.

[1020] Step 4:

[1021] The server analyzes the received text data and extracts the keyword "easily tired."

[1022] Step 5:

[1023] The server evaluates the user's health condition based on these keywords and generates a warning message, such as "You appear tired, so please try to get plenty of rest."

[1024] Step 6:

[1025] The server sends the generated warning message to the terminal.

[1026] Step 7:

[1027] The device converts the received warning message into speech using a speech synthesis engine, and notifies the user by voice, saying, "You seem tired, so please make sure to get plenty of rest." The device also displays a message on the screen.

[1028] Emergency response

[1029] Saying "Help me"

[1030] Step 1:

[1031] The user utters the words "help me."

[1032] Step 2:

[1033] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[1034] Step 3:

[1035] The device analyzes the converted text data "help" and detects it as an emergency keyword.

[1036] Step 4:

[1037] The device sends an emergency notification to the server, which also includes the user's location information.

[1038] Step 5:

[1039] Based on the received emergency notification, the server automatically notifies pre-registered emergency contacts (e.g., family members or medical institutions).

[1040] Step 6:

[1041] The emergency contact receives the notification from the server and initiates appropriate action.

[1042] As described above, through specific processing at each step, the system of the present invention can support the lives of elderly people and respond quickly in emergencies.

[1043] Example 1

[1044] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1045] To effectively support the daily lives of the elderly, it is necessary to provide systems that are easy for users to operate. In particular, it is necessary to realize a system that uses voice recognition technology to support the daily lives of the elderly and can respond quickly in emergencies.

[1046] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1047] In this invention, the server includes a means for receiving voice input from the user, a means for converting the received voice data into digital data, a means for transmitting the converted voice data to the server, and a means for displaying the notification transmitted from the server by voice and on a screen. This makes it easy for elderly people to operate and enables a wide range of applications from assistance with daily life to emergency response.

[1048] "Users" refer to elderly people who use the system.

[1049] "Voice input" refers to the voice that the user speaks through a microphone, and includes instructions and requests for information to the system.

[1050] "Voice Data" means data that has been converted into digital form from a User's voice.

[1051] A "server" is a central processing unit that analyzes received voice data and generates responses or notifications.

[1052] A "notification" is an information message generated by a server and sent to a terminal, and is conveyed to the user by sound or on-screen display.

[1053] "Digital data" refers to data that has been converted from analog audio signals into a format that can be processed by a computer.

[1054] "Speech recognition technology" refers to the technology that converts voice into text data.

[1055] A "machine learning algorithm" is a technology that learns patterns from large amounts of data and makes predictions and classifications.

[1056] "Natural language processing (NLP)" is the technology that enables computers to understand, interpret, and generate human language.

[1057] "Changes in physical condition and lifestyle" refers to changes in the user's health condition and behavioral patterns in their daily lives.

[1058] "Emergency keyword voice" refers to voices calling for help or specific words indicating an emergency.

[1059] An "emergency notification" is a notification generated by the server and sent to emergency contacts when an emergency keyword voice is detected.

[1060] "Emergency contacts" are pre-registered contacts such as family members or medical institutions to be contacted in an emergency.

[1061] A "device" is a device that a user directly controls, receives voice input, and displays notifications.

[1062] The system of this invention is a voice recognition IoT device designed to support the lives of the elderly. It is designed so that users (elderly people) can easily use the system. The system mainly consists of three components: a terminal, a server, and a user.

[1063] Device Features

[1064] Terminals are devices used by seniors in their daily lives. They have the following functions:

[1065] Receiving voice input: Receives voice input from the user through the microphone. For example, when a user says "Tell me the weather today," the device's microphone receives this voice.

[1066] Digital conversion and transmission of voice data: The received voice is converted into digital data and sent to the server. Specifically, the voice is converted into text using a voice recognition engine (e.g., Google Speech-to-Text), and the converted data is sent to the server via the Internet.

[1067] Server Features

[1068] The server is a central processing unit that receives and analyzes the voice data sent from the terminal. It performs the following processes:

[1069] Voice data analysis: Using machine learning algorithms and natural language processing (NLP) techniques, voice data is converted into text and intent and keywords are extracted. For example, if a user says, "Tell me the weather today," the server will extract "weather" as a keyword and query a weather database to obtain weather information.

[1070] Health assessment: Analyzes changes in the user's physical condition and lifestyle to assess risk. For example, if a user says, "I've been feeling tired lately," the server will detect the keyword "fatigue" and generate a notification saying, "You appear tired, so please try to get plenty of rest."

[1071] Creating notifications: Based on the analysis results, appropriate notifications are created and sent back to the device.

[1072] User Actions

[1073] The user operates the system by issuing voice commands to the terminal. For example, if the user says "Turn on the TV," the terminal receives this voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and returns a command to the terminal. The terminal then operates the TV based on that command. A specific example of how this works is a sequence in which a voice recognition engine analyzes the command and operates an infrared remote control.

[1074] In addition, in the event of an emergency, if the user utters "help," the device will detect this as an emergency keyword and immediately send a notification to the server. The server will then automatically notify pre-set emergency contacts (family members or medical institutions). For example, if you say "help," the server will receive that information and send an emergency notification via message or call to the registered emergency contacts.

[1075] Specific examples

[1076] 1. Daily use:

[1077] The user says, "What's the weather like today?"

[1078] The terminal converts the voice into digital data and sends it to the server.

[1079] The server retrieves information such as "It's sunny today" from a weather database and sends a notification to the device.

[1080] The device will notify the user by voice and on the screen that "It's sunny today."

[1081] 2. Health management:

[1082] The user says, "I've been feeling tired a lot lately."

[1083] The server analyzes the voice and detects the keyword "fatigue."

[1084] The server generates a notification saying, "You appear tired, so please make sure to get plenty of rest," and sends it to the device.

[1085] The device will notify the user.

[1086] 3. Emergency Response:

[1087] The user says "help."

[1088] The device detects the emergency keyword and immediately sends a notification to the server.

[1089] The server will then notify pre-defined emergency contacts.

[1090] In this way, the system of the present invention is designed to be easy for the elderly to operate, and by covering a wide range of functions from assistance with daily life to responding to emergencies, it is expected to improve the quality of life of the elderly.

[1091] Prompt Sentence Examples

[1092] "Design a voice-activated IoT device to assist the elderly in their daily lives. Explain its specific functions, including the data flow between the device and the server, and provide examples."

[1093] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1094] Step 1:

[1095] The user inputs voice

[1096] The user speaks instructions and questions into the device.

[1097] Specific action: The user speaks to the device, "Tell me what the weather is like today."

[1098] Input: User's voice

[1099] Output: Audio data (analog signal)

[1100] Step 2:

[1101] The device converts the audio into digital data

[1102] The terminal uses a speech recognition engine to convert the received voice into digital data.

[1103] Specific operation: The device's microphone picks up an analog voice signal and inputs this signal into a voice recognition engine (e.g., Google Speech-to-Text).

[1104] Input: Analog audio signal

[1105] Output: Text data

[1106] Step 3:

[1107] The device sends text data to the server.

[1108] The terminal transmits the converted text data to a server via the Internet.

[1109] Specific operation: The text data generated by the device is sent to the specified API endpoint on the server.

[1110] Input: Text data

[1111] Output: Network transmission

[1112] Step 4:

[1113] The server parses the text data

[1114] The server analyzes the received text data using machine learning algorithms and natural language processing techniques.

[1115] Specific operation: After the server receives the text data, it passes the data to an NLP module (e.g., SpaCy or BERT) to extract intent and keywords.

[1116] Input: Text data

[1117] Output: Analysis results (keywords and intent)

[1118] Step 5:

[1119] The server obtains the necessary information

[1120] Based on the analysis results, the server retrieves the necessary information from external databases and APIs.

[1121] Specific operation: For example, if the analysis result contains the keyword "weather," the server will query the weather database to obtain the latest weather information.

[1122] Input: Analysis results (keywords and intent)

[1123] Output: Information from an external database

[1124] Step 6:

[1125] The server generates the notification

[1126] Based on the information obtained, the server generates the appropriate notifications.

[1127] Specific operation: Create a notification message saying "It's sunny today" based on weather information.

[1128] Input: Information retrieved from an external database

[1129] Output: Notification message

[1130] Step 7:

[1131] The server sends a notification to the device

[1132] The generated notification is sent from the server to the terminal.

[1133] Specific operation: The server sends a notification message to the terminal via the network.

[1134] Input: Notification message

[1135] Output: Network transmission

[1136] Step 8:

[1137] Your device will notify you with audio and visual notifications

[1138] The device will present the received notification to the user via audio and visual notification.

[1139] Specific operation: The device converts the notification message into voice using a speech synthesis engine (e.g., Google Text-to-Speech) and plays it over the speaker. The notification message is also displayed on the screen.

[1140] Input: Notification message

[1141] Output: Audio notification and screen display

[1142] The above are the main processing steps of this system. The specific operations and data processing performed at each step have been explained in detail.

[1143] (Application example 1)

[1144] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1145] In factory work sites, many workers rely on manual and visual confirmation to execute complex instructions, leaving room for improvement in terms of efficiency and safety. Rapid response in emergencies is also required, but due to the nature of the work, it is often difficult to immediately detect and respond to emergencies. Therefore, there is a need for a system that improves the efficiency of factory work, ensures safety, and enables rapid response in emergencies.

[1146] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1147] In this invention, the server includes means for receiving voice input from a user, means for transmitting the received voice data to the server, means for displaying the notification transmitted from the server by voice and on a screen, means for controlling the robot based on the received voice data, and means for detecting the voice data from the user in an emergency and generating and transmitting a corresponding notification, thereby improving work efficiency in the factory and enabling a rapid response in an emergency.

[1148] A "user" is a human user who operates a system.

[1149] "Voice input" refers to instructions or information provided by a user to a system through voice.

[1150] A "server" is a central processing unit that receives and analyzes the voice data and generates appropriate notifications and control signals.

[1151] "Notifications" are messages generated by the server to inform users of analysis results or important information.

[1152] A "robot" is a mechanical device that performs physical tasks based on instructions received from an automated control system connected to the system.

[1153] "Emergency" refers to a series of conditions or events that cannot be handled through normal operations or tasks and require immediate action.

[1154] "Keyword voice" is a specific word or phrase that is preset to indicate an emergency or a specific command.

[1155] "Analysis" is the process by which the server analyzes the received voice data, interprets the meaning, and determines the appropriate action.

[1156] "Control" refers to the system's ability to operate robots and other devices based on voice commands.

[1157] An "emergency notification" is a special message generated by the server to notify you of an emergency problem.

[1158] This invention is a system that uses a voice recognition IoT device to improve efficiency and safety at factory work sites. The detailed configuration of this system and its operating principle will be described below.

[1159] System configuration

[1160] The system consists of three main components:

[1161] 1. Terminal

[1162] 2. Server

[1163] 3. Users

[1164] Device Features

[1165] The device receives voice input from the user and converts the voice into digital data, which is then sent to a server via the Internet. It also receives notifications sent from the server and notifies the user via voice and on a screen.

[1166] Server Features

[1167] The server receives and analyzes the voice data sent from the device. This analysis uses machine learning algorithms and natural language processing (NLP) technology. The server analyzes the voice instructions, extracts appropriate commands, and generates work instructions and emergency response notifications. These notifications are then sent back to the device and displayed to the user via voice and on the screen.

[1168] Robot Control

[1169] The robot performs specific tasks based on instructions from the server. This control includes linking with IoT devices, and the robot operates according to received voice instructions. This allows users to give voice instructions to the robot and efficiently carry out tasks.

[1170] User Actions

[1171] Users can give voice commands to the device. For example, if they say "start the machine," the voice is converted into digital data on the device, which is then analyzed by the server and sent to the robot. Also, if an emergency keyword such as "help" is spoken, the system will immediately initiate an emergency response and notify emergency contacts.

[1172] Hardware and software used

[1173] The following hardware and software are used to realize this system.

[1174] Microphone and speaker: A high-sensitivity microphone receives audio input and a speaker provides audio output.

[1175] Robot control device: Uses control hardware for IOT devices.

[1176] Natural Language Processing and Speech Recognition Module: Python module 'speech_recognition' is used to convert voice data to text, and 'text_to_speech' is used to output speech. In addition, 'iot_device_control' module is used to control IOT devices.

[1177] Specific examples

[1178] 1. Daily tasks:

[1179] "Start the machine": The terminal sends the voice to the server, which analyzes it and sends instructions to the robot to start working.

[1180] "Stop the machine": Similarly, the voice command from the terminal is analyzed by the server and a command to stop is sent to the robot.

[1181] 2. Emergency Response:

[1182] "Help me": This voice command is detected by the device and sent to the server, which then generates an emergency notification and sends it to your emergency contacts.

[1183] Prompt Sentence Examples

[1184] Create a voice assistant application for a factory robot that will assist me. Features include:

[1185] 1. Receive user instructions through voice recognition.

[1186] 2. Operate IOT devices based on instructions (start, stop).

[1187] 3. Send emergency notifications with the command "Help" in case of an emergency.

[1188] This system can significantly improve work efficiency and safety within the factory.

[1189] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1190] Step 1:

[1191] Audio reception and conversion

[1192] The device receives voice input from the user. This voice data (input) is received by the microphone and sent to the speech recognition module. This module converts the voice data into text and generates digital data (output). Specifically, the 'speech_recognition' module is used to analyze the voice and convert it into a string of characters.

[1193] Step 2:

[1194] Data transmission

[1195] The device sends the text data generated in step 1 to the server. This data transmission (input) is performed via the Internet. Specifically, an HTTP request is used to send the data to the server's API.

[1196] Step 3:

[1197] Data analysis

[1198] The server analyzes the received text data (input) and understands the user's instructions. This analysis uses machine learning algorithms and natural language processing (NLP). The analysis results (output) are generated as instruction commands. Specifically, the server uses an NLP module to extract keywords and determine the appropriate action.

[1199] Step 4:

[1200] Notification Generation and Delivery

[1201] The server generates an appropriate notification based on the analysis results of step 3 and sends it to the device. This notification (input) is generated as data for voice and screen display. Specific operations include using the 'text_to_speech' module, which converts text data into voice data.

[1202] Step 5:

[1203] Robot Control

[1204] Based on the analysis results of step 3, the server generates a signal to control the IOT device and sends it to the robot. This control signal (input) is sent using a protocol for controlling IOT devices. Specifically, the 'iot_device_control' module generates a command to operate the robot and sends it to the device.

[1205] Step 6:

[1206] Receiving and displaying notifications

[1207] The device receives notifications sent from the server and notifies the user via voice and on the screen. This notification (input) includes audio data played on the device and text data displayed on the screen. Specifically, the 'text_to_speech' module generates audio and renders data for display on the screen.

[1208] Step 7:

[1209] Emergency response

[1210] When a user utters an emergency keyword, the device detects it and immediately sends an emergency notification to the server. This emergency notification (input) is analyzed by the server and a notification (output) is generated to be sent to emergency contacts. Specifically, when a specific keyword is detected, a function to send an email or SMS to emergency contacts is triggered.

[1211] In this way, each processing step begins with the user's voice input, and ultimately leads to robot control and emergency response, aiming to improve work efficiency and safety within the factory.

[1212] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1213] The system of the present invention provides a voice recognition IoT device combined with an emotion engine to more actively support the lives of the elderly. The system is designed to be easy for users (elderly people) to use and consists of three components: a terminal, a server, and a user. Each component and its function are explained in detail below.

[1214] Device Features

[1215] The terminal is a device used by elderly people in their daily lives, and is mainly used to receive voice input, send voice data, and receive and display notifications. The terminal receives voice input from the user through a microphone and converts it into digital data using a voice recognition engine and an emotion engine, while also identifying the user's emotions. This digital data and emotion data are sent to a server via the Internet.

[1216] Server Features

[1217] The server is a central processing unit that receives and analyzes the voice and emotion data sent from the device. The server uses machine learning algorithms and natural language processing (NLP) techniques to analyze the data and detect changes in physical condition and lifestyle habits. Furthermore, by analyzing the emotion data, the server can assess the user's stress and anxiety levels.

[1218] For example, if a user says, "I've been feeling tired lately," but speaks in an anxious tone, the server analyzes the user's emotions along with the voice data and generates an appropriate notification. In this case, the server generates a notification that advises the user to relax and, if necessary, instructs the user to consult a medical professional. The generated notification is then sent back to the device and notified to the user by voice and on-screen display.

[1219] User Actions

[1220] Users operate the system by giving voice commands to the device. For example, if they say "Turn on the TV," the device receives this voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and sends it back to the device. The device then controls the TV based on that command.

[1221] In addition, if the user utters a keyword such as "help" in an emergency, the device detects this and immediately sends an emergency notification to the server, which then notifies pre-registered emergency contacts (family members or medical institutions).

[1222] Specific examples

[1223] 1. Daily use:

[1224] When a user says, "Tell me the weather today," the device converts the voice into text and sends it to the server. The server obtains the weather information and sends it to the device. The device then announces, "It's sunny today," by voice and also displays it on the screen.

[1225] 2. Health management:

[1226] If a user says, "I get tired easily recently," but their voice is unsteady, the server will recognize the keywords "tired easily" and "anxiety" and generate a notification offering advice on how to relax. The device will then notify the user by voice and on-screen display.

[1227] 3. Emergency Response:

[1228] When a user shouts "help," the device detects this as an emergency keyword and sends an emergency notification to the server, which then notifies emergency contacts.

[1229] 4. Emotional Continuity Management:

[1230] The server stores emotional data extracted from daily conversations and periodically analyzes the user's emotional patterns. Based on the results of this analysis, the system provides individually customized relaxation suggestions and health-related advice during times of high stress.

[1231] As described above, the system of the present invention uses an emotion engine to support the lives of the elderly in a more personalized way, enabling early detection of health risks and rapid response in emergencies.

[1232] The processing flow will be explained below.

[1233] health care

[1234] When you say "I've been feeling tired lately" and it's accompanied by emotion recognition

[1235] Step 1:

[1236] The user vocally says, "I've been feeling tired a lot recently."

[1237] Step 2:

[1238] The device receives the user's voice through a microphone and converts it into text data using a voice recognition engine, while also analyzing the tone and pitch of the user's voice using an emotion engine to generate emotion data.

[1239] Step 3:

[1240] The device sends the converted text data "I've been feeling tired lately" and the analyzed emotional data (e.g., anxiety) to the server.

[1241] Step 4:

[1242] The server analyzes the received text data, extracts health-related keywords such as "getting tired easily," and analyzes the emotion data to determine that the user is feeling anxious.

[1243] Step 5:

[1244] Based on this information, the server will perform a risk assessment of the user's health status and emotional data, and generate an appropriate notification message, such as "You seem to be feeling tired and anxious recently. Please get plenty of rest and consult a medical professional if necessary."

[1245] Step 6:

[1246] The server sends the generated notification message to the terminal.

[1247] Step 7:

[1248] The device converts the received notification message into speech using a speech synthesis engine, and notifies the user aloud, saying, "You seem to have become tired and anxious recently. Please get plenty of rest and consult a medical professional if necessary." The same message is also displayed on the screen.

[1249] Emergency response

[1250] When you say "Help me"

[1251] Step 1:

[1252] The user utters the words "help me."

[1253] Step 2:

[1254] The device receives the user's voice through a microphone, converts it into text data using a speech recognition engine, and analyzes the urgency of the user's voice using an emotion engine.

[1255] Step 3:

[1256] The device sends an emergency notification to the server, including the converted text data "Help me" and the analyzed emotion data (e.g., extreme stress).

[1257] Step 4:

[1258] The server analyzes the received emergency notification and detects the emergency keyword "help."

[1259] Step 5:

[1260] The server will promptly notify pre-registered emergency contacts (e.g., family members or medical institutions), including the user's location information.

[1261] Step 6:

[1262] The emergency contact receives the notification from the server and initiates appropriate action.

[1263] Daily use

[1264] When saying "What's the weather today?"

[1265] Step 1:

[1266] The user utters "What's the weather like today?"

[1267] Step 2:

[1268] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[1269] Step 3:

[1270] The terminal transmits the converted text data "Tell me the weather today" to the server.

[1271] Step 4:

[1272] The server analyzes the received text data and sends a request to retrieve weather information to an internal weather database or an external API.

[1273] Step 5:

[1274] The server sends the acquired weather information (e.g., sunny) to the terminal in text format.

[1275] Step 6:

[1276] The device converts the received weather information into voice using a voice synthesis engine, notifies the user by voice, such as "It's sunny today," and also displays the weather information on the screen.

[1277] Emotional Continuity Management

[1278] Analysis of daily conversation data

[1279] Step 1:

[1280] The terminal receives voice input from the user on a daily basis, converts it into text data using a voice recognition engine, and simultaneously generates emotion data using an emotion engine.

[1281] Step 2:

[1282] The terminal transmits the converted text data and emotion data to the server.

[1283] Step 3:

[1284] The server stores the voice data and emotion data accumulated daily in a database.

[1285] Step 4:

[1286] The server periodically analyzes the accumulated data and evaluates the user's emotional patterns.

[1287] Step 5:

[1288] Based on the analysis results, the server generates notification messages to provide personalized health support to the user. For example, if the user's long-term stress level is high, the server generates a message recommending relaxation techniques or counseling.

[1289] Step 6:

[1290] The server sends the generated notification message to the terminal.

[1291] Step 7:

[1292] The device converts the received notification message into voice using a speech synthesis engine and notifies the user. The device also displays the same message on the screen.

[1293] By combining this system with an emotion engine, it will further personalize support for the elderly, enabling more effective health management and faster emergency response.

[1294] Example 2

[1295] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1296] In recent years, there has been a growing need for life support systems for the elderly, but conventional systems simply receive voice input and notify users, without taking into account their emotions or changes in their physical condition. This makes it difficult to provide personalized support to users, and there are problems with being unable to respond quickly and appropriately, especially in emergencies.

[1297] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1298] In this invention, the server includes means for analyzing the user's voice data and emotion data to detect changes in physical condition and lifestyle habits, means for generating a notification based on the detected changes, and means for transmitting the generated notification to the terminal. This enables personalized notifications that take into account the user's emotions and changes in physical condition, enabling a quick and appropriate response in an emergency.

[1299] "Voice input" means spoken information such as instructions or questions spoken by a user.

[1300] "Device" means any equipment or hardware used to receive, analyze, or transmit audio input.

[1301] "Text data" refers to character string data obtained by analyzing voice input.

[1302] "Emotional Data" means data used to identify a user's emotional state through voice analysis.

[1303] "Server" refers to a central processing unit that receives voice and emotion data, analyzes it, and generates notifications.

[1304] "Notification" means information or instructions provided to the user, whether through audio or visual display.

[1305] An "emergency" is defined as the moment a user reports a dangerous situation or health-related issue.

[1306] "Terminal" refers to a device that a user operates and uses to input voice information.

[1307] "Personalization" means providing information and assistance in a way that is adapted based on the user's individual situation and emotional state.

[1308] A "machine learning algorithm" refers to a computational method for learning from data and making predictions and analyses.

[1309] The system of the present invention provides an IoT device that combines an emotion engine and voice recognition technology to more actively support the lives of users (elderly people). This system is designed to be easy for users to use and consists of three components: a terminal, a server, and a user. The following describes in detail the modes for implementing the present invention.

[1310] Device Features

[1311] A device is a device that users use in their daily lives, and it mainly receives voice input, processes voice data, and receives and displays notifications. Specifically, a device has the following functions:

[1312] The user's voice is received by a microphone and converted into digital data using a voice recognition engine.

[1313] Analyze user emotions from voice using an emotion engine.

[1314] The converted voice data and emotion data are transmitted to a server.

[1315] Notify the user of notifications sent from the server via audio and visual.

[1316] Server Features

[1317] The server is a central processing unit that receives and analyzes the voice and emotion data sent from the device. Specifically, the server has the following functions:

[1318] The received data is analyzed using machine learning algorithms and natural language processing (NLP) to detect changes in the user's physical condition and lifestyle.

[1319] Evaluate users' stress and anxiety levels based on emotional data.

[1320] Based on the analysis results, an appropriate notification is generated and sent to the device.

[1321] User Actions

[1322] Users operate the system by issuing voice commands to the device. For example, if a command is given to "turn on the TV," the device receives the voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and sends it back to the device, which then operates the TV based on that command. Also, if a user utters "help" in an emergency, the device detects this and immediately sends an emergency notification to the server. The server receives this emergency notification and notifies pre-registered emergency contacts.

[1323] Specific use cases

[1324] Below are some specific examples of how this system can be used.

[1325] 1. Daily use

[1326] When a user says, "Tell me the weather today," the device converts the voice into text and sends it to the server. The server obtains the weather information and sends it to the device. The device then announces, "It's sunny today," by voice and also displays it on the screen.

[1327] 2. Health management

[1328] If a user says, "I get tired easily recently," but their voice is unsteady, the device recognizes this and sends it to the server. The server then recognizes the keywords "tired easily" and "anxiety" and generates a notification offering advice on how to relax. The device then notifies the user with the advice via voice and on the screen.

[1329] 3. Emergency Response

[1330] When a user shouts "help," the device detects this as an emergency keyword and sends an emergency notification to the server, which then notifies emergency contacts.

[1331] 4. Emotional Continuity Management

[1332] The server stores emotional data extracted from daily conversations and periodically analyzes the user's emotional patterns. Based on the results of this analysis, the system provides individually customized relaxation suggestions and health-related advice during times of high stress.

[1333] Example prompts for generative AI models

[1334] Below are some example prompts to input to the generative AI model:

[1335] Example prompt 1:

[1336] If a user says "I've been feeling tired lately" to a device by voice and is emotionally unstable, explain how the server generates appropriate advice and notifies the device.

[1337] Example prompt 2:

[1338] Explain how the device and server work together to notify emergency contacts when a user calls for help in an emergency.

[1339] As described above, the system of the present invention uses an emotion engine to support the lives of elderly people in a personalized way, enabling early detection of health risks and rapid response in emergencies.

[1340] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1341] Step 1:

[1342] The user provides voice input.

[1343] Specific operation: The user speaks to the device, saying, "Tell me the weather today."

[1344] Input: User's voice

[1345] Output: Audio signal

[1346] Step 2:

[1347] The device collects and processes the voice data.

[1348] Specific operation: The device uses a built-in microphone to collect voice signals, which are then converted into text data by a voice recognition engine. At the same time, the emotion engine analyzes the user's emotion data from the voice.

[1349] Input: Audio signal

[1350] Output: Text data, emotion data

[1351] Step 3:

[1352] The device transmits emotion data and voice data to the server.

[1353] Specific operation: The device assembles the generated text data "Tell me today's weather" and emotion data into a packet and sends it to a server via the Internet.

[1354] Input: Text data, emotion data

[1355] Output: Data packets

[1356] Step 4:

[1357] The server receives and processes the data.

[1358] How it works: The server decodes data packets received via the internet, extracts text and emotion data, and uses NLP techniques to analyze the user's intent and evaluates voice data to understand their emotions.

[1359] Input: Data packet

[1360] Output: User's intention, emotion evaluation results

[1361] Step 5:

[1362] The server generates the analysis results and generates the necessary notifications.

[1363] Specific operation: Based on the user's intent to "check the weather," the server retrieves the latest weather information and creates a notification saying "It's sunny today." If the emotion is "anxiety," the server also generates a notification about relaxation methods, if necessary.

[1364] Input: User's intention, emotion evaluation results

[1365] Output: Notification data

[1366] Step 6:

[1367] The server transmits the notification data to the terminal.

[1368] Specific operation: The server sends notification data "It's sunny today" to the terminal via the Internet.

[1369] Input: Notification data

[1370] Output: Data packets

[1371] Step 7:

[1372] The terminal provides a notification to the user.

[1373] Specific behavior: The device analyzes the received notification data, notifies the user by voice, saying "It's sunny today," and displays weather information on the screen. If additional notifications based on emotions are needed, they will also be displayed.

[1374] Input: Notification data

[1375] Output: Audio notification, screen display

[1376] Through the above processing steps, the system provides personalized responses to the user's needs and realizes advanced support based on changes in emotions and physical condition.

[1377] (Application example 2)

[1378] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1379] When elderly people use self-driving vehicles, there is a problem that there is insufficient support to grasp changes in their physical condition and emotions in real time and to ensure safe and comfortable use. In addition, simply recognizing voice commands tends to delay response in emergency situations involving elderly people. Therefore, it is necessary to establish a system that analyzes the emotional state of elderly drivers and provides appropriate advice and warnings.

[1380] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1381] In this invention, the server includes means for receiving voice input, means for analyzing the received voice data and emotion data, and means for generating appropriate notifications based on the analysis results, thereby improving safety and comfort for elderly drivers.

[1382] A "means for receiving audio input" is a device that captures the user's spoken voice and converts it into digital data.

[1383] The "means for transmitting voice data to a server" is a network communication function for sending the received voice data to a server via the Internet.

[1384] "Means for displaying notifications by voice and on screen" refers to a function that displays notifications sent from the server to the user by voice or on a display.

[1385] "Means for identifying emotional data" refers to a function that analyzes a user's emotional state from the user's voice or other input.

[1386] The "means for transmitting emotion data to a server" is a network communication function for transmitting the identified emotion data to a server.

[1387] The "means for generating advice or warning based on the analysis results" is a function for generating appropriate advice or warning based on the analysis results obtained by the server.

[1388] "Means for notifying advice and warnings" refers to a function that notifies the user of generated advice and warnings by voice or display.

[1389] "Means for detecting changes in physical condition and lifestyle habits" refers to a function that detects changes in the user's physical condition and lifestyle habits by analyzing voice data and emotional data.

[1390] "Means for detecting emergency keyword voice" is a function that recognizes emergency keywords uttered by the user in real time.

[1391] The "means for notifying the server of voice in an emergency" is a function that immediately transmits voice data detected in an emergency to the server.

[1392] "Means for generating and notifying appropriate measures in an emergency" is a function that creates appropriate measures in an emergency from the results of analysis on the server and notifies the user of them.

[1393] The system of the present invention is a system for assisting elderly drivers in an autonomous vehicle, and includes the following components: a voice recognition device, an emotion analysis engine, a server, and a user interface.

[1394] Hardware and Software Configuration

[1395] 1. Voice Recognition Device

[1396] A microphone is installed inside the vehicle to receive the user's voice input.

[1397] Convert speech to text data using a speech recognition engine (e.g., speech_recognition library).

[1398] 2. Sentiment Analysis Engine

[1399] Use an emotion analysis engine (e.g., emotion_recognition library) to identify emotions from user voice data.

[1400] 3. Server

[1401] Voice data and emotion data are received and analyzed.

[1402] Data is analyzed using machine learning algorithms and natural language processing (NLP) techniques.

[1403] It detects changes in the user's physical condition and lifestyle and generates appropriate advice and warnings.

[1404] 4. User Interface

[1405] The system has the function of notifying the user of the analysis results by voice and on-screen display, allowing the user to receive instructions in real time while in the vehicle.

[1406] Program processing

[1407] The server receives the voice data transmitted from the voice recognition device.

[1408] The emotion analysis engine identifies emotions from the voice data and sends them to the server.

[1409] The server analyzes the received voice and emotional data to detect changes in the user's physical condition and lifestyle.

[1410] Based on the detected changes, the server generates appropriate advice and warnings and notifies the user via the user interface.

[1411] Specific examples

[1412] For example, if a user says "I'm tired" while driving, the voice recognition device converts this speech into text data, and the emotion analysis engine identifies the emotional state "tired." The server analyzes this data and confirms that the user is tired. The server then generates advice such as "We recommend you stop driving and take a break," and notifies the user by voice and on-screen display.

[1413] Prompt Sentence Examples

[1414] You can use the following prompts to leverage specific generative AI models:

[1415] Build a voice recognition app for driving assistance for the elderly. The app recognizes voice input, analyzes the user's emotions, and provides appropriate advice and warnings while driving. For example, if a user says "I'm tired," the app will recognize it, analyze the emotion data, and notify the user, "We recommend that you stop driving and take a break." The libraries used are speech_recognition and emotion_recognition.

[1416] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1417] Step 1:

[1418] The device receives the user's voice input through a microphone, which is converted into digital data by a speech recognition engine.

[1419] Step 2:

[1420] The device sends the converted voice data to an emotion analysis engine, which identifies the user's emotion from the voice data. In this process, an emotion analysis model is used to analyze the voice tone and content to generate emotion data.

[1421] Step 3:

[1422] The terminal transmits the generated voice data and emotion data to the server, and the transmitted data includes the text content of the voice and the emotion data.

[1423] Step 4:

[1424] The server analyzes the received voice and emotion data using machine learning algorithms and natural language processing (NLP) techniques, allowing it to detect changes in the user's physical condition and lifestyle.

[1425] Step 5:

[1426] The server generates appropriate advice and warnings based on the analysis results. For example, if the user's stress level is high, it will generate a notification such as "We recommend that you stop driving and take a break."

[1427] Step 6:

[1428] The server sends the generated advice or warning to the terminal, and the notification includes the specific advice and the reason for it.

[1429] Step 7:

[1430] The device will notify the user of the notifications it receives through audio and visual displays, which are updated in real time to provide the user with information to take appropriate action.

[1431] Step 8:

[1432] Users can take appropriate action based on the notification, such as stopping driving and taking a break if they feel tired or stressed.

[1433] The above processing steps provide appropriate advice and warnings according to changes in the user's physical condition and emotions, thereby improving the safety and comfort of elderly drivers.

[1434] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1435] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1436] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1437] [Fourth embodiment]

[1438] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1439] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1440] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1441] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1442] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1443] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1444] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1445] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1446] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1447] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1448] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1449] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1450] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1451] The system of this invention is a voice recognition IoT device that supports the lives of elderly people and aims to make it easy for users (elderly people) to use. This system mainly consists of three components: a terminal, a server, and a user. Each component and its function will be explained in detail below.

[1452] Device Features

[1453] The terminal is a device used by the elderly in their daily lives, and is mainly used to receive voice input, send voice data, and receive and display notifications. The terminal receives voice input from the user through a microphone and converts it into digital data using voice recognition technology. This digital data is then sent to a server via the Internet.

[1454] Server Features

[1455] The server is a central processing unit that receives and analyzes the voice data sent from the device. The server analyzes the data using machine learning algorithms and natural language processing (NLP) technology to detect changes in physical condition and lifestyle habits.

[1456] For example, if a user says, "I've had a stomachache recently," the server converts this voice data into text and extracts the keyword "stomachache." Based on this information, the server then assesses the user's health risk and generates an appropriate notification. The generated notification is then sent back to the device, where it is displayed to the user both audibly and on the screen.

[1457] User Actions

[1458] Users operate the system by giving voice commands to the device. For example, if they say "Turn on the TV," the device receives this voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and sends it back to the device. The device then controls the TV based on that command.

[1459] In addition, if the user utters a keyword such as "help" in an emergency, the device detects this and immediately sends an emergency notification to the server, which then notifies pre-registered emergency contacts (family members or medical institutions).

[1460] Specific examples

[1461] 1. Daily use:

[1462] When a user says, "Tell me the weather today," the device converts the speech into text and requests weather information from the server. The server retrieves the information from the weather database and notifies the user by voice and on the screen, saying, "It's sunny today."

[1463] 2. Health management:

[1464] When a user says, "I've been getting tired easily lately," the server analyzes this information, and if it determines there is an abnormality, it notifies the user, "You seem tired, so make sure you get plenty of rest."

[1465] 3. Emergency Response:

[1466] When a user shouts "help," the device detects this as an emergency keyword and immediately sends a notification to the server, which then automatically notifies the linked emergency contacts.

[1467] As described above, the system of the present invention is designed to be easy for the elderly to use, and aims to improve the quality of life and ensure safety for the elderly by providing a wide range of support, from assistance with daily life to responding to emergencies.

[1468] The processing flow will be explained below.

[1469] Daily use

[1470] When saying "What's the weather today?"

[1471] Step 1:

[1472] The user utters "What's the weather like today?"

[1473] Step 2:

[1474] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[1475] Step 3:

[1476] The terminal transmits the converted text data "Tell me the weather today" to the server.

[1477] Step 4:

[1478] The server analyzes the received text data and sends a request to retrieve weather information to an internal weather database or an external API.

[1479] Step 5:

[1480] The server sends the acquired weather information (e.g., sunny) to the terminal in text format.

[1481] Step 6:

[1482] The device converts the received text data into speech using a speech synthesis engine, and notifies the user by voice, saying, "It's sunny today." The device also displays the weather information on the screen.

[1483] health care

[1484] When saying "I get tired easily these days"

[1485] Step 1:

[1486] The user vocally says, "I've been feeling tired a lot recently."

[1487] Step 2:

[1488] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[1489] Step 3:

[1490] The terminal transmits the converted text data "I've been feeling tired lately" to the server.

[1491] Step 4:

[1492] The server analyzes the received text data and extracts the keyword "easily tired."

[1493] Step 5:

[1494] The server evaluates the user's health condition based on these keywords and generates a warning message, such as "You appear tired, so please try to get plenty of rest."

[1495] Step 6:

[1496] The server sends the generated warning message to the terminal.

[1497] Step 7:

[1498] The device converts the received warning message into speech using a speech synthesis engine, and notifies the user by voice, saying, "You seem tired, so please make sure to get plenty of rest." The device also displays a message on the screen.

[1499] Emergency response

[1500] Saying "Help me"

[1501] Step 1:

[1502] The user utters the words "help me."

[1503] Step 2:

[1504] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[1505] Step 3:

[1506] The device analyzes the converted text data "help" and detects it as an emergency keyword.

[1507] Step 4:

[1508] The device sends an emergency notification to the server, which also includes the user's location information.

[1509] Step 5:

[1510] Based on the received emergency notification, the server automatically notifies pre-registered emergency contacts (e.g., family members or medical institutions).

[1511] Step 6:

[1512] The emergency contact receives the notification from the server and initiates appropriate action.

[1513] As described above, through specific processing at each step, the system of the present invention can support the lives of elderly people and respond quickly in emergencies.

[1514] Example 1

[1515] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1516] To effectively support the daily lives of the elderly, it is necessary to provide systems that are easy for users to operate. In particular, it is necessary to realize a system that uses voice recognition technology to support the daily lives of the elderly and can respond quickly in emergencies.

[1517] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1518] In this invention, the server includes a means for receiving voice input from the user, a means for converting the received voice data into digital data, a means for transmitting the converted voice data to the server, and a means for displaying the notification transmitted from the server by voice and on a screen. This makes it easy for elderly people to operate and enables a wide range of applications from assistance with daily life to emergency response.

[1519] "Users" refer to elderly people who use the system.

[1520] "Voice input" refers to the voice that the user speaks through a microphone, and includes instructions and requests for information to the system.

[1521] "Voice Data" means data that has been converted into digital form from a User's voice.

[1522] A "server" is a central processing unit that analyzes received voice data and generates responses or notifications.

[1523] A "notification" is an information message generated by a server and sent to a terminal, and is conveyed to the user by sound or on-screen display.

[1524] "Digital data" refers to data that has been converted from analog audio signals into a format that can be processed by a computer.

[1525] "Speech recognition technology" refers to the technology that converts voice into text data.

[1526] A "machine learning algorithm" is a technology that learns patterns from large amounts of data and makes predictions and classifications.

[1527] "Natural language processing (NLP)" is the technology that enables computers to understand, interpret, and generate human language.

[1528] "Changes in physical condition and lifestyle" refers to changes in the user's health condition and behavioral patterns in their daily lives.

[1529] "Emergency keyword voice" refers to voices calling for help or specific words indicating an emergency.

[1530] An "emergency notification" is a notification generated by the server and sent to emergency contacts when an emergency keyword voice is detected.

[1531] "Emergency contacts" are pre-registered contacts such as family members or medical institutions to be contacted in an emergency.

[1532] A "device" is a device that a user directly controls, receives voice input, and displays notifications.

[1533] The system of this invention is a voice recognition IoT device designed to support the lives of the elderly. It is designed so that users (elderly people) can easily use the system. The system mainly consists of three components: a terminal, a server, and a user.

[1534] Device Features

[1535] Terminals are devices used by seniors in their daily lives. They have the following functions:

[1536] Receiving voice input: Receives voice input from the user through the microphone. For example, when a user says "Tell me the weather today," the device's microphone receives this voice.

[1537] Digital conversion and transmission of voice data: The received voice is converted into digital data and sent to the server. Specifically, the voice is converted into text using a voice recognition engine (e.g., Google Speech-to-Text), and the converted data is sent to the server via the Internet.

[1538] Server Features

[1539] The server is a central processing unit that receives and analyzes the voice data sent from the terminal. It performs the following processes:

[1540] Voice data analysis: Using machine learning algorithms and natural language processing (NLP) techniques, voice data is converted into text and intent and keywords are extracted. For example, if a user says, "Tell me the weather today," the server will extract "weather" as a keyword and query a weather database to obtain weather information.

[1541] Health assessment: Analyzes changes in the user's physical condition and lifestyle to assess risk. For example, if a user says, "I've been feeling tired lately," the server will detect the keyword "fatigue" and generate a notification saying, "You appear tired, so please try to get plenty of rest."

[1542] Creating notifications: Based on the analysis results, appropriate notifications are created and sent back to the device.

[1543] User Actions

[1544] The user operates the system by issuing voice commands to the terminal. For example, if the user says "Turn on the TV," the terminal receives this voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and returns a command to the terminal. The terminal then operates the TV based on that command. A specific example of how this works is a sequence in which a voice recognition engine analyzes the command and operates an infrared remote control.

[1545] In addition, in the event of an emergency, if the user utters "help," the device will detect this as an emergency keyword and immediately send a notification to the server. The server will then automatically notify pre-set emergency contacts (family members or medical institutions). For example, if you say "help," the server will receive that information and send an emergency notification via message or call to the registered emergency contacts.

[1546] Specific examples

[1547] 1. Daily use:

[1548] The user says, "What's the weather like today?"

[1549] The terminal converts the voice into digital data and sends it to the server.

[1550] The server retrieves information such as "It's sunny today" from a weather database and sends a notification to the device.

[1551] The device will notify the user by voice and on the screen that "It's sunny today."

[1552] 2. Health management:

[1553] The user says, "I've been feeling tired a lot lately."

[1554] The server analyzes the voice and detects the keyword "fatigue."

[1555] The server generates a notification saying, "You appear tired, so please make sure to get plenty of rest," and sends it to the device.

[1556] The device will notify the user.

[1557] 3. Emergency Response:

[1558] The user says "help."

[1559] The device detects the emergency keyword and immediately sends a notification to the server.

[1560] The server will then notify pre-defined emergency contacts.

[1561] In this way, the system of the present invention is designed to be easy for the elderly to operate, and by covering a wide range of functions from assistance with daily life to responding to emergencies, it is expected to improve the quality of life of the elderly.

[1562] Prompt Sentence Examples

[1563] "Design a voice-activated IoT device to assist the elderly in their daily lives. Explain its specific functions, including the data flow between the device and the server, and provide examples."

[1564] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1565] Step 1:

[1566] The user inputs voice

[1567] The user speaks instructions and questions into the device.

[1568] Specific action: The user speaks to the device, "Tell me what the weather is like today."

[1569] Input: User's voice

[1570] Output: Audio data (analog signal)

[1571] Step 2:

[1572] The device converts the audio into digital data

[1573] The terminal uses a speech recognition engine to convert the received voice into digital data.

[1574] Specific operation: The device's microphone picks up an analog voice signal and inputs this signal into a voice recognition engine (e.g., Google Speech-to-Text).

[1575] Input: Analog audio signal

[1576] Output: Text data

[1577] Step 3:

[1578] The device sends text data to the server.

[1579] The terminal transmits the converted text data to a server via the Internet.

[1580] Specific operation: The text data generated by the device is sent to the specified API endpoint on the server.

[1581] Input: Text data

[1582] Output: Network transmission

[1583] Step 4:

[1584] The server parses the text data

[1585] The server analyzes the received text data using machine learning algorithms and natural language processing techniques.

[1586] Specific operation: After the server receives the text data, it passes the data to an NLP module (e.g., SpaCy or BERT) to extract intent and keywords.

[1587] Input: Text data

[1588] Output: Analysis results (keywords and intent)

[1589] Step 5:

[1590] The server obtains the necessary information

[1591] Based on the analysis results, the server retrieves the necessary information from external databases and APIs.

[1592] Specific operation: For example, if the analysis result contains the keyword "weather," the server will query the weather database to obtain the latest weather information.

[1593] Input: Analysis results (keywords and intent)

[1594] Output: Information from an external database

[1595] Step 6:

[1596] The server generates the notification

[1597] Based on the information obtained, the server generates the appropriate notifications.

[1598] Specific operation: Create a notification message saying "It's sunny today" based on weather information.

[1599] Input: Information retrieved from an external database

[1600] Output: Notification message

[1601] Step 7:

[1602] The server sends a notification to the device

[1603] The generated notification is sent from the server to the terminal.

[1604] Specific operation: The server sends a notification message to the terminal via the network.

[1605] Input: Notification message

[1606] Output: Network transmission

[1607] Step 8:

[1608] Your device will notify you with audio and visual notifications

[1609] The device will present the received notification to the user via audio and visual notification.

[1610] Specific operation: The device converts the notification message into voice using a speech synthesis engine (e.g., Google Text-to-Speech) and plays it over the speaker. The notification message is also displayed on the screen.

[1611] Input: Notification message

[1612] Output: Audio notification and screen display

[1613] The above are the main processing steps of this system. The specific operations and data processing performed at each step have been explained in detail.

[1614] (Application example 1)

[1615] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1616] In factory work sites, many workers rely on manual and visual confirmation to execute complex instructions, leaving room for improvement in terms of efficiency and safety. Rapid response in emergencies is also required, but due to the nature of the work, it is often difficult to immediately detect and respond to emergencies. Therefore, there is a need for a system that improves the efficiency of factory work, ensures safety, and enables rapid response in emergencies.

[1617] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1618] In this invention, the server includes means for receiving voice input from a user, means for transmitting the received voice data to the server, means for displaying the notification transmitted from the server by voice and on a screen, means for controlling the robot based on the received voice data, and means for detecting the voice data from the user in an emergency and generating and transmitting a corresponding notification, thereby improving work efficiency in the factory and enabling a rapid response in an emergency.

[1619] A "user" is a human user who operates a system.

[1620] "Voice input" refers to instructions or information provided by a user to a system through voice.

[1621] A "server" is a central processing unit that receives and analyzes the voice data and generates appropriate notifications and control signals.

[1622] "Notifications" are messages generated by the server to inform users of analysis results or important information.

[1623] A "robot" is a mechanical device that performs physical tasks based on instructions received from an automated control system connected to the system.

[1624] "Emergency" refers to a series of conditions or events that cannot be handled through normal operations or tasks and require immediate action.

[1625] "Keyword voice" is a specific word or phrase that is preset to indicate an emergency or a specific command.

[1626] "Analysis" is the process by which the server analyzes the received voice data, interprets the meaning, and determines the appropriate action.

[1627] "Control" refers to the system's ability to operate robots and other devices based on voice commands.

[1628] An "emergency notification" is a special message generated by the server to notify you of an emergency problem.

[1629] This invention is a system that uses a voice recognition IoT device to improve efficiency and safety at factory work sites. The detailed configuration of this system and its operating principle will be described below.

[1630] System configuration

[1631] The system consists of three main components:

[1632] 1. Terminal

[1633] 2. Server

[1634] 3. Users

[1635] Device Features

[1636] The device receives voice input from the user and converts the voice into digital data, which is then sent to a server via the Internet. It also receives notifications sent from the server and notifies the user via voice and on a screen.

[1637] Server Features

[1638] The server receives and analyzes the voice data sent from the device. This analysis uses machine learning algorithms and natural language processing (NLP) technology. The server analyzes the voice instructions, extracts appropriate commands, and generates work instructions and emergency response notifications. These notifications are then sent back to the device and displayed to the user via voice and on the screen.

[1639] Robot Control

[1640] The robot performs specific tasks based on instructions from the server. This control includes linking with IoT devices, and the robot operates according to received voice instructions. This allows users to give voice instructions to the robot and efficiently carry out tasks.

[1641] User Actions

[1642] Users can give voice commands to the device. For example, if they say "start the machine," the voice is converted into digital data on the device, which is then analyzed by the server and sent to the robot. Also, if an emergency keyword such as "help" is spoken, the system will immediately initiate an emergency response and notify emergency contacts.

[1643] Hardware and software used

[1644] The following hardware and software are used to realize this system.

[1645] Microphone and speaker: A high-sensitivity microphone receives audio input and a speaker provides audio output.

[1646] Robot control device: Uses control hardware for IOT devices.

[1647] Natural Language Processing and Speech Recognition Module: Python module 'speech_recognition' is used to convert voice data to text, and 'text_to_speech' is used to output speech. In addition, 'iot_device_control' module is used to control IOT devices.

[1648] Specific examples

[1649] 1. Daily tasks:

[1650] "Start the machine": The terminal sends the voice to the server, which analyzes it and sends instructions to the robot to start working.

[1651] "Stop the machine": Similarly, the voice command from the terminal is analyzed by the server and a command to stop is sent to the robot.

[1652] 2. Emergency Response:

[1653] "Help me": This voice command is detected by the device and sent to the server, which then generates an emergency notification and sends it to your emergency contacts.

[1654] Prompt Sentence Examples

[1655] Create a voice assistant application for a factory robot that will assist me. Features include:

[1656] 1. Receive user instructions through voice recognition.

[1657] 2. Operate IOT devices based on instructions (start, stop).

[1658] 3. Send emergency notifications with the command "Help" in case of an emergency.

[1659] This system can significantly improve work efficiency and safety within the factory.

[1660] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1661] Step 1:

[1662] Audio reception and conversion

[1663] The device receives voice input from the user. This voice data (input) is received by the microphone and sent to the speech recognition module. This module converts the voice data into text and generates digital data (output). Specifically, the 'speech_recognition' module is used to analyze the voice and convert it into a string of characters.

[1664] Step 2:

[1665] Data transmission

[1666] The device sends the text data generated in step 1 to the server. This data transmission (input) is performed via the Internet. Specifically, an HTTP request is used to send the data to the server's API.

[1667] Step 3:

[1668] Data analysis

[1669] The server analyzes the received text data (input) and understands the user's instructions. This analysis uses machine learning algorithms and natural language processing (NLP). The analysis results (output) are generated as instruction commands. Specifically, the server uses an NLP module to extract keywords and determine the appropriate action.

[1670] Step 4:

[1671] Notification Generation and Delivery

[1672] The server generates an appropriate notification based on the analysis results of step 3 and sends it to the device. This notification (input) is generated as data for voice and screen display. Specific operations include using the 'text_to_speech' module, which converts text data into voice data.

[1673] Step 5:

[1674] Robot Control

[1675] Based on the analysis results of step 3, the server generates a signal to control the IOT device and sends it to the robot. This control signal (input) is sent using a protocol for controlling IOT devices. Specifically, the 'iot_device_control' module generates a command to operate the robot and sends it to the device.

[1676] Step 6:

[1677] Receiving and displaying notifications

[1678] The device receives notifications sent from the server and notifies the user via voice and on the screen. This notification (input) includes audio data played on the device and text data displayed on the screen. Specifically, the 'text_to_speech' module generates audio and renders data for display on the screen.

[1679] Step 7:

[1680] Emergency response

[1681] When a user utters an emergency keyword, the device detects it and immediately sends an emergency notification to the server. This emergency notification (input) is analyzed by the server and a notification (output) is generated to be sent to emergency contacts. Specifically, when a specific keyword is detected, a function to send an email or SMS to emergency contacts is triggered.

[1682] In this way, each processing step begins with the user's voice input, and ultimately leads to robot control and emergency response, aiming to improve work efficiency and safety within the factory.

[1683] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1684] The system of the present invention provides a voice recognition IoT device combined with an emotion engine to more actively support the lives of the elderly. The system is designed to be easy for users (elderly people) to use and consists of three components: a terminal, a server, and a user. Each component and its function are explained in detail below.

[1685] Device Features

[1686] The terminal is a device used by elderly people in their daily lives, and is mainly used to receive voice input, send voice data, and receive and display notifications. The terminal receives voice input from the user through a microphone and converts it into digital data using a voice recognition engine and an emotion engine, while also identifying the user's emotions. This digital data and emotion data are sent to a server via the Internet.

[1687] Server Features

[1688] The server is a central processing unit that receives and analyzes the voice and emotion data sent from the device. The server uses machine learning algorithms and natural language processing (NLP) techniques to analyze the data and detect changes in physical condition and lifestyle habits. Furthermore, by analyzing the emotion data, the server can assess the user's stress and anxiety levels.

[1689] For example, if a user says, "I've been feeling tired lately," but speaks in an anxious tone, the server analyzes the user's emotions along with the voice data and generates an appropriate notification. In this case, the server generates a notification that advises the user to relax and, if necessary, instructs the user to consult a medical professional. The generated notification is then sent back to the device and notified to the user by voice and on-screen display.

[1690] User Actions

[1691] Users operate the system by giving voice commands to the device. For example, if they say "Turn on the TV," the device receives this voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and sends it back to the device. The device then controls the TV based on that command.

[1692] In addition, if the user utters a keyword such as "help" in an emergency, the device detects this and immediately sends an emergency notification to the server, which then notifies pre-registered emergency contacts (family members or medical institutions).

[1693] Specific examples

[1694] 1. Daily use:

[1695] When a user says, "Tell me the weather today," the device converts the voice into text and sends it to the server. The server obtains the weather information and sends it to the device. The device then announces, "It's sunny today," by voice and also displays it on the screen.

[1696] 2. Health management:

[1697] If a user says, "I get tired easily recently," but their voice is unsteady, the server will recognize the keywords "tired easily" and "anxiety" and generate a notification offering advice on how to relax. The device will then notify the user by voice and on-screen display.

[1698] 3. Emergency Response:

[1699] When a user shouts "help," the device detects this as an emergency keyword and sends an emergency notification to the server, which then notifies emergency contacts.

[1700] 4. Emotional Continuity Management:

[1701] The server stores emotional data extracted from daily conversations and periodically analyzes the user's emotional patterns. Based on the results of this analysis, the system provides individually customized relaxation suggestions and health-related advice during times of high stress.

[1702] As described above, the system of the present invention uses an emotion engine to support the lives of the elderly in a more personalized way, enabling early detection of health risks and rapid response in emergencies.

[1703] The processing flow will be explained below.

[1704] health care

[1705] When you say "I've been feeling tired lately" and it's accompanied by emotion recognition

[1706] Step 1:

[1707] The user vocally says, "I've been feeling tired a lot recently."

[1708] Step 2:

[1709] The device receives the user's voice through a microphone and converts it into text data using a voice recognition engine, while also analyzing the tone and pitch of the user's voice using an emotion engine to generate emotion data.

[1710] Step 3:

[1711] The device sends the converted text data "I've been feeling tired lately" and the analyzed emotional data (e.g., anxiety) to the server.

[1712] Step 4:

[1713] The server analyzes the received text data, extracts health-related keywords such as "getting tired easily," and analyzes the emotion data to determine that the user is feeling anxious.

[1714] Step 5:

[1715] Based on this information, the server will perform a risk assessment of the user's health status and emotional data, and generate an appropriate notification message, such as "You seem to be feeling tired and anxious recently. Please get plenty of rest and consult a medical professional if necessary."

[1716] Step 6:

[1717] The server sends the generated notification message to the terminal.

[1718] Step 7:

[1719] The device converts the received notification message into speech using a speech synthesis engine, and notifies the user aloud, saying, "You seem to have become tired and anxious recently. Please get plenty of rest and consult a medical professional if necessary." The same message is also displayed on the screen.

[1720] Emergency response

[1721] When you say "Help me"

[1722] Step 1:

[1723] The user utters the words "help me."

[1724] Step 2:

[1725] The device receives the user's voice through a microphone, converts it into text data using a speech recognition engine, and analyzes the urgency of the user's voice using an emotion engine.

[1726] Step 3:

[1727] The device sends an emergency notification to the server, including the converted text data "Help me" and the analyzed emotion data (e.g., extreme stress).

[1728] Step 4:

[1729] The server analyzes the received emergency notification and detects the emergency keyword "help."

[1730] Step 5:

[1731] The server will promptly notify pre-registered emergency contacts (e.g., family members or medical institutions), including the user's location information.

[1732] Step 6:

[1733] The emergency contact receives the notification from the server and initiates appropriate action.

[1734] Daily use

[1735] When saying "What's the weather today?"

[1736] Step 1:

[1737] The user utters "What's the weather like today?"

[1738] Step 2:

[1739] The terminal receives the user's voice through a microphone and converts it into text data using a voice recognition engine.

[1740] Step 3:

[1741] The terminal transmits the converted text data "Tell me the weather today" to the server.

[1742] Step 4:

[1743] The server analyzes the received text data and sends a request to retrieve weather information to an internal weather database or an external API.

[1744] Step 5:

[1745] The server sends the acquired weather information (e.g., sunny) to the terminal in text format.

[1746] Step 6:

[1747] The device converts the received weather information into voice using a voice synthesis engine, notifies the user by voice, such as "It's sunny today," and also displays the weather information on the screen.

[1748] Emotional Continuity Management

[1749] Analysis of daily conversation data

[1750] Step 1:

[1751] The terminal receives voice input from the user on a daily basis, converts it into text data using a voice recognition engine, and simultaneously generates emotion data using an emotion engine.

[1752] Step 2:

[1753] The terminal transmits the converted text data and emotion data to the server.

[1754] Step 3:

[1755] The server stores the voice data and emotion data accumulated daily in a database.

[1756] Step 4:

[1757] The server periodically analyzes the accumulated data and evaluates the user's emotional patterns.

[1758] Step 5:

[1759] Based on the analysis results, the server generates notification messages to provide personalized health support to the user. For example, if the user's long-term stress level is high, the server generates a message recommending relaxation techniques or counseling.

[1760] Step 6:

[1761] The server sends the generated notification message to the terminal.

[1762] Step 7:

[1763] The device converts the received notification message into voice using a speech synthesis engine and notifies the user. The device also displays the same message on the screen.

[1764] By combining this system with an emotion engine, it will further personalize support for the elderly, enabling more effective health management and faster emergency response.

[1765] Example 2

[1766] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1767] In recent years, there has been a growing need for life support systems for the elderly, but conventional systems simply receive voice input and notify users, without taking into account their emotions or changes in their physical condition. This makes it difficult to provide personalized support to users, and there are problems with being unable to respond quickly and appropriately, especially in emergencies.

[1768] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1769] In this invention, the server includes means for analyzing the user's voice data and emotion data to detect changes in physical condition and lifestyle habits, means for generating a notification based on the detected changes, and means for transmitting the generated notification to the terminal. This enables personalized notifications that take into account the user's emotions and changes in physical condition, enabling a quick and appropriate response in an emergency.

[1770] "Voice input" means spoken information such as instructions or questions spoken by a user.

[1771] "Device" means any equipment or hardware used to receive, analyze, or transmit audio input.

[1772] "Text data" refers to character string data obtained by analyzing voice input.

[1773] "Emotional Data" means data used to identify a user's emotional state through voice analysis.

[1774] "Server" refers to a central processing unit that receives voice and emotion data, analyzes it, and generates notifications.

[1775] "Notification" means information or instructions provided to the user, whether through audio or visual display.

[1776] An "emergency" is defined as the moment a user reports a dangerous situation or health-related issue.

[1777] "Terminal" refers to a device that a user operates and uses to input voice information.

[1778] "Personalization" means providing information and assistance in a way that is adapted based on the user's individual situation and emotional state.

[1779] A "machine learning algorithm" refers to a computational method for learning from data and making predictions and analyses.

[1780] The system of the present invention provides an IoT device that combines an emotion engine and voice recognition technology to more actively support the lives of users (elderly people). This system is designed to be easy for users to use and consists of three components: a terminal, a server, and a user. The following describes in detail the modes for implementing the present invention.

[1781] Device Features

[1782] A device is a device that users use in their daily lives, and it mainly receives voice input, processes voice data, and receives and displays notifications. Specifically, a device has the following functions:

[1783] The user's voice is received by a microphone and converted into digital data using a voice recognition engine.

[1784] Analyze user emotions from voice using an emotion engine.

[1785] The converted voice data and emotion data are transmitted to a server.

[1786] Notify the user of notifications sent from the server via audio and visual.

[1787] Server Features

[1788] The server is a central processing unit that receives and analyzes the voice and emotion data sent from the device. Specifically, the server has the following functions:

[1789] The received data is analyzed using machine learning algorithms and natural language processing (NLP) to detect changes in the user's physical condition and lifestyle.

[1790] Evaluate users' stress and anxiety levels based on emotional data.

[1791] Based on the analysis results, an appropriate notification is generated and sent to the device.

[1792] User Actions

[1793] Users operate the system by issuing voice commands to the device. For example, if a command is given to "turn on the TV," the device receives the voice and sends it to the server. The server analyzes the voice data, recognizes the appropriate command, and sends it back to the device, which then operates the TV based on that command. Also, if a user utters "help" in an emergency, the device detects this and immediately sends an emergency notification to the server. The server receives this emergency notification and notifies pre-registered emergency contacts.

[1794] Specific use cases

[1795] Below are some specific examples of how this system can be used.

[1796] 1. Daily use

[1797] When a user says, "Tell me the weather today," the device converts the voice into text and sends it to the server. The server obtains the weather information and sends it to the device. The device then announces, "It's sunny today," by voice and also displays it on the screen.

[1798] 2. Health management

[1799] If a user says, "I get tired easily recently," but their voice is unsteady, the device recognizes this and sends it to the server. The server then recognizes the keywords "tired easily" and "anxiety" and generates a notification offering advice on how to relax. The device then notifies the user with the advice via voice and on the screen.

[1800] 3. Emergency Response

[1801] When a user shouts "help," the device detects this as an emergency keyword and sends an emergency notification to the server, which then notifies emergency contacts.

[1802] 4. Emotional Continuity Management

[1803] The server stores emotional data extracted from daily conversations and periodically analyzes the user's emotional patterns. Based on the results of this analysis, the system provides individually customized relaxation suggestions and health-related advice during times of high stress.

[1804] Example prompts for generative AI models

[1805] Below are some example prompts to input to the generative AI model:

[1806] Example prompt 1:

[1807] If a user says "I've been feeling tired lately" to a device by voice and is emotionally unstable, explain how the server generates appropriate advice and notifies the device.

[1808] Example prompt 2:

[1809] Explain how the device and server work together to notify emergency contacts when a user calls for help in an emergency.

[1810] As described above, the system of the present invention uses an emotion engine to support the lives of elderly people in a personalized way, enabling early detection of health risks and rapid response in emergencies.

[1811] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1812] Step 1:

[1813] The user provides voice input.

[1814] Specific operation: The user speaks to the device, saying, "Tell me the weather today."

[1815] Input: User's voice

[1816] Output: Audio signal

[1817] Step 2:

[1818] The device collects and processes the voice data.

[1819] Specific operation: The device uses a built-in microphone to collect voice signals, which are then converted into text data by a voice recognition engine. At the same time, the emotion engine analyzes the user's emotion data from the voice.

[1820] Input: Audio signal

[1821] Output: Text data, emotion data

[1822] Step 3:

[1823] The device transmits emotion data and voice data to the server.

[1824] Specific operation: The device assembles the generated text data "Tell me today's weather" and emotion data into a packet and sends it to a server via the Internet.

[1825] Input: Text data, emotion data

[1826] Output: Data packets

[1827] Step 4:

[1828] The server receives and processes the data.

[1829] How it works: The server decodes data packets received via the internet, extracts text and emotion data, and uses NLP techniques to analyze the user's intent and evaluates voice data to understand their emotions.

[1830] Input: Data packet

[1831] Output: User's intention, emotion evaluation results

[1832] Step 5:

[1833] The server generates the analysis results and generates the necessary notifications.

[1834] Specific operation: Based on the user's intent to "check the weather," the server retrieves the latest weather information and creates a notification saying "It's sunny today." If the emotion is "anxiety," the server also generates a notification about relaxation methods, if necessary.

[1835] Input: User's intention, emotion evaluation results

[1836] Output: Notification data

[1837] Step 6:

[1838] The server transmits the notification data to the terminal.

[1839] Specific operation: The server sends notification data "It's sunny today" to the terminal via the Internet.

[1840] Input: Notification data

[1841] Output: Data packets

[1842] Step 7:

[1843] The terminal provides a notification to the user.

[1844] Specific behavior: The device analyzes the received notification data, notifies the user by voice, saying "It's sunny today," and displays weather information on the screen. If additional notifications based on emotions are needed, they will also be displayed.

[1845] Input: Notification data

[1846] Output: Audio notification, screen display

[1847] Through the above processing steps, the system provides personalized responses to the user's needs and realizes advanced support based on changes in emotions and physical condition.

[1848] (Application example 2)

[1849] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1850] When elderly people use self-driving vehicles, there is a problem that there is insufficient support to grasp changes in their physical condition and emotions in real time and to ensure safe and comfortable use. In addition, simply recognizing voice commands tends to delay response in emergency situations involving elderly people. Therefore, it is necessary to establish a system that analyzes the emotional state of elderly drivers and provides appropriate advice and warnings.

[1851] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1852] In this invention, the server includes means for receiving voice input, means for analyzing the received voice data and emotion data, and means for generating appropriate notifications based on the analysis results, thereby improving safety and comfort for elderly drivers.

[1853] A "means for receiving audio input" is a device that captures the user's spoken voice and converts it into digital data.

[1854] The "means for transmitting voice data to a server" is a network communication function for sending the received voice data to a server via the Internet.

[1855] "Means for displaying notifications by voice and on screen" refers to a function that displays notifications sent from the server to the user by voice or on a display.

[1856] "Means for identifying emotional data" refers to a function that analyzes a user's emotional state from the user's voice or other input.

[1857] The "means for transmitting emotion data to a server" is a network communication function for transmitting the identified emotion data to a server.

[1858] The "means for generating advice or warning based on the analysis results" is a function for generating appropriate advice or warning based on the analysis results obtained by the server.

[1859] "Means for notifying advice and warnings" refers to a function that notifies the user of generated advice and warnings by voice or display.

[1860] "Means for detecting changes in physical condition and lifestyle habits" refers to a function that detects changes in the user's physical condition and lifestyle habits by analyzing voice data and emotional data.

[1861] "Means for detecting emergency keyword voice" is a function that recognizes emergency keywords uttered by the user in real time.

[1862] The "means for notifying the server of voice in an emergency" is a function that immediately transmits voice data detected in an emergency to the server.

[1863] "Means for generating and notifying appropriate measures in an emergency" is a function that creates appropriate measures in an emergency from the results of analysis on the server and notifies the user of them.

[1864] The system of the present invention is a system for assisting elderly drivers in an autonomous vehicle, and includes the following components: a voice recognition device, an emotion analysis engine, a server, and a user interface.

[1865] Hardware and Software Configuration

[1866] 1. Voice Recognition Device

[1867] A microphone is installed inside the vehicle to receive the user's voice input.

[1868] Convert speech to text data using a speech recognition engine (e.g., speech_recognition library).

[1869] 2. Sentiment Analysis Engine

[1870] Use an emotion analysis engine (e.g., emotion_recognition library) to identify emotions from user voice data.

[1871] 3. Server

[1872] Voice data and emotion data are received and analyzed.

[1873] Data is analyzed using machine learning algorithms and natural language processing (NLP) techniques.

[1874] It detects changes in the user's physical condition and lifestyle and generates appropriate advice and warnings.

[1875] 4. User Interface

[1876] The system has the function of notifying the user of the analysis results by voice and on-screen display, allowing the user to receive instructions in real time while in the vehicle.

[1877] Program processing

[1878] The server receives the voice data transmitted from the voice recognition device.

[1879] The emotion analysis engine identifies emotions from the voice data and sends them to the server.

[1880] The server analyzes the received voice and emotional data to detect changes in the user's physical condition and lifestyle.

[1881] Based on the detected changes, the server generates appropriate advice and warnings and notifies the user via the user interface.

[1882] Specific examples

[1883] For example, if a user says "I'm tired" while driving, the voice recognition device converts this speech into text data, and the emotion analysis engine identifies the emotional state "tired." The server analyzes this data and confirms that the user is tired. The server then generates advice such as "We recommend you stop driving and take a break," and notifies the user by voice and on-screen display.

[1884] Prompt Sentence Examples

[1885] You can use the following prompts to leverage specific generative AI models:

[1886] Build a voice recognition app for driving assistance for the elderly. The app recognizes voice input, analyzes the user's emotions, and provides appropriate advice and warnings while driving. For example, if a user says "I'm tired," the app will recognize it, analyze the emotion data, and notify the user, "We recommend that you stop driving and take a break." The libraries used are speech_recognition and emotion_recognition.

[1887] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1888] Step 1:

[1889] The device receives the user's voice input through a microphone, which is converted into digital data by a speech recognition engine.

[1890] Step 2:

[1891] The device sends the converted voice data to an emotion analysis engine, which identifies the user's emotion from the voice data. In this process, an emotion analysis model is used to analyze the voice tone and content to generate emotion data.

[1892] Step 3:

[1893] The terminal transmits the generated voice data and emotion data to the server, and the transmitted data includes the text content of the voice and the emotion data.

[1894] Step 4:

[1895] The server analyzes the received voice and emotion data using machine learning algorithms and natural language processing (NLP) techniques, allowing it to detect changes in the user's physical condition and lifestyle.

[1896] Step 5:

[1897] The server generates appropriate advice and warnings based on the analysis results. For example, if the user's stress level is high, it will generate a notification such as "We recommend that you stop driving and take a break."

[1898] Step 6:

[1899] The server sends the generated advice or warning to the terminal, and the notification includes the specific advice and the reason for it.

[1900] Step 7:

[1901] The device will notify the user of the notifications it receives through audio and visual displays, which are updated in real time to provide the user with information to take appropriate action.

[1902] Step 8:

[1903] Users can take appropriate action based on the notification, such as stopping driving and taking a break if they feel tired or stressed.

[1904] The above processing steps provide appropriate advice and warnings according to changes in the user's physical condition and emotions, thereby improving the safety and comfort of elderly drivers.

[1905] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1906] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1907] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1908] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1909] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1910] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1911] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1912] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1913] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1914] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1915] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1916] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1917] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1918] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1919] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1920] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1921] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1922] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1923] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1924] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1925] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1926] The following is further disclosed regarding the above embodiment.

[1927] (Claim 1)

[1928] means for receiving voice input from a user;

[1929] means for transmitting the received voice data to a server;

[1930] means for audibly and visually displaying notifications sent from the server;

[1931] A system including:

[1932] (Claim 2)

[1933] A means of analyzing the user's voice data to detect changes in their physical condition and lifestyle,

[1934] means for generating a notification based on the detected change;

[1935] means for transmitting the generated notification to the terminal;

[1936] 10. The system of claim 1, comprising:

[1937] (Claim 3)

[1938] A means for detecting an emergency keyword voice;

[1939] a means for notifying a server of the detected emergency sound;

[1940] 10. The system of claim 1, comprising:

[1941] "Example 1"

[1942] (Claim 1)

[1943] means for receiving voice input from a user;

[1944] means for converting received audio data into digital data;

[1945] means for transmitting the converted voice data to a server;

[1946] means for audibly and visually displaying notifications sent from the server;

[1947] A system including:

[1948] (Claim 2)

[1949] A means of analyzing user voice data using machine learning algorithms and natural language processing technology,

[1950] A means of detecting changes in physical condition and lifestyle habits based on the analysis results,

[1951] means for generating a notification based on the detected change;

[1952] means for transmitting the generated notification to the terminal;

[1953] 10. The system of claim 1, comprising:

[1954] (Claim 3)

[1955] A means for detecting an emergency keyword voice;

[1956] means for notifying a server of the detected emergency voice data;

[1957] a means by which the server notifies an emergency contact;

[1958] 10. The system of claim 1, comprising:

[1959] "Application Example 1"

[1960] (Claim 1)

[1961] means for receiving voice input from a user;

[1962] means for transmitting the received voice data to a server;

[1963] means for audibly and visually displaying notifications sent from the server;

[1964] means for controlling a robot based on the received voice data;

[1965] means for detecting voice data from a user in an emergency and generating and transmitting a corresponding notification;

[1966] A system including:

[1967] (Claim 2)

[1968] A means of analyzing the user's voice data and detecting changes in physical condition and work situation;

[1969] means for generating a notification based on the detected change;

[1970] means for transmitting the generated notification to the terminal;

[1971] A means for controlling a robot that performs appropriate actions based on the work instructions;

[1972] 10. The system of claim 1, comprising:

[1973] (Claim 3)

[1974] A means for detecting an emergency keyword voice;

[1975] a means for notifying a server of the detected emergency sound;

[1976] a means for automatically notifying emergency contacts in the event of an emergency;

[1977] 10. The system of claim 1, comprising:

[1978] "Example 2: Combining Emotion Engines"

[1979] (Claim 1)

[1980] a device for receiving voice input from a user;

[1981] a device for analyzing received voice data and converting it into text data and emotion data;

[1982] a device for transmitting the converted text data and emotion data to a server;

[1983] a device for audibly and visually displaying notifications sent from the server;

[1984] A system including:

[1985] (Claim 2)

[1986] A means of analyzing the user's voice data and emotional data to detect changes in their physical condition and lifestyle habits;

[1987] means for generating a notification based on the detected change;

[1988] means for transmitting the generated notification to the terminal;

[1989] 10. The system of claim 1, comprising:

[1990] (Claim 3)

[1991] A means for detecting an emergency keyword voice;

[1992] a means for notifying a server of the detected emergency sound;

[1993] a means for sending notifications to emergency contacts;

[1994] 10. The system of claim 1, comprising:

[1995] "Application example 2 when combining emotion engines"

[1996] (Claim 1)

[1997] means for receiving voice input from a user;

[1998] means for transmitting the received voice data to a server;

[1999] means for audibly and visually displaying notifications sent from the server;

[2000] a means for identifying user emotional data;

[2001] means for transmitting the identified emotion data to a server;

[2002] A means for the server to generate appropriate advice or warnings based on the analysis results; and

[2003] a means of notifying the user of any advice or warnings generated;

[2004] A system including:

[2005] (Claim 2)

[2006] A means of analyzing the user's voice data and emotional data to detect changes in their physical condition and lifestyle habits;

[2007] means for generating a notification based on the detected change;

[2008] means for transmitting the generated notification to the terminal;

[2009] A means of notifying the user of the changes in the form of appropriate advice or warnings while driving;

[2010] 10. The system of claim 1, comprising:

[2011] (Claim 3)

[2012] A means for detecting an emergency keyword voice;

[2013] a means for notifying a server of the detected emergency sound;

[2014] A means for generating appropriate measures in an emergency based on the analysis results and notifying users;

[2015] 10. The system of claim 1, comprising: [Explanation of symbols]

[2016] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving voice input from a user; means for transmitting the received voice data to a server; means for audibly and visually displaying notifications sent from the server; A system including:

2. A means of analyzing the user's voice data to detect changes in their physical condition and lifestyle, means for generating a notification based on the detected change; means for transmitting the generated notification to the terminal; The system of claim 1 , comprising:

3. A means for detecting an emergency keyword voice; a means for notifying a server of the detected emergency sound; The system of claim 1 , comprising:

Images