system

Abstract

Provide a system. [Solution] an acquisition means for acquiring a voice command of a user; a conversion means for converting the acquired voice command into voice data; a transmitting means for transmitting the converted voice data to a server; an analysis means for analyzing the voice data on the server side and obtaining corresponding information; A synthesis means for converting the acquired information into speech on the server side; output means for outputting the synthesized voice data to a user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] While current voice-recognition IoT devices can process basic user voice commands, they require additional functionality to meet diverse needs, such as providing local information, managing one's health, and responding to emergencies. In particular, there are few systems on the market that can manage a user's health or respond immediately to emergencies. This calls for the development of new IoT devices that comprehensively support users' daily lives and provide a sense of security. [Means for solving the problem]

[0005] The present invention includes an acquisition means, a conversion means, and a transmission means for acquiring a user's voice command, converting it into voice data, and transmitting it to a server. It also includes an analysis means and a synthesis means for analyzing the voice data on the server side, acquiring the analyzed information, and converting it into voice. Furthermore, it provides an output means for outputting the acquired voice data to the user, thereby interacting with the user. This makes it possible to meet a variety of needs, such as providing weather information, supporting health management, and reporting emergencies. Furthermore, by adding a support means for supporting health management using information acquired on the server side and a reporting means for making immediate reports in emergencies, it is possible to improve the quality of the user's daily life and provide a sense of security.

[0006] "User" refers to a person who utilizes the system to input voice commands.

[0007] "Voice command" refers to an instruction given by a user to a system using voice.

[0008] "Acquisition means" refers to hardware and software functions for acquiring a user's voice command.

[0009] The "conversion means" refers to a function for converting the voice command acquired by the acquisition means into voice data.

[0010] The "transmitting means" refers to a function for transmitting the audio data converted by the converting means to the server.

[0011] "Server" refers to the back-end computer system that analyzes voice data and retrieves and generates corresponding information.

[0012] "Analysis means" refers to a function for analyzing voice data on the server and generating appropriate information based on the content.

[0013] The "synthesis means" refers to a function for converting the information generated by the analysis means into voice data.

[0014] The "output means" refers to a function for reproducing the voice data generated by the synthesis means to the user.

[0015] "Support means" refers to a function for supporting the user's health management using information acquired on the server side.

[0016] "Notification means" refers to a function for quickly recognizing a user's voice command in an emergency and immediately making an emergency call. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention relates to a voice recognition IoT device and system that analyzes a user's voice commands and realizes functions such as providing information, managing health, and making emergency calls. This system includes an edge terminal and a back-end server that execute a series of processes from receiving a user's voice command to analyzing it, providing information, and responding to an emergency.

[0039] System Configuration

[0040] 1. Terminal (edge ​​device) side

[0041] Microphone and voice input module: to capture the user's voice commands.

[0042] Voice data processing module: converts voice data into digital format and sends it to the server.

[0043] Communication module: Sends voice data to the server and receives responses from the server.

[0044] Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[0045] 2. Server (backend) side

[0046] Speech recognition engine: Converts voice data sent by the user into text data.

[0047] Natural Language Processing (NLP) module: Analyzes text data and understands user requirements.

[0048] Information provision engine: Obtains information based on user requests and generates the required response.

[0049] Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[0050] Database: Stores and manages user health data and emergency contact information.

[0051] Program processing and specific examples

[0052] Voice input recognition and information provision

[0053] A user speaks to the terminal, "What's the weather like today?" This voice command is picked up by the microphone and digitized by the voice data processing module. This digital voice data is sent to the server through the communication module.

[0054] The server uses a speech recognition engine to convert the received voice data into text data. Next, the NLP module analyzes the text data and understands the user's request, "I want weather information." The server uses the information provision engine to call the weather information API and obtain the required weather information. The speech synthesis engine then converts the weather information into voice data, generating the message, "Today's weather is sunny." This voice data is sent to the device and is conveyed to the user through the device's speaker.

[0055] Support for health management

[0056] The user says, "Record my blood pressure today." The voice command is captured by the device and sent to the server. Similarly, the voice data is analyzed by the speech recognition engine and NLP module, and the request "record blood pressure data" is understood.

[0057] The server accesses the database and records the user's blood pressure data. Furthermore, it monitors the user's health condition as needed and generates a warning message if an abnormality is detected. The generated information is converted into voice data through a voice synthesis engine and notified to the user via the device.

[0058] Emergency response

[0059] The user shouts "An emergency has occurred." The voice command is immediately captured by the device and sent to the server as emergency data. The server uses a speech recognition engine to convert it into text data, and an NLP module analyzes it to indicate an emergency.

[0060] The server retrieves emergency contact information from the database and notifies the designated emergency contacts using an automated notification system. At the same time, the server generates feedback information such as "An emergency call has been made" and converts it into voice data using a voice synthesis engine. This voice data is sent to the device and is played back to the user through the device's speaker.

[0061] In this way, a comprehensive system of voice-recognition IoT devices to support users' daily lives is realized.

[0062] The processing flow will be explained below.

[0063] Voice input recognition and information provision

[0064] Step 1:

[0065] The user speaks the voice command "What's the weather like today?" into the device.

[0066] Step 2:

[0067] The device uses a built-in microphone to record voice commands.

[0068] Step 3:

[0069] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[0070] Step 4:

[0071] The terminal transmits the converted voice data to a server via the Internet.

[0072] Step 5:

[0073] The server's voice recognition engine converts the received voice data into text data.

[0074] Step 6:

[0075] The server's natural language processing (NLP) module analyzes the text data "What's the weather like today?" and understands the user's request.

[0076] Step 7:

[0077] The server calls the weather information API based on the analysis results and obtains current weather information.

[0078] Step 8:

[0079] The server's information engine generates a text message saying "Today's weather is sunny" based on the acquired weather information.

[0080] Step 9:

[0081] A speech synthesis engine on the server converts the generated text message into voice data.

[0082] Step 10:

[0083] The server then transmits the converted voice data to the terminal again via the Internet.

[0084] Step 11:

[0085] The audio output module of the terminal plays the received data over the speaker and tells the user aloud, "Today's weather is sunny."

[0086] Support for health management

[0087] Step 1:

[0088] The user speaks to the terminal, saying, "Record my blood pressure today."

[0089] Step 2:

[0090] The device will record your voice commands using the built-in microphone.

[0091] Step 3:

[0092] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[0093] Step 4:

[0094] The terminal transmits the converted voice data to a server via the Internet.

[0095] Step 5:

[0096] The server's voice recognition engine converts the received voice data into text data.

[0097] Step 6:

[0098] The server's natural language processing (NLP) module analyzes the text data "Record my blood pressure today" and understands the user's request.

[0099] Step 7:

[0100] The server accesses the user's physical condition database and records the blood pressure data.

[0101] Step 8:

[0102] The server generates a feedback message indicating that the recording was successful, and converts it into voice data using a voice synthesis engine.

[0103] Step 9:

[0104] The server then transmits the converted voice data to the terminal again via the Internet.

[0105] Step 10:

[0106] The device's audio output module plays the received data over the speaker and tells the user in a voice message, "Your blood pressure has been recorded. Take care."

[0107] Emergency response

[0108] Step 1:

[0109] The user shouts into the terminal, "An emergency has occurred."

[0110] Step 2:

[0111] The device will record emergency voice commands using the built-in microphone.

[0112] Step 3:

[0113] The voice data processing module of the terminal converts the recorded voice into digital data and passes it to the emergency processing module.

[0114] Step 4:

[0115] The terminal immediately transmits the formatted emergency voice data to the server.

[0116] Step 5:

[0117] The server's voice recognition engine converts the received emergency voice data into text data.

[0118] Step 6:

[0119] The server's natural language processing (NLP) module analyzes the text data "An emergency has occurred" and understands that it is an emergency.

[0120] Step 7:

[0121] A server retrieves emergency contact information from a database and sends emergency messages to the contacts using an automated notification system.

[0122] Step 8:

[0123] The server checks the status of the call transmission, generates a feedback message such as "An emergency call has been made. Please wait until the ambulance arrives," and converts it into voice data using a speech synthesis engine.

[0124] Step 9:

[0125] The server then transmits the converted voice data to the terminal again via the Internet.

[0126] Step 10:

[0127] The terminal's audio output module plays the received data over the speaker and tells the user in a voice message, "An emergency call has been made. Please wait until the ambulance arrives."

[0128] Example 1

[0129] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] Conventional speech recognition systems have had difficulty responding quickly and accurately to diverse user requests. Furthermore, there has been a lack of technology to accurately analyze user requests and provide appropriate information, particularly in the areas of health management and emergency response. This has led to problems such as a loss of convenience and safety for users.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0132] In this invention, the server includes a conversion means for converting voice data into text data, an analysis means for analyzing the text data and understanding the user's request, and an information provision engine for acquiring information based on the request and generating the necessary response. This enables quick and accurate responses to various user requests. It also enables appropriate data analysis and information provision for health management and emergency response.

[0133] The "acquisition means" is a module for detecting a user's voice command and acquiring it as voice data.

[0134] "Conversion means" is a module for converting the acquired voice commands into a digital format.

[0135] The "transmitting means" is a module for transmitting the converted digital audio data to the server.

[0136] "Text data" refers to data that has been analyzed and converted into text information from voice data.

[0137] A "voice recognition engine" is software for converting received voice data into text data.

[0138] A "natural language processing (NLP) module" is software that analyzes text data and understands user requests.

[0139] An "information engine" is software that acquires information based on a user's request and generates the necessary response.

[0140] The "voice synthesis means" is a module for converting the generated response into voice data.

[0141] The "output means" is a module for outputting audio data to the user.

[0142] The "monitoring means" is a module for monitoring the user's health condition and issuing a warning if an abnormality is detected.

[0143] An "automatic reporting system" is a system that recognizes a user's voice command in an emergency and immediately makes an emergency call.

[0144] This invention relates to a voice recognition IoT device and system that analyzes a user's voice commands and realizes functions such as providing information, managing health, and making emergency calls. This system includes an edge terminal and a back-end server that execute a series of processes from receiving a user's voice command to analyzing it, providing information, and responding to an emergency.

[0145] System Configuration

[0146] Terminal (edge ​​device) configuration

[0147] 1. Microphone and voice input module: to capture the user's voice commands.

[0148] 2. Audio data processing module: converts the captured audio into digital form.

[0149] 3. Communication module: Sends digital voice data to the server and receives responses from the server.

[0150] 4. Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[0151] Server (backend) configuration

[0152] 1. Speech recognition engine: Converts voice data sent by the user into text data.

[0153] 2. Natural Language Processing (NLP) module: Analyzes text data and understands user requirements.

[0154] 3. Information engine: Obtains information based on user requests and generates the necessary response.

[0155] 4. Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[0156] 5. Database: Stores and manages user health data and emergency contact information.

[0157] Voice input recognition and information provision

[0158] A user speaks to the terminal, "What's the weather like today?" This voice command is picked up by the microphone and digitized by the voice data processing module. This digital voice data is sent to the server through the communication module.

[0159] The server uses a speech recognition engine to convert the received voice data into text data. Next, the NLP module analyzes the text data and understands the user's request, "I want weather information." The server's information provision engine calls the weather information API and obtains the required weather information. The speech synthesis engine then converts the weather information into voice data, generating the message, "Today's weather is sunny." This voice data is sent to the device and conveyed to the user through the device's speaker.

[0160] Example prompt:

[0161] Get the voice command "What's the weather today?" and query the server for weather information.

[0162] Support for health management

[0163] The user says, "Record my blood pressure today." The voice command is captured by the device and sent to the server. Similarly, the voice data is analyzed by the speech recognition engine and NLP module, and the request "record blood pressure data" is understood.

[0164] The server accesses the database and records the user's blood pressure data. Furthermore, it monitors the user's health condition as needed and generates a warning message if an abnormality is detected. The generated information is converted into voice data through a voice synthesis engine and notified to the user via the device.

[0165] Example prompt:

[0166] When you receive the voice command "Record today's blood pressure," save the blood pressure data on your server.

[0167] Emergency response

[0168] The user shouts "An emergency has occurred." The voice command is immediately captured by the device and sent to the server as emergency data. The server uses a speech recognition engine to convert it into text data, and an NLP module analyzes it to indicate an emergency.

[0169] The server retrieves emergency contact information from the database and notifies the designated emergency contacts using an automated notification system. At the same time, the server generates feedback information such as "An emergency call has been made" and converts it into voice data using a voice synthesis engine. This voice data is sent to the device and is played back to the user through the device's speaker.

[0170] Example prompt:

[0171] Instantly process the voice command "an emergency has occurred" and make an emergency call to your designated emergency contacts.

[0172] In this way, a comprehensive voice-recognition IoT device system is realized to support users' daily lives, allowing them to easily use a variety of functions through voice commands and also enabling rapid response in emergencies.

[0173] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0174] Voice input recognition and information provision

[0175] Processing steps and specific explanations

[0176] Step 1:

[0177] The user issues a voice command: "What's the weather like today?"

[0178] Input: Human speech.

[0179] Specific action: The user speaks clearly near the device.

[0180] Step 2:

[0181] The device picks up audio through a microphone.

[0182] Input: Human speech.

[0183] Output: Analog audio signal.

[0184] Specific operation: A microphone picks up an audio signal.

[0185] Step 3:

[0186] The terminal converts the voice into digital data using a voice data processing module.

[0187] Input: Analog audio signal.

[0188] Output: Digital audio data.

[0189] Specific operation: The audio data processing module digitizes the signal.

[0190] Step 4:

[0191] The terminal transmits digital audio data to the server via the communication module.

[0192] Input: Digital audio data.

[0193] Output: Digital audio data sent to the server.

[0194] Specific operation: The communication module transmits data over the Internet.

[0195] Step 5:

[0196] The server uses a voice recognition engine to convert the received voice data into text data.

[0197] Input: Digital audio data.

[0198] Output: Text data.

[0199] Specific operation: Speech recognition software analyzes the voice data and converts it into text.

[0200] Step 6:

[0201] The server analyzes the text data using a natural language processing (NLP) module to understand the user's request.

[0202] Input: Text data.

[0203] Output: User request information ("I want weather information").

[0204] What it does: The NLP module extracts intent from the text.

[0205] Step 7:

[0206] The server uses an information provision engine to obtain the required information from the weather information API.

[0207] Input: User requested information.

[0208] Output: Weather information data (e.g. "Sunny").

[0209] Specific operation: The information engine sends an API request and gets a response.

[0210] Step 8:

[0211] The server converts the weather information obtained using a voice synthesis engine into voice data and sends it to the terminal.

[0212] Input: Weather information data.

[0213] Output: Audio data.

[0214] Specific operation: The speech synthesis engine converts the text into speech and sends it to the terminal through the communication module.

[0215] Step 9:

[0216] The terminal uses a speaker to play audio data and convey information to the user.

[0217] Input: Audio data.

[0218] Output: Audio output ("The weather is sunny today").

[0219] Specific operation: The speaker plays audio data and provides weather information to the user.

[0220] Support for health management

[0221] Processing steps and specific explanations

[0222] Step 1:

[0223] The user issues a voice command such as "Record my blood pressure today."

[0224] Input: Human speech.

[0225] Specific action: The user speaks clearly near the device.

[0226] Step 2:

[0227] The device picks up audio through a microphone.

[0228] Input: Human speech.

[0229] Output: Analog audio signal.

[0230] Specific operation: A microphone picks up an audio signal.

[0231] Step 3:

[0232] The terminal converts the voice into digital data using a voice data processing module.

[0233] Input: Analog audio signal.

[0234] Output: Digital audio data.

[0235] Specific operation: The audio data processing module digitizes the signal.

[0236] Step 4:

[0237] The terminal transmits digital audio data to the server via the communication module.

[0238] Input: Digital audio data.

[0239] Output: Digital audio data sent to the server.

[0240] Specific operation: The communication module transmits data over the Internet.

[0241] Step 5:

[0242] The server uses a voice recognition engine to convert the received voice data into text data.

[0243] Input: Digital audio data.

[0244] Output: Text data.

[0245] Specific operation: Speech recognition software analyzes the voice data and converts it into text.

[0246] Step 6:

[0247] The server analyzes the text data using a natural language processing (NLP) module to understand the user's request.

[0248] Input: Text data.

[0249] Output: User requested information ("Record blood pressure data").

[0250] What it does: The NLP module extracts intent from the text.

[0251] Step 7:

[0252] The server accesses a database and records the user's blood pressure data.

[0253] Input: User requested information.

[0254] Output: Save results to database.

[0255] Specific operation: The server stores the blood pressure data in a database in the appropriate format.

[0256] Step 8:

[0257] The server uses a speech synthesis engine to convert the database update results into voice data and send it to the terminal.

[0258] Input: Database update results.

[0259] Output: Audio data.

[0260] Specific operation: The speech synthesis engine converts the text into speech and sends it to the terminal through the communication module.

[0261] Step 9:

[0262] The terminal uses a speaker to play audio data and convey information to the user.

[0263] Input: Audio data.

[0264] Output: Audio output ("Blood pressure data recorded").

[0265] Specific operation: The speaker plays the audio data and notifies the user.

[0266] Emergency response

[0267] Processing steps and specific explanations

[0268] Step 1:

[0269] The user shouts, "There's an emergency!"

[0270] Input: Human speech.

[0271] Specific action: The user shouts for help in an emergency.

[0272] Step 2:

[0273] The device picks up audio through a microphone.

[0274] Input: Human speech.

[0275] Output: Analog audio signal.

[0276] Specific action: The microphone picks up an emergency voice signal.

[0277] Step 3:

[0278] The terminal converts the voice into digital data using a voice data processing module.

[0279] Input: Analog audio signal.

[0280] Output: Digital audio data.

[0281] Specific operation: The audio data processing module digitizes the signal.

[0282] Step 4:

[0283] The terminal transmits digital audio data to the server via the communication module.

[0284] Input: Digital audio data.

[0285] Output: Digital audio data sent to the server.

[0286] Specific operation: The communication module transmits data over the Internet.

[0287] Step 5:

[0288] The server uses a voice recognition engine to convert the received voice data into text data.

[0289] Input: Digital audio data.

[0290] Output: Text data.

[0291] Specific operation: Speech recognition software analyzes the voice data and converts it into text.

[0292] Step 6:

[0293] The server analyzes the text data using a natural language processing (NLP) module and understands that it is an emergency.

[0294] Input: Text data.

[0295] Output: Emergency call request information.

[0296] Specific behavior: The NLP module extracts the emergency intent from the text.

[0297] Step 7:

[0298] The server retrieves the emergency contact information from the database and places an emergency call using an automated call system.

[0299] Input: Emergency call request information.

[0300] Output: Emergency call result.

[0301] Specific operation: The server automatically notifies emergency contacts.

[0302] Step 8:

[0303] The server converts the results of the emergency call into voice data using a voice synthesis engine and sends it to the terminal.

[0304] Input: Emergency call result.

[0305] Output: Audio data.

[0306] Specific operation: The speech synthesis engine converts the text into speech and sends it to the terminal through the communication module.

[0307] Step 9:

[0308] The terminal uses a speaker to play audio data and convey information to the user.

[0309] Input: Audio data.

[0310] Output: Audio output ("An emergency call has been made").

[0311] Specific action: The speaker plays audio data to notify the user that emergency measures have been taken.

[0312] Thus, the processing steps for quickly and accurately providing various functions based on the user's voice commands have been specifically described.

[0313] (Application example 1)

[0314] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0315] There are situations where it is difficult for consumers to quickly and accurately obtain information about products and services in physical stores, and where it is difficult to properly manage employee health and respond to emergencies. For these reasons, there is a need for a system that can provide information quickly and accurately, manage employee health, and effectively respond to emergencies.

[0316] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0317] In this invention, the server includes an acquisition means for acquiring a user's voice command, a conversion means for converting the acquired voice command into voice data, a transmission means for transmitting the converted voice data to the server, an analysis means for analyzing the voice data on the server side and acquiring corresponding information, a synthesis means for converting the acquired information on the server side into voice, an output means for outputting the synthesized voice data to the user, a means for providing product information and service information on a device installed in the physical store, a means for recording and managing employee health data, and a means for detecting and reporting an emergency within the store. This improves the speed and accuracy of information provision within the physical store, supports employee health management, and enables rapid and effective emergency response.

[0318] The "acquisition means" is a means for acquiring a voice command from a user.

[0319] The "conversion means" is a means for converting the acquired voice command into voice data.

[0320] The "transmitting means" is a means for transmitting the converted voice data to the server.

[0321] The "analysis means" is a means for analyzing the voice data on the server side and acquiring the corresponding information.

[0322] The "synthesis means" is a means for converting information acquired on the server side into voice.

[0323] The "output means" is a means for outputting synthesized voice data to the user.

[0324] "Means for providing product information and service information" refers to means for providing product information and service information using devices installed in physical stores.

[0325] "Means for recording and managing employee health data" means means for recording and managing employee health data.

[0326] "Means for detecting and reporting emergencies within the store" refers to means for detecting emergencies within a physical store and reporting them promptly.

[0327] This invention is a system that uses IoT devices that utilize voice recognition to provide information in a physical store, manage the health of employees, and make emergency calls. Below, we will explain how to specifically implement this system.

[0328] System Configuration

[0329] The system consists of both terminals (edge ​​devices) and servers (backends).

[0330] Terminal (edge ​​device) side

[0331] 1. Microphone and voice input module: to capture the user's voice commands.

[0332] 2. Voice data processing module: converts the captured voice commands into digital form.

[0333] 3. Communication module: Transmits the converted audio data to the server using wireless communication technologies such as Wi-Fi or 4G / 5G.

[0334] 4. Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[0335] Server (backend) side

[0336] 1. Speech recognition engine: Converts voice data sent from the device into text data. Google (registered trademark) Speech Recognition API is used.

[0337] 2. Natural Language Processing (NLP) module: Analyzes text data and understands user requirements. This analysis is performed using a generative AI model.

[0338] 3. Information engine: Obtains information based on user requests and generates the necessary response.

[0339] 4. Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[0340] 5. Database: Stores and manages employee health data and emergency call information.

[0341] Program processing

[0342] The server performs a series of processes including the following steps.

[0343] 1. A means of acquiring the user's voice commands.

[0344] 2. A conversion means for converting the acquired voice command into voice data.

[0345] 3. A transmitting means for transmitting the converted audio data to the server.

[0346] 4. An analysis means for analyzing the voice data on the server side and obtaining the corresponding information.

[0347] 5. A synthesis means for converting the information obtained on the server side into speech.

[0348] 6. Output means for outputting the synthesized voice data to the user.

[0349] 7. A means of providing product and service information on devices installed in physical stores.

[0350] 8. Means for recording and managing employee health data.

[0351] 9. A means of detecting and reporting emergencies within the store.

[0352] This will improve the speed and accuracy of information provision in physical stores, support employee health management, and enable quick and effective response in emergencies.

[0353] Examples of concrete examples and prompts

[0354] Providing product information

[0355] When a user says, "Tell me more about this product," the system provides detailed information about the product.

[0356] Example prompt: "Tell me more about this product."

[0357] Health management

[0358] When an employee says, "Record my blood pressure today," the system records the blood pressure information in a database to help manage their health.

[0359] Example prompt: "Record your blood pressure today."

[0360] Emergency response

[0361] When a user says "an emergency has occurred," the system will quickly make an emergency call and prompt appropriate response.

[0362] Example prompt: "An emergency has occurred."

[0363] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0364] Step 1:

[0365] The user issues a voice command.

[0366] Input: User utterance

[0367] Output: Audio data

[0368] Step 2:

[0369] The terminal uses a microphone and a voice input module to obtain voice commands from the user.

[0370] Input: User's voice

[0371] Output: Analog audio signal

[0372] What it does: A microphone captures sound and generates an analog audio signal.

[0373] Step 3:

[0374] The voice data processing module of the terminal converts the analog voice signal into digital form.

[0375] Input: Analog audio signal

[0376] Output: Digital audio data

[0377] Specific operation: AD conversion (analog-to-digital conversion) processing is performed to convert analog signals into digital signals.

[0378] Step 4:

[0379] The communication module of the terminal transmits the digital voice data to the server.

[0380] Input: Digital audio data

[0381] Output: Audio data sent to the server

[0382] Specific operation: Digital audio data is uploaded to a server using a wireless network (Wi-Fi, 4G / 5G, etc.).

[0383] Step 5:

[0384] The server converts the received digital voice data into text data using a voice recognition engine.

[0385] Input: Digital audio data

[0386] Output: Text data

[0387] Specific behavior: Uses the Google Speech Recognition API to convert audio data into text.

[0388] Step 6:

[0389] The server's natural language processing (NLP) module analyzes the text data and understands the user's request.

[0390] Input: Text data

[0391] Output: Interpretation of user requests

[0392] Specific operation: Using a generative AI model, the text data is analyzed to identify the requested content (product information, health data recording, emergency call, etc.).

[0393] Step 7:

[0394] The server's information providing engine acquires the necessary information based on the user's request and generates a response.

[0395] Input: Interpretation of user requests

[0396] Output: Response data

[0397] Specific operation: For example, when providing product information, retrieve related information from the product database and generate a response.

[0398] Step 8:

[0399] The server's voice synthesis engine converts the response data into voice data and sends it to the terminal.

[0400] Input: Response data

[0401] Output: Audio data

[0402] Specific operation: Text-to-speech (TTS) is performed and the audio data is sent to the terminal.

[0403] Step 9:

[0404] The terminal receives the transmitted voice data and conveys information to the user using a speaker and a voice output module.

[0405] Input: Audio data

[0406] Output: Audio output

[0407] Specific operation: Plays the received audio data and outputs information to the user through the speaker.

[0408] This allows for a fast and accurate response to the user's voice commands.

[0409] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0410] This invention adds emotion recognition functionality to a voice recognition IoT device that analyzes a user's voice commands and provides information, health management, and emergency notification functions, providing appropriate feedback and support according to the user's emotional state. This system includes an edge terminal and a back-end server for executing a series of processes from voice command acquisition and analysis to information provision, emotion recognition, and emergency response.

[0411] System Configuration

[0412] 1. Terminal (edge ​​device) side

[0413] Microphone and voice input module: to capture the user's voice commands.

[0414] Voice data processing module: converts voice data into digital format and sends it to the server.

[0415] Communication module: Sends voice data to the server and receives responses from the server.

[0416] Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[0417] 2. Server (backend) side

[0418] Speech recognition engine: Converts voice data sent by the user into text data.

[0419] Natural Language Processing (NLP) module: Analyzes text data and understands user requirements.

[0420] Information provision engine: Obtains information based on user requests and generates the required response.

[0421] Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[0422] Database: Stores and manages user health data and emergency contact information.

[0423] Emotion engine: Analyzes the user's emotions from voice data and generates emotional information.

[0424] Feedback engine: Generates appropriate feedback to users based on emotional information.

[0425] Monitoring system: A system for monitoring the user's health and mental state.

[0426] Program processing and specific examples

[0427] Voice input recognition and information provision

[0428] A user speaks to the terminal, "What's the weather like today?" This voice command is picked up by the microphone and digitized by the voice data processing module. This digital voice data is sent to the server through the communication module.

[0429] The server uses a speech recognition engine to convert the received voice data into text data. Next, the NLP module analyzes the text data and understands the user's request, "I want weather information." The server uses the information provision engine to call the weather information API and obtain the required weather information. The emotion engine then analyzes the user's emotion from the voice data and generates emotion information such as "curiosity" or "confusion." Based on this, the feedback engine adjusts the text message and adds positive feedback such as "The weather is sunny today. Have a nice day." The message is then converted into voice data by the speech synthesis engine and sent to the device. It is then audibly conveyed to the user through the device's speaker.

[0430] Support for health management

[0431] The user says, "Record my blood pressure today." The voice command is captured by the device and sent to the server. Similarly, the voice data is analyzed by the speech recognition engine and NLP module, and the request "record blood pressure data" is understood.

[0432] The server accesses the database and records the user's blood pressure data. Furthermore, an emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "anxiety" or "worry." A feedback engine generates feedback messages to reassure the user. The generated information is converted into voice data through a voice synthesis engine, and the user is notified via their device that "Your blood pressure has been recorded. Please relax."

[0433] Emergency response

[0434] The user shouts "An emergency has occurred." The voice command is immediately captured by the device and sent to the server as emergency data. The server uses a speech recognition engine to convert it into text data, and an NLP module analyzes it to indicate an emergency.

[0435] The server retrieves emergency contact information from the database and uses an automatic notification system to notify the designated emergency contact. At the same time, an emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "fear" or "tension." Based on this, a feedback engine generates a feedback message to reassure the user. The message, "An emergency call has been made. Please wait until the ambulance arrives," is converted into voice data by a speech synthesis engine and sent to the terminal for transmission to the user.

[0436] In this way, adding emotion recognition functionality enables flexible responses according to the user's emotional state, improving the user experience.

[0437] The processing flow will be explained below.

[0438] Voice input recognition and information provision

[0439] Step 1:

[0440] The user speaks the voice command "What's the weather like today?" into the device.

[0441] Step 2:

[0442] The device uses a built-in microphone to record voice commands.

[0443] Step 3:

[0444] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[0445] Step 4:

[0446] The terminal transmits the converted voice data to a server via the Internet.

[0447] Step 5:

[0448] The server's voice recognition engine converts the received voice data into text data.

[0449] Step 6:

[0450] The server's natural language processing (NLP) module analyzes the text data "What's the weather like today?" and understands the user's request.

[0451] Step 7:

[0452] The server calls the weather information API based on the analysis results and obtains current weather information.

[0453] Step 8:

[0454] The server's information engine generates a text message saying "Today's weather is sunny" based on the acquired weather information.

[0455] Step 9:

[0456] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "curiosity" or "confusion."

[0457] Step 10:

[0458] The server's feedback engine generates feedback appropriate for the user based on the emotional information, generating a message such as "The weather is sunny today. Have a nice day."

[0459] Step 11:

[0460] A speech synthesis engine on the server converts the generated text message into voice data.

[0461] Step 12:

[0462] The server transmits the converted audio data to the terminal via the Internet.

[0463] Step 13:

[0464] The terminal's audio output module plays the received data over the speaker and tells the user in a voice message, "Today's weather is sunny. Have a nice day."

[0465] Support for health management

[0466] Step 1:

[0467] The user speaks to the terminal, saying, "Record my blood pressure today."

[0468] Step 2:

[0469] The device will record your voice commands using the built-in microphone.

[0470] Step 3:

[0471] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[0472] Step 4:

[0473] The terminal transmits the converted voice data to a server via the Internet.

[0474] Step 5:

[0475] The server's voice recognition engine converts the received voice data into text data.

[0476] Step 6:

[0477] The server's natural language processing (NLP) module analyzes the text data "Record my blood pressure today" and understands the user's request.

[0478] Step 7:

[0479] The server accesses the user's physical condition database and records the blood pressure data.

[0480] Step 8:

[0481] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "anxiety" or "worry."

[0482] Step 9:

[0483] The server's feedback engine generates a feedback message based on the emotional information to reassure the user, such as "Your blood pressure has been recorded. Please relax."

[0484] Step 10:

[0485] A speech synthesis engine on the server converts the generated text message into voice data.

[0486] Step 11:

[0487] The server transmits the converted audio data to the terminal via the Internet.

[0488] Step 12:

[0489] The device's audio output module plays the received data over the speaker and tells the user in a voice message, "Your blood pressure has been recorded. Please relax."

[0490] Emergency response

[0491] Step 1:

[0492] The user shouts into the terminal, "An emergency has occurred."

[0493] Step 2:

[0494] The device will record emergency voice commands using the built-in microphone.

[0495] Step 3:

[0496] The voice data processing module of the terminal converts the recorded voice into digital data and passes it to the emergency processing module.

[0497] Step 4:

[0498] The terminal immediately transmits the formatted emergency voice data to the server.

[0499] Step 5:

[0500] The server's voice recognition engine converts the received emergency voice data into text data.

[0501] Step 6:

[0502] The server's natural language processing (NLP) module analyzes the text data "An emergency has occurred" and understands that it is an emergency.

[0503] Step 7:

[0504] The server retrieves the emergency contact information from the database and sends an emergency message to the emergency contact using an automated notification system.

[0505] Step 8:

[0506] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "fear" or "tension."

[0507] Step 9:

[0508] The server's feedback engine generates a reassuring feedback message based on the emotional information, such as "An emergency call has been made. Please wait until the ambulance arrives."

[0509] Step 10:

[0510] A speech synthesis engine on the server converts the generated feedback message into voice data.

[0511] Step 11:

[0512] The server transmits the converted audio data to the terminal via the Internet.

[0513] Step 12:

[0514] The terminal's audio output module plays the received data over the speaker and tells the user in a voice message, "An emergency call has been made. Please wait until the ambulance arrives."

[0515] Example 2

[0516] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0517] Conventional voice recognition systems only analyze users' voice commands and provide information, but are unable to provide appropriate feedback or support based on the user's emotional state. This makes it difficult to improve the user experience or respond quickly and appropriately in emergencies.

[0518] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0519] In this invention, the server includes a speech recognition means for converting voice data into text data, a natural language processing means for analyzing the text data and understanding the request, and an information providing means for acquiring information based on the request and generating a response. This enables not only the provision of information based on the user's voice command, but also appropriate feedback based on emotion recognition and a rapid response in emergencies.

[0520] The "acquisition means" is a device or module for acquiring a user's voice command.

[0521] The "conversion means" is a device or module for converting the acquired audio data into digital data.

[0522] The "transmitting means" is a communication device or module for transmitting the converted voice data to the server.

[0523] "Speech recognition means" refers to a device or algorithm that converts voice data into text data on the server side.

[0524] "Natural language processing means" refers to devices and algorithms that analyze text data and understand user requests.

[0525] An "information providing means" is a device or algorithm for obtaining information based on a user request and generating a response.

[0526] A "voice synthesis means" is a device or algorithm for converting a generated response into voice data.

[0527] "Output means" refers to a device or module for outputting audio data to the user.

[0528] "Emotion recognition means" refers to a device or algorithm for analyzing a user's emotions from voice data.

[0529] The "feedback means" refers to a device or algorithm for generating appropriate feedback to the user based on emotional information.

[0530] The "support means" is a device or system that supports the user's health management using the health data and emotion information acquired on the server side.

[0531] The "reporting means" refers to a device or system that quickly recognizes a user's voice command in an emergency and immediately makes an emergency call.

[0532] This invention is a system that adds emotion recognition functionality to a voice recognition IoT device that analyzes a user's voice commands and realizes functions such as providing information, managing one's health, and making an emergency call, and provides appropriate feedback and support according to the user's emotional state. Here, an embodiment of the present invention will be specifically described.

[0533] System Configuration

[0534] Terminal (edge ​​device) side

[0535] Microphone and voice input module: Acquires the user's voice commands by capturing the user's voice as an analog signal.

[0536] Audio data processing module: Converts the captured analog audio signal into digital format. Samples the audio signal and encodes it into digital data.

[0537] Communication module: Sends digital audio data to the server and receives responses from the server. Uses Wi-Fi or mobile data.

[0538] Speaker and audio output module: Plays back audio data received from the server and conveys information to the user. Converts digital audio data into an analog signal and plays it as sound through a speaker.

[0539] Server (backend) side

[0540] Speech recognition engine: Converts voice data sent from the device into text data. It uses machine learning models to analyze the voice data and convert it into text format.

[0541] Natural Language Processing (NLP) module: Analyzes text data to understand user requirements, analyzes text data grammatically, and extracts keywords and context.

[0542] Information engine: Obtains information based on user requests and generates responses. Specifically, it accesses weather information APIs and other sources to obtain the necessary information.

[0543] Speech synthesis engine: Converts the generated response into voice data and sends it to the device. It converts text information into natural-sounding voice using speech synthesis technology.

[0544] Database: Stores and manages user health data and emergency contact information.

[0545] Emotion Engine: Analyzes user emotions from voice data and generates emotional information. Analyzes voice tone and word usage to identify emotional state.

[0546] Feedback Engine: Generates appropriate feedback based on emotional information. Adjusts the tone and content of messages depending on the emotional state.

[0547] Monitoring system: A system that monitors the user's health and mental state.

[0548] Specific examples

[0549] Specific examples of voice input recognition and information provision

[0550] The user speaks to the device, "What's the weather like today?" The device's microphone picks up this voice, and the voice data processing module converts the analog voice signal into digital data. The communication module sends the digital voice data to the server.

[0551] The server's speech recognition engine converts the voice data into text data, and the NLP module analyzes the text to understand the request, "I want weather information." The information provision engine calls the weather information API to obtain the required information. The emotion engine analyzes emotions such as "curiosity" from the voice data, and the feedback engine generates a positive message such as "The weather is sunny today. Have a nice day."

[0552] The speech synthesis engine converts this message into voice data and sends it to the terminal via the communication module, where the terminal's speaker plays the voice back to the user.

[0553] example:

[0554] User: "What's the weather like today?"

[0555] Server: "The weather is sunny today. Have a nice day."

[0556] Specific examples of support for health management

[0557] The user says, "Record my blood pressure today." The terminal receives the voice command, and the voice data processing module converts it into digital data and sends it to the server.

[0558] The server's speech recognition engine and NLP module analyze the voice data and understand the request to "record blood pressure data." The database is accessed and the user's blood pressure data is recorded. The emotion engine analyzes emotions such as "anxiety" and "worry," and the feedback engine generates a message saying, "Your blood pressure has been recorded. Please relax."

[0559] The speech synthesis engine converts this message into voice data, sends it to the device, and plays it back to the user through the device's speaker.

[0560] example:

[0561] User: "Log my blood pressure today."

[0562] Server: "Your blood pressure has been recorded. Please relax."

[0563] Specific examples of emergency response

[0564] The user shouts, "An emergency has occurred." The device immediately receives the voice command and sends it to the server as emergency data.

[0565] The server's speech recognition engine converts the voice data into text data, and the NLP module analyzes the emergency situation. Emergency contact information is retrieved from the database, and the automated notification system notifies the emergency contacts specified in the database. At the same time, the emotion engine analyzes emotions such as "fear" and "tension," and the feedback engine generates a message saying, "An emergency call has been made. Please wait until the ambulance arrives."

[0566] The speech synthesis engine converts this message into voice data, sends it to the device, and plays it back to the user through the device's speaker.

[0567] example:

[0568] User: "We have an emergency."

[0569] Server: "We've called emergency services. Please wait until the ambulance arrives."

[0570] In this way, this system executes a series of processes from acquiring voice commands to analyzing them, providing information, recognizing emotions, and responding to emergencies, enabling flexible responses according to the user's emotional state.

[0571] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0572] Step 1:

[0573] The user issues a voice command. The input is the user's voice command (e.g., "What's the weather today?"). The device's microphone and voice input module pick up this voice command and capture it as an analog voice signal. The output is an analog voice signal.

[0574] Step 2:

[0575] The terminal's voice data processing module converts the acquired analog voice signal into digital data. The input is an analog voice signal and the output is digital voice data. Specifically, the voice signal is sampled and encoded into a digital format (bit string).

[0576] Step 3:

[0577] The communication module of the device sends digital audio data to the server. The input is digital audio data, and the output is digital audio data sent to the server. Data transmission uses Wi-Fi or mobile data communication.

[0578] Step 4:

[0579] The server's speech recognition engine converts the received digital voice data into text data. The input is the digital voice data sent to the server, and the output is text data. Specifically, it uses a machine learning model to analyze phonemes and context and convert the data into text format.

[0580] Step 5:

[0581] The server's natural language processing (NLP) module analyzes the text data and understands the user's request. The input is text data, and the output is the analyzed request (e.g., "Get weather information"). Specifically, it analyzes the text data grammatically and extracts keywords and context.

[0582] Step 6:

[0583] The server's information engine retrieves information based on the user's request and generates a response. The input is the analyzed request (retrieving weather information), and the output is a response based on the retrieved information (e.g., "Today's weather is sunny"). Specifically, it accesses the weather information API and retrieves today's weather information.

[0584] Step 7:

[0585] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information. The input is the user's voice data, and the output is emotional information (e.g., "curiosity"). Specific operations include analyzing the voice tone and word usage to identify the emotional state.

[0586] Step 8:

[0587] The server's feedback engine generates appropriate feedback based on the emotional information. The input is the emotional information and acquired information (e.g., "Curiosity" and "The weather is sunny today"), and the output is a feedback message (e.g., "The weather is sunny today. Have a nice day."). Specific behavior is to adjust the tone and content of the message depending on the emotional state.

[0588] Step 9:

[0589] The server's speech synthesis engine converts the feedback message into speech data. The input is the feedback message (text) and the output is speech data. Specifically, the text information is converted into natural speech using speech synthesis technology.

[0590] Step 10:

[0591] The communication module of the server transmits the voice data to the terminal. The input is the voice data, and the output is the voice data transmitted to the terminal.

[0592] Step 11:

[0593] The device's speaker and audio output module plays the audio data received from the server and provides feedback to the user. The input is audio data, and the output is the audio being played. Specifically, the digital audio data is converted into an analog signal and played as sound through the speaker.

[0594] example:

[0595] User: "What's the weather like today?"

[0596] Server: "The weather is sunny today. Have a nice day."

[0597] (Application example 2)

[0598] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0599] Conventional voice recognition systems can analyze a user's voice commands and provide information, but they do not provide feedback or support that takes into account the user's emotional state. As a result, they are unable to respond flexibly to the user's emotional state, and there is a lack of appropriate support to alleviate the user's anxiety and fear, especially in emergencies. There is a need to solve this problem.

[0600] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0601] In this invention, the server includes emotion analysis means for analyzing emotions from the user's voice, feedback generation means for generating appropriate feedback based on the emotion information, support means for supporting the user's health management, and reporting means for quickly recognizing the user's voice command in an emergency and immediately reporting to an emergency service, thereby enabling flexible responses and appropriate support according to the user's voice command as well as their emotional state.

[0602] The "acquisition means" is a device or mechanism that acquires a user's voice command.

[0603] The "conversion means" is a device or mechanism that converts the acquired voice command into digital voice data.

[0604] The "transmitting means" is a device or mechanism that transmits the converted voice data to the server.

[0605] The "analysis means" is a device or mechanism that analyzes the voice data received on the server side and acquires the corresponding information.

[0606] The "synthesizing means" is a device or mechanism that converts the information acquired on the server side into voice data.

[0607] The "output means" is a device or mechanism that outputs synthesized voice data to the user.

[0608] The "emotion analysis means" is a device or mechanism that analyzes emotions from the user's voice.

[0609] The "feedback generating means" is a device or mechanism that generates appropriate feedback based on emotional information.

[0610] The "support means" is a device or mechanism that supports the user's health management using information acquired on the server side.

[0611] The "notification means" is a device or mechanism that quickly recognizes a user's voice command in an emergency and immediately makes an emergency call.

[0612] This invention adds emotion recognition functionality to a voice recognition IoT system that analyzes users' voice commands and provides information, health management, and emergency notification functions, thereby providing appropriate feedback and support according to the user's emotional state. This system consists of a terminal (edge ​​device) and a server (backend).

[0613] System Configuration

[0614] 1. Terminal (edge ​​device) side

[0615] Microphone and voice input module: A device that captures the user's voice commands, which are converted into a digital format by a conversion means, which will be described later.

[0616] Voice data processing module: A device that converts the captured voice commands into digital format and transmits them to the server.

[0617] Communication module: A device that transmits converted voice data to a server and receives responses from the server.

[0618] Speaker and audio output module: A device that plays back audio data received from the server and conveys information to the user.

[0619] 2. Server (backend) side

[0620] Speech recognition engine: Software that converts voice data sent by a user into text data, so that the user's commands can be analyzed.

[0621] Natural Language Processing (NLP) module: Software that analyzes text data and understands user requests, enabling it to provide appropriate information and take appropriate action.

[0622] Information provision engine: Software that obtains information based on user requests and generates the necessary responses.

[0623] Speech synthesis engine: Software that converts the generated response into voice data and sends it to the terminal.

[0624] Database: A device that stores and manages a user's health data and emergency contact information.

[0625] Emotion analysis means: A device that analyzes the user's emotions from voice data and generates emotional information.

[0626] Feedback generation means: Software that generates appropriate feedback to the user based on emotional information.

[0627] Monitoring system: A system for monitoring the user's health and mental state.

[0628] Support means: A device that uses the acquired information to support the user in managing their physical condition.

[0629] Notification means: A device that quickly recognizes the user's voice command in an emergency and immediately makes an emergency call.

[0630] System processing example

[0631] Voice input recognition and information provision

[0632] For example, when a user says, "What's the weather today?", the voice command is picked up by the microphone and converted into digital form by the voice data processing module. This digital voice data is sent to the server through the communication module. The server uses a voice recognition engine to convert the received voice data into text data. The NLP module then analyzes the text data and understands the user's request, "I want weather information." The information provision engine obtains the weather information, and the emotion analysis means analyzes the user's emotions to generate appropriate feedback and send it to the terminal as voice data. The terminal's speaker then tells the user, "The weather is sunny today. Have a nice day."

[0633] Support for health management

[0634] When a user says, "Record my blood pressure today," the voice command is captured and converted into digital voice data. This is sent to the server, where it is analyzed by a voice recognition engine and an NLP module to understand the user's request. The server accesses the database and records the blood pressure data. At the same time, an emotion analysis means analyzes the user's emotions and generates reassuring feedback. This notifies the user, "Your blood pressure has been recorded. Please relax."

[0635] Emergency response

[0636] When a user shouts "An emergency has occurred," the voice command is immediately captured and sent to the server as emergency data. The voice recognition engine and NLP module convert the voice command into text data, which is then analyzed to indicate an emergency. The server retrieves emergency contact information from the database and notifies the emergency contacts using the notification means. At the same time, the emotion analysis means analyzes the user's emotions and generates emotional information such as "fear" or "tension," thereby generating feedback to reassure the user. The message "An emergency call has been made. Please wait until the ambulance arrives" is converted into voice data, sent to the terminal, and conveyed to the user.

[0637] Prompt Sentence Examples

[0638] If the user says, "Did you lock the house?"

[0639] 1. The microphone picks up the voice and the speech recognition engine converts it into text: "Have you locked your house?"

[0640] 2. Emotion analysis determines that the user is "anxious."

[0641] 3. The command is parsed and a response is generated: "Check home security."

[0642] 4. The security status is queried from the server and the result is obtained.

[0643] 5. The feedback to the user, "The house is locked. Don't worry," is converted into audio data and played back.

[0644] This system can provide appropriate feedback based on the user's emotional state, improving the user's experience.

[0645] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0646] Step 1:

[0647] The user issues a voice command through a microphone. The microphone captures this voice. The input is the user's voice command, and the output is the captured analog voice data. Specifically, the user speaks a command such as "What's the weather like today?"

[0648] Step 2:

[0649] The terminal converts the captured analog voice data into a digital format using a voice data processing module. The input is analog voice data, and the output is digital voice data. Specifically, the analog data is converted into digital data through sampling and quantization.

[0650] Step 3:

[0651] The communication module of the terminal transmits the converted digital voice data to the server. The input is the digital voice data, and the output is data transmission to the server. The data is transferred using a communication protocol.

[0652] Step 4:

[0653] The server uses a speech recognition engine to convert the received digital voice data into text data. The input is digital voice data, and the output is text data. Specifically, the server analyzes the voice waveform and converts it into words.

[0654] Step 5:

[0655] The server's natural language processing (NLP) module analyzes text data and understands the user's request. The input is text data from the speech recognition engine, and the output is the user's request, which is the analysis result. Intent is identified using general-purpose deep learning models and rule-based analysis.

[0656] Step 6:

[0657] The server's information engine retrieves information based on the user's request and generates the necessary response. The input is the analysis result, and the output is the response information. For example, accessing a weather information API to retrieve current weather data.

[0658] Step 7:

[0659] The emotion analysis means of the server analyzes the user's emotion from the voice data. The input is the voice data, and the output is the user's emotion information. The emotion is identified by analyzing the voice tone, speed, and intonation.

[0660] Step 8:

[0661] The server's feedback generation means generates appropriate feedback for the user based on the emotion information. The input is emotion information and response information, and the output is adjusted feedback text. For example, a message such as "Today's weather is sunny. Have a nice day" is generated.

[0662] Step 9:

[0663] The server's speech synthesis engine converts the generated feedback text into speech data. The input is the feedback text, and the output is speech data. Text-to-speech (TTS) technology is used to convert the text into speech.

[0664] Step 10:

[0665] The communication module of the terminal receives the voice data received from the server, where the input is the voice data sent from the server and the output is the voice data received by the terminal.

[0666] Step 11:

[0667] The device's speaker and audio output module plays the received audio data and conveys information to the user. The input is audio data, and the output is the audio heard by the user. The audio is played on the device's built-in speaker.

[0668] Prompt Sentence Examples

[0669] If the user says, "Did you lock the house?"

[0670] 1. The microphone picks up the voice and the speech recognition engine converts it into text: "Have you locked your house?"

[0671] 2. Emotion analysis determines that the user is "anxious."

[0672] 3. The command is parsed and a response is generated: "Check home security."

[0673] 4. The security status is queried from the server and the result is obtained.

[0674] 5. The feedback to the user, "The house is locked. Don't worry," is converted into audio data and played back.

[0675] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0676] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0677] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0678] [Second embodiment]

[0679] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0680] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0681] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0682] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0683] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0684] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0685] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0686] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0687] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0688] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0689] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0690] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0691] This invention relates to a voice recognition IoT device and system that analyzes a user's voice commands and realizes functions such as providing information, managing health, and making emergency calls. This system includes an edge terminal and a back-end server that execute a series of processes from receiving a user's voice command to analyzing it, providing information, and responding to an emergency.

[0692] System Configuration

[0693] 1. Terminal (edge ​​device) side

[0694] Microphone and voice input module: to capture the user's voice commands.

[0695] Voice data processing module: converts voice data into digital format and sends it to the server.

[0696] Communication module: Sends voice data to the server and receives responses from the server.

[0697] Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[0698] 2. Server (backend) side

[0699] Speech recognition engine: Converts voice data sent by the user into text data.

[0700] Natural Language Processing (NLP) module: Analyzes text data and understands user requirements.

[0701] Information provision engine: Obtains information based on user requests and generates the required response.

[0702] Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[0703] Database: Stores and manages user health data and emergency contact information.

[0704] Program processing and specific examples

[0705] Voice input recognition and information provision

[0706] A user speaks to the terminal, "What's the weather like today?" This voice command is picked up by the microphone and digitized by the voice data processing module. This digital voice data is sent to the server through the communication module.

[0707] The server uses a speech recognition engine to convert the received voice data into text data. Next, the NLP module analyzes the text data and understands the user's request, "I want weather information." The server uses the information provision engine to call the weather information API and obtain the required weather information. The speech synthesis engine then converts the weather information into voice data, generating the message, "Today's weather is sunny." This voice data is sent to the device and is conveyed to the user through the device's speaker.

[0708] Support for health management

[0709] The user says, "Record my blood pressure today." The voice command is captured by the device and sent to the server. Similarly, the voice data is analyzed by the speech recognition engine and NLP module, and the request "record blood pressure data" is understood.

[0710] The server accesses the database and records the user's blood pressure data. Furthermore, it monitors the user's health condition as needed and generates a warning message if an abnormality is detected. The generated information is converted into voice data through a voice synthesis engine and notified to the user via the device.

[0711] Emergency response

[0712] The user shouts "An emergency has occurred." The voice command is immediately captured by the device and sent to the server as emergency data. The server uses a speech recognition engine to convert it into text data, and an NLP module analyzes it to indicate an emergency.

[0713] The server retrieves emergency contact information from the database and notifies the designated emergency contacts using an automated notification system. At the same time, the server generates feedback information such as "An emergency call has been made" and converts it into voice data using a voice synthesis engine. This voice data is sent to the device and is played back to the user through the device's speaker.

[0714] In this way, a comprehensive system of voice-recognition IoT devices to support users' daily lives is realized.

[0715] The processing flow will be explained below.

[0716] Voice input recognition and information provision

[0717] Step 1:

[0718] The user speaks the voice command "What's the weather like today?" into the device.

[0719] Step 2:

[0720] The device uses a built-in microphone to record voice commands.

[0721] Step 3:

[0722] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[0723] Step 4:

[0724] The terminal transmits the converted voice data to a server via the Internet.

[0725] Step 5:

[0726] The server's voice recognition engine converts the received voice data into text data.

[0727] Step 6:

[0728] The server's natural language processing (NLP) module analyzes the text data "What's the weather like today?" and understands the user's request.

[0729] Step 7:

[0730] The server calls the weather information API based on the analysis results and obtains current weather information.

[0731] Step 8:

[0732] The server's information engine generates a text message saying "Today's weather is sunny" based on the acquired weather information.

[0733] Step 9:

[0734] A speech synthesis engine on the server converts the generated text message into voice data.

[0735] Step 10:

[0736] The server then transmits the converted voice data to the terminal again via the Internet.

[0737] Step 11:

[0738] The audio output module of the terminal plays the received data over the speaker and tells the user aloud, "Today's weather is sunny."

[0739] Support for health management

[0740] Step 1:

[0741] The user speaks to the terminal, saying, "Record my blood pressure today."

[0742] Step 2:

[0743] The device will record your voice commands using the built-in microphone.

[0744] Step 3:

[0745] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[0746] Step 4:

[0747] The terminal transmits the converted voice data to a server via the Internet.

[0748] Step 5:

[0749] The server's voice recognition engine converts the received voice data into text data.

[0750] Step 6:

[0751] The server's natural language processing (NLP) module analyzes the text data "Record my blood pressure today" and understands the user's request.

[0752] Step 7:

[0753] The server accesses the user's physical condition database and records the blood pressure data.

[0754] Step 8:

[0755] The server generates a feedback message indicating that the recording was successful, and converts it into voice data using a voice synthesis engine.

[0756] Step 9:

[0757] The server then transmits the converted voice data to the terminal again via the Internet.

[0758] Step 10:

[0759] The device's audio output module plays the received data over the speaker and tells the user in a voice message, "Your blood pressure has been recorded. Take care."

[0760] Emergency response

[0761] Step 1:

[0762] The user shouts into the terminal, "An emergency has occurred."

[0763] Step 2:

[0764] The device will record emergency voice commands using the built-in microphone.

[0765] Step 3:

[0766] The voice data processing module of the terminal converts the recorded voice into digital data and passes it to the emergency processing module.

[0767] Step 4:

[0768] The terminal immediately transmits the formatted emergency voice data to the server.

[0769] Step 5:

[0770] The server's voice recognition engine converts the received emergency voice data into text data.

[0771] Step 6:

[0772] The server's natural language processing (NLP) module analyzes the text data "An emergency has occurred" and understands that it is an emergency.

[0773] Step 7:

[0774] A server retrieves emergency contact information from a database and sends emergency messages to the contacts using an automated notification system.

[0775] Step 8:

[0776] The server checks the status of the call transmission, generates a feedback message such as "An emergency call has been made. Please wait until the ambulance arrives," and converts it into voice data using a speech synthesis engine.

[0777] Step 9:

[0778] The server then transmits the converted voice data to the terminal again via the Internet.

[0779] Step 10:

[0780] The terminal's audio output module plays the received data over the speaker and tells the user in a voice message, "An emergency call has been made. Please wait until the ambulance arrives."

[0781] Example 1

[0782] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0783] Conventional speech recognition systems have had difficulty responding quickly and accurately to diverse user requests. Furthermore, there has been a lack of technology to accurately analyze user requests and provide appropriate information, particularly in the areas of health management and emergency response. This has led to problems such as a loss of convenience and safety for users.

[0784] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0785] In this invention, the server includes a conversion means for converting voice data into text data, an analysis means for analyzing the text data and understanding the user's request, and an information provision engine for acquiring information based on the request and generating the necessary response. This enables quick and accurate responses to various user requests. It also enables appropriate data analysis and information provision for health management and emergency response.

[0786] The "acquisition means" is a module for detecting a user's voice command and acquiring it as voice data.

[0787] "Conversion means" is a module for converting the acquired voice commands into a digital format.

[0788] The "transmitting means" is a module for transmitting the converted digital audio data to the server.

[0789] "Text data" refers to data that has been analyzed and converted into text information from voice data.

[0790] A "voice recognition engine" is software for converting received voice data into text data.

[0791] A "natural language processing (NLP) module" is software that analyzes text data and understands user requests.

[0792] An "information engine" is software that acquires information based on a user's request and generates the necessary response.

[0793] The "voice synthesis means" is a module for converting the generated response into voice data.

[0794] The "output means" is a module for outputting audio data to the user.

[0795] The "monitoring means" is a module for monitoring the user's health condition and issuing a warning if an abnormality is detected.

[0796] An "automatic reporting system" is a system that recognizes a user's voice command in an emergency and immediately makes an emergency call.

[0797] This invention relates to a voice recognition IoT device and system that analyzes a user's voice commands and realizes functions such as providing information, managing health, and making emergency calls. This system includes an edge terminal and a back-end server that execute a series of processes from receiving a user's voice command to analyzing it, providing information, and responding to an emergency.

[0798] System Configuration

[0799] Terminal (edge ​​device) configuration

[0800] 1. Microphone and voice input module: to capture the user's voice commands.

[0801] 2. Audio data processing module: converts the captured audio into digital form.

[0802] 3. Communication module: Sends digital voice data to the server and receives responses from the server.

[0803] 4. Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[0804] Server (backend) configuration

[0805] 1. Speech recognition engine: Converts voice data sent by the user into text data.

[0806] 2. Natural Language Processing (NLP) module: Analyzes text data and understands user requirements.

[0807] 3. Information engine: Obtains information based on user requests and generates the necessary response.

[0808] 4. Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[0809] 5. Database: Stores and manages user health data and emergency contact information.

[0810] Voice input recognition and information provision

[0811] A user speaks to the terminal, "What's the weather like today?" This voice command is picked up by the microphone and digitized by the voice data processing module. This digital voice data is sent to the server through the communication module.

[0812] The server uses a speech recognition engine to convert the received voice data into text data. Next, the NLP module analyzes the text data and understands the user's request, "I want weather information." The server's information provision engine calls the weather information API and obtains the required weather information. The speech synthesis engine then converts the weather information into voice data, generating the message, "Today's weather is sunny." This voice data is sent to the device and conveyed to the user through the device's speaker.

[0813] Example prompt:

[0814] Get the voice command "What's the weather today?" and query the server for weather information.

[0815] Support for health management

[0816] The user says, "Record my blood pressure today." The voice command is captured by the device and sent to the server. Similarly, the voice data is analyzed by the speech recognition engine and NLP module, and the request "record blood pressure data" is understood.

[0817] The server accesses the database and records the user's blood pressure data. Furthermore, it monitors the user's health condition as needed and generates a warning message if an abnormality is detected. The generated information is converted into voice data through a voice synthesis engine and notified to the user via the device.

[0818] Example prompt:

[0819] When you receive the voice command "Record today's blood pressure," save the blood pressure data on your server.

[0820] Emergency response

[0821] The user shouts "An emergency has occurred." The voice command is immediately captured by the device and sent to the server as emergency data. The server uses a speech recognition engine to convert it into text data, and an NLP module analyzes it to indicate an emergency.

[0822] The server retrieves emergency contact information from the database and notifies the designated emergency contacts using an automated notification system. At the same time, the server generates feedback information such as "An emergency call has been made" and converts it into voice data using a voice synthesis engine. This voice data is sent to the device and is played back to the user through the device's speaker.

[0823] Example prompt:

[0824] Instantly process the voice command "an emergency has occurred" and make an emergency call to your designated emergency contacts.

[0825] In this way, a comprehensive voice-recognition IoT device system is realized to support users' daily lives, allowing them to easily use a variety of functions through voice commands and also enabling rapid response in emergencies.

[0826] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0827] Voice input recognition and information provision

[0828] Processing steps and specific explanations

[0829] Step 1:

[0830] The user issues a voice command: "What's the weather like today?"

[0831] Input: Human speech.

[0832] Specific action: The user speaks clearly near the device.

[0833] Step 2:

[0834] The device picks up audio through a microphone.

[0835] Input: Human speech.

[0836] Output: Analog audio signal.

[0837] Specific operation: A microphone picks up an audio signal.

[0838] Step 3:

[0839] The terminal converts the voice into digital data using a voice data processing module.

[0840] Input: Analog audio signal.

[0841] Output: Digital audio data.

[0842] Specific operation: The audio data processing module digitizes the signal.

[0843] Step 4:

[0844] The terminal transmits digital audio data to the server via the communication module.

[0845] Input: Digital audio data.

[0846] Output: Digital audio data sent to the server.

[0847] Specific operation: The communication module transmits data over the Internet.

[0848] Step 5:

[0849] The server uses a voice recognition engine to convert the received voice data into text data.

[0850] Input: Digital audio data.

[0851] Output: Text data.

[0852] Specific operation: Speech recognition software analyzes the voice data and converts it into text.

[0853] Step 6:

[0854] The server analyzes the text data using a natural language processing (NLP) module to understand the user's request.

[0855] Input: Text data.

[0856] Output: User request information ("I want weather information").

[0857] What it does: The NLP module extracts intent from the text.

[0858] Step 7:

[0859] The server uses an information provision engine to obtain the required information from the weather information API.

[0860] Input: User requested information.

[0861] Output: Weather information data (e.g. "Sunny").

[0862] Specific operation: The information engine sends an API request and gets a response.

[0863] Step 8:

[0864] The server converts the weather information obtained using a voice synthesis engine into voice data and sends it to the terminal.

[0865] Input: Weather information data.

[0866] Output: Audio data.

[0867] Specific operation: The speech synthesis engine converts the text into speech and sends it to the terminal through the communication module.

[0868] Step 9:

[0869] The terminal uses a speaker to play audio data and convey information to the user.

[0870] Input: Audio data.

[0871] Output: Audio output ("The weather is sunny today").

[0872] Specific operation: The speaker plays audio data and provides weather information to the user.

[0873] Support for health management

[0874] Processing steps and specific explanations

[0875] Step 1:

[0876] The user issues a voice command such as "Record my blood pressure today."

[0877] Input: Human speech.

[0878] Specific action: The user speaks clearly near the device.

[0879] Step 2:

[0880] The device picks up audio through a microphone.

[0881] Input: Human speech.

[0882] Output: Analog audio signal.

[0883] Specific operation: A microphone picks up an audio signal.

[0884] Step 3:

[0885] The terminal converts the voice into digital data using a voice data processing module.

[0886] Input: Analog audio signal.

[0887] Output: Digital audio data.

[0888] Specific operation: The audio data processing module digitizes the signal.

[0889] Step 4:

[0890] The terminal transmits digital audio data to the server via the communication module.

[0891] Input: Digital audio data.

[0892] Output: Digital audio data sent to the server.

[0893] Specific operation: The communication module transmits data over the Internet.

[0894] Step 5:

[0895] The server uses a voice recognition engine to convert the received voice data into text data.

[0896] Input: Digital audio data.

[0897] Output: Text data.

[0898] Specific operation: Speech recognition software analyzes the voice data and converts it into text.

[0899] Step 6:

[0900] The server analyzes the text data using a natural language processing (NLP) module to understand the user's request.

[0901] Input: Text data.

[0902] Output: User requested information ("Record blood pressure data").

[0903] What it does: The NLP module extracts intent from the text.

[0904] Step 7:

[0905] The server accesses a database and records the user's blood pressure data.

[0906] Input: User requested information.

[0907] Output: Save results to database.

[0908] Specific operation: The server stores the blood pressure data in a database in the appropriate format.

[0909] Step 8:

[0910] The server uses a speech synthesis engine to convert the database update results into voice data and send it to the terminal.

[0911] Input: Database update results.

[0912] Output: Audio data.

[0913] Specific operation: The speech synthesis engine converts the text into speech and sends it to the terminal through the communication module.

[0914] Step 9:

[0915] The terminal uses a speaker to play audio data and convey information to the user.

[0916] Input: Audio data.

[0917] Output: Audio output ("Blood pressure data recorded").

[0918] Specific operation: The speaker plays the audio data and notifies the user.

[0919] Emergency response

[0920] Processing steps and specific explanations

[0921] Step 1:

[0922] The user shouts, "There's an emergency!"

[0923] Input: Human speech.

[0924] Specific action: The user shouts for help in an emergency.

[0925] Step 2:

[0926] The device picks up audio through a microphone.

[0927] Input: Human speech.

[0928] Output: Analog audio signal.

[0929] Specific action: The microphone picks up an emergency voice signal.

[0930] Step 3:

[0931] The terminal converts the voice into digital data using a voice data processing module.

[0932] Input: Analog audio signal.

[0933] Output: Digital audio data.

[0934] Specific operation: The audio data processing module digitizes the signal.

[0935] Step 4:

[0936] The terminal transmits digital audio data to the server via the communication module.

[0937] Input: Digital audio data.

[0938] Output: Digital audio data sent to the server.

[0939] Specific operation: The communication module transmits data over the Internet.

[0940] Step 5:

[0941] The server uses a voice recognition engine to convert the received voice data into text data.

[0942] Input: Digital audio data.

[0943] Output: Text data.

[0944] Specific operation: Speech recognition software analyzes the voice data and converts it into text.

[0945] Step 6:

[0946] The server analyzes the text data using a natural language processing (NLP) module and understands that it is an emergency.

[0947] Input: Text data.

[0948] Output: Emergency call request information.

[0949] Specific behavior: The NLP module extracts the emergency intent from the text.

[0950] Step 7:

[0951] The server retrieves the emergency contact information from the database and places an emergency call using an automated call system.

[0952] Input: Emergency call request information.

[0953] Output: Emergency call result.

[0954] Specific operation: The server automatically notifies emergency contacts.

[0955] Step 8:

[0956] The server converts the results of the emergency call into voice data using a voice synthesis engine and sends it to the terminal.

[0957] Input: Emergency call result.

[0958] Output: Audio data.

[0959] Specific operation: The speech synthesis engine converts the text into speech and sends it to the terminal through the communication module.

[0960] Step 9:

[0961] The terminal uses a speaker to play audio data and convey information to the user.

[0962] Input: Audio data.

[0963] Output: Audio output ("An emergency call has been made").

[0964] Specific action: The speaker plays audio data to notify the user that emergency measures have been taken.

[0965] Thus, the processing steps for quickly and accurately providing various functions based on the user's voice commands have been specifically described.

[0966] (Application example 1)

[0967] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0968] There are situations where it is difficult for consumers to quickly and accurately obtain information about products and services in physical stores, and where it is difficult to properly manage employee health and respond to emergencies. For these reasons, there is a need for a system that can provide information quickly and accurately, manage employee health, and effectively respond to emergencies.

[0969] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0970] In this invention, the server includes an acquisition means for acquiring a user's voice command, a conversion means for converting the acquired voice command into voice data, a transmission means for transmitting the converted voice data to the server, an analysis means for analyzing the voice data on the server side and acquiring corresponding information, a synthesis means for converting the acquired information on the server side into voice, an output means for outputting the synthesized voice data to the user, a means for providing product information and service information on a device installed in the physical store, a means for recording and managing employee health data, and a means for detecting and reporting an emergency within the store. This improves the speed and accuracy of information provision within the physical store, supports employee health management, and enables rapid and effective emergency response.

[0971] The "acquisition means" is a means for acquiring a voice command from a user.

[0972] The "conversion means" is a means for converting the acquired voice command into voice data.

[0973] The "transmitting means" is a means for transmitting the converted voice data to the server.

[0974] The "analysis means" is a means for analyzing the voice data on the server side and acquiring the corresponding information.

[0975] The "synthesis means" is a means for converting information acquired on the server side into voice.

[0976] The "output means" is a means for outputting synthesized voice data to the user.

[0977] "Means for providing product information and service information" refers to means for providing product information and service information using devices installed in physical stores.

[0978] "Means for recording and managing employee health data" means means for recording and managing employee health data.

[0979] "Means for detecting and reporting emergencies within the store" refers to means for detecting emergencies within a physical store and reporting them promptly.

[0980] This invention is a system that uses IoT devices that utilize voice recognition to provide information in a physical store, manage the health of employees, and make emergency calls. Below, we will explain how to specifically implement this system.

[0981] System Configuration

[0982] The system consists of both terminals (edge ​​devices) and servers (backends).

[0983] Terminal (edge ​​device) side

[0984] 1. Microphone and voice input module: to capture the user's voice commands.

[0985] 2. Voice data processing module: converts the captured voice commands into digital form.

[0986] 3. Communication module: Transmits the converted audio data to the server using wireless communication technologies such as Wi-Fi or 4G / 5G.

[0987] 4. Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[0988] Server (backend) side

[0989] 1. Speech recognition engine: Converts voice data sent from the device into text data. Google Speech Recognition API is used.

[0990] 2. Natural Language Processing (NLP) module: Analyzes text data and understands user requirements. This analysis is performed using a generative AI model.

[0991] 3. Information engine: Obtains information based on user requests and generates the necessary response.

[0992] 4. Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[0993] 5. Database: Stores and manages employee health data and emergency call information.

[0994] Program processing

[0995] The server performs a series of processes including the following steps.

[0996] 1. A means of acquiring the user's voice commands.

[0997] 2. A conversion means for converting the acquired voice command into voice data.

[0998] 3. A transmitting means for transmitting the converted audio data to the server.

[0999] 4. An analysis means for analyzing the voice data on the server side and obtaining the corresponding information.

[1000] 5. A synthesis means for converting the information obtained on the server side into speech.

[1001] 6. Output means for outputting the synthesized voice data to the user.

[1002] 7. A means of providing product and service information on devices installed in physical stores.

[1003] 8. Means for recording and managing employee health data.

[1004] 9. A means of detecting and reporting emergencies within the store.

[1005] This will improve the speed and accuracy of information provision in physical stores, support employee health management, and enable quick and effective response in emergencies.

[1006] Examples of concrete examples and prompts

[1007] Providing product information

[1008] When a user says, "Tell me more about this product," the system provides detailed information about the product.

[1009] Example prompt: "Tell me more about this product."

[1010] Health management

[1011] When an employee says, "Record my blood pressure today," the system records the blood pressure information in a database to help manage their health.

[1012] Example prompt: "Record your blood pressure today."

[1013] Emergency response

[1014] When a user says "an emergency has occurred," the system will quickly make an emergency call and prompt appropriate response.

[1015] Example prompt: "An emergency has occurred."

[1016] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1017] Step 1:

[1018] The user issues a voice command.

[1019] Input: User utterance

[1020] Output: Audio data

[1021] Step 2:

[1022] The terminal uses a microphone and a voice input module to obtain voice commands from the user.

[1023] Input: User's voice

[1024] Output: Analog audio signal

[1025] What it does: A microphone captures sound and generates an analog audio signal.

[1026] Step 3:

[1027] The voice data processing module of the terminal converts the analog voice signal into digital form.

[1028] Input: Analog audio signal

[1029] Output: Digital audio data

[1030] Specific operation: AD conversion (analog-to-digital conversion) processing is performed to convert analog signals into digital signals.

[1031] Step 4:

[1032] The communication module of the terminal transmits the digital voice data to the server.

[1033] Input: Digital audio data

[1034] Output: Audio data sent to the server

[1035] Specific operation: Digital audio data is uploaded to a server using a wireless network (Wi-Fi, 4G / 5G, etc.).

[1036] Step 5:

[1037] The server converts the received digital voice data into text data using a voice recognition engine.

[1038] Input: Digital audio data

[1039] Output: Text data

[1040] Specific behavior: Uses the Google Speech Recognition API to convert audio data into text.

[1041] Step 6:

[1042] The server's natural language processing (NLP) module analyzes the text data and understands the user's request.

[1043] Input: Text data

[1044] Output: Interpretation of user requests

[1045] Specific operation: Using a generative AI model, the text data is analyzed to identify the requested content (product information, health data recording, emergency call, etc.).

[1046] Step 7:

[1047] The server's information providing engine acquires the necessary information based on the user's request and generates a response.

[1048] Input: Interpretation of user requests

[1049] Output: Response data

[1050] Specific operation: For example, when providing product information, retrieve related information from the product database and generate a response.

[1051] Step 8:

[1052] The server's voice synthesis engine converts the response data into voice data and sends it to the terminal.

[1053] Input: Response data

[1054] Output: Audio data

[1055] Specific operation: Text-to-speech (TTS) is performed and the audio data is sent to the terminal.

[1056] Step 9:

[1057] The terminal receives the transmitted voice data and conveys information to the user using a speaker and a voice output module.

[1058] Input: Audio data

[1059] Output: Audio output

[1060] Specific operation: Plays the received audio data and outputs information to the user through the speaker.

[1061] This allows for a fast and accurate response to the user's voice commands.

[1062] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1063] This invention adds emotion recognition functionality to a voice recognition IoT device that analyzes a user's voice commands and provides information, health management, and emergency notification functions, providing appropriate feedback and support according to the user's emotional state. This system includes an edge terminal and a back-end server for executing a series of processes from voice command acquisition and analysis to information provision, emotion recognition, and emergency response.

[1064] System Configuration

[1065] 1. Terminal (edge ​​device) side

[1066] Microphone and voice input module: to capture the user's voice commands.

[1067] Voice data processing module: converts voice data into digital format and sends it to the server.

[1068] Communication module: Sends voice data to the server and receives responses from the server.

[1069] Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[1070] 2. Server (backend) side

[1071] Speech recognition engine: Converts voice data sent by the user into text data.

[1072] Natural Language Processing (NLP) module: Analyzes text data and understands user requirements.

[1073] Information provision engine: Obtains information based on user requests and generates the required response.

[1074] Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[1075] Database: Stores and manages user health data and emergency contact information.

[1076] Emotion engine: Analyzes the user's emotions from voice data and generates emotional information.

[1077] Feedback engine: Generates appropriate feedback to users based on emotional information.

[1078] Monitoring system: A system for monitoring the user's health and mental state.

[1079] Program processing and specific examples

[1080] Voice input recognition and information provision

[1081] A user speaks to the terminal, "What's the weather like today?" This voice command is picked up by the microphone and digitized by the voice data processing module. This digital voice data is sent to the server through the communication module.

[1082] The server uses a speech recognition engine to convert the received voice data into text data. Next, the NLP module analyzes the text data and understands the user's request, "I want weather information." The server uses the information provision engine to call the weather information API and obtain the required weather information. The emotion engine then analyzes the user's emotion from the voice data and generates emotion information such as "curiosity" or "confusion." Based on this, the feedback engine adjusts the text message and adds positive feedback such as "The weather is sunny today. Have a nice day." The message is then converted into voice data by the speech synthesis engine and sent to the device. It is then audibly conveyed to the user through the device's speaker.

[1083] Support for health management

[1084] The user says, "Record my blood pressure today." The voice command is captured by the device and sent to the server. Similarly, the voice data is analyzed by the speech recognition engine and NLP module, and the request "record blood pressure data" is understood.

[1085] The server accesses the database and records the user's blood pressure data. Furthermore, an emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "anxiety" or "worry." A feedback engine generates feedback messages to reassure the user. The generated information is converted into voice data through a voice synthesis engine, and the user is notified via their device that "Your blood pressure has been recorded. Please relax."

[1086] Emergency response

[1087] The user shouts "An emergency has occurred." The voice command is immediately captured by the device and sent to the server as emergency data. The server uses a speech recognition engine to convert it into text data, and an NLP module analyzes it to indicate an emergency.

[1088] The server retrieves emergency contact information from the database and uses an automatic notification system to notify the designated emergency contact. At the same time, an emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "fear" or "tension." Based on this, a feedback engine generates a feedback message to reassure the user. The message, "An emergency call has been made. Please wait until the ambulance arrives," is converted into voice data by a speech synthesis engine and sent to the terminal for transmission to the user.

[1089] In this way, adding emotion recognition functionality enables flexible responses according to the user's emotional state, improving the user experience.

[1090] The processing flow will be explained below.

[1091] Voice input recognition and information provision

[1092] Step 1:

[1093] The user speaks the voice command "What's the weather like today?" into the device.

[1094] Step 2:

[1095] The device uses a built-in microphone to record voice commands.

[1096] Step 3:

[1097] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[1098] Step 4:

[1099] The terminal transmits the converted voice data to a server via the Internet.

[1100] Step 5:

[1101] The server's voice recognition engine converts the received voice data into text data.

[1102] Step 6:

[1103] The server's natural language processing (NLP) module analyzes the text data "What's the weather like today?" and understands the user's request.

[1104] Step 7:

[1105] The server calls the weather information API based on the analysis results and obtains current weather information.

[1106] Step 8:

[1107] The server's information engine generates a text message saying "Today's weather is sunny" based on the acquired weather information.

[1108] Step 9:

[1109] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "curiosity" or "confusion."

[1110] Step 10:

[1111] The server's feedback engine generates feedback appropriate for the user based on the emotional information, generating a message such as "The weather is sunny today. Have a nice day."

[1112] Step 11:

[1113] A speech synthesis engine on the server converts the generated text message into voice data.

[1114] Step 12:

[1115] The server transmits the converted audio data to the terminal via the Internet.

[1116] Step 13:

[1117] The terminal's audio output module plays the received data over the speaker and tells the user in a voice message, "Today's weather is sunny. Have a nice day."

[1118] Support for health management

[1119] Step 1:

[1120] The user speaks to the terminal, saying, "Record my blood pressure today."

[1121] Step 2:

[1122] The device will record your voice commands using the built-in microphone.

[1123] Step 3:

[1124] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[1125] Step 4:

[1126] The terminal transmits the converted voice data to a server via the Internet.

[1127] Step 5:

[1128] The server's voice recognition engine converts the received voice data into text data.

[1129] Step 6:

[1130] The server's natural language processing (NLP) module analyzes the text data "Record my blood pressure today" and understands the user's request.

[1131] Step 7:

[1132] The server accesses the user's physical condition database and records the blood pressure data.

[1133] Step 8:

[1134] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "anxiety" or "worry."

[1135] Step 9:

[1136] The server's feedback engine generates a feedback message based on the emotional information to reassure the user, such as "Your blood pressure has been recorded. Please relax."

[1137] Step 10:

[1138] A speech synthesis engine on the server converts the generated text message into voice data.

[1139] Step 11:

[1140] The server transmits the converted audio data to the terminal via the Internet.

[1141] Step 12:

[1142] The device's audio output module plays the received data over the speaker and tells the user in a voice message, "Your blood pressure has been recorded. Please relax."

[1143] Emergency response

[1144] Step 1:

[1145] The user shouts into the terminal, "An emergency has occurred."

[1146] Step 2:

[1147] The device will record emergency voice commands using the built-in microphone.

[1148] Step 3:

[1149] The voice data processing module of the terminal converts the recorded voice into digital data and passes it to the emergency processing module.

[1150] Step 4:

[1151] The terminal immediately transmits the formatted emergency voice data to the server.

[1152] Step 5:

[1153] The server's voice recognition engine converts the received emergency voice data into text data.

[1154] Step 6:

[1155] The server's natural language processing (NLP) module analyzes the text data "An emergency has occurred" and understands that it is an emergency.

[1156] Step 7:

[1157] The server retrieves the emergency contact information from the database and sends an emergency message to the emergency contact using an automated notification system.

[1158] Step 8:

[1159] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "fear" or "tension."

[1160] Step 9:

[1161] The server's feedback engine generates a reassuring feedback message based on the emotional information, such as "An emergency call has been made. Please wait until the ambulance arrives."

[1162] Step 10:

[1163] A speech synthesis engine on the server converts the generated feedback message into voice data.

[1164] Step 11:

[1165] The server transmits the converted audio data to the terminal via the Internet.

[1166] Step 12:

[1167] The terminal's audio output module plays the received data over the speaker and tells the user in a voice message, "An emergency call has been made. Please wait until the ambulance arrives."

[1168] Example 2

[1169] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1170] Conventional voice recognition systems only analyze users' voice commands and provide information, but are unable to provide appropriate feedback or support based on the user's emotional state. This makes it difficult to improve the user experience or respond quickly and appropriately in emergencies.

[1171] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1172] In this invention, the server includes a speech recognition means for converting voice data into text data, a natural language processing means for analyzing the text data and understanding the request, and an information providing means for acquiring information based on the request and generating a response. This enables not only the provision of information based on the user's voice command, but also appropriate feedback based on emotion recognition and a rapid response in emergencies.

[1173] The "acquisition means" is a device or module for acquiring a user's voice command.

[1174] The "conversion means" is a device or module for converting the acquired audio data into digital data.

[1175] The "transmitting means" is a communication device or module for transmitting the converted voice data to the server.

[1176] "Speech recognition means" refers to a device or algorithm that converts voice data into text data on the server side.

[1177] "Natural language processing means" refers to devices and algorithms that analyze text data and understand user requests.

[1178] An "information providing means" is a device or algorithm for obtaining information based on a user request and generating a response.

[1179] A "voice synthesis means" is a device or algorithm for converting a generated response into voice data.

[1180] "Output means" refers to a device or module for outputting audio data to the user.

[1181] "Emotion recognition means" refers to a device or algorithm for analyzing a user's emotions from voice data.

[1182] The "feedback means" refers to a device or algorithm for generating appropriate feedback to the user based on emotional information.

[1183] The "support means" is a device or system that supports the user's health management using the health data and emotion information acquired on the server side.

[1184] The "reporting means" refers to a device or system that quickly recognizes a user's voice command in an emergency and immediately makes an emergency call.

[1185] This invention is a system that adds emotion recognition functionality to a voice recognition IoT device that analyzes a user's voice commands and realizes functions such as providing information, managing one's health, and making an emergency call, and provides appropriate feedback and support according to the user's emotional state. Here, an embodiment of the present invention will be specifically described.

[1186] System Configuration

[1187] Terminal (edge ​​device) side

[1188] Microphone and voice input module: Acquires the user's voice commands by capturing the user's voice as an analog signal.

[1189] Audio data processing module: Converts the captured analog audio signal into digital format. Samples the audio signal and encodes it into digital data.

[1190] Communication module: Sends digital audio data to the server and receives responses from the server. Uses Wi-Fi or mobile data.

[1191] Speaker and audio output module: Plays back audio data received from the server and conveys information to the user. Converts digital audio data into an analog signal and plays it as sound through a speaker.

[1192] Server (backend) side

[1193] Speech recognition engine: Converts voice data sent from the device into text data. It uses machine learning models to analyze the voice data and convert it into text format.

[1194] Natural Language Processing (NLP) module: Analyzes text data to understand user requirements, analyzes text data grammatically, and extracts keywords and context.

[1195] Information engine: Obtains information based on user requests and generates responses. Specifically, it accesses weather information APIs and other sources to obtain the necessary information.

[1196] Speech synthesis engine: Converts the generated response into voice data and sends it to the device. It converts text information into natural-sounding voice using speech synthesis technology.

[1197] Database: Stores and manages user health data and emergency contact information.

[1198] Emotion Engine: Analyzes user emotions from voice data and generates emotional information. Analyzes voice tone and word usage to identify emotional state.

[1199] Feedback Engine: Generates appropriate feedback based on emotional information. Adjusts the tone and content of messages depending on the emotional state.

[1200] Monitoring system: A system that monitors the user's health and mental state.

[1201] Specific examples

[1202] Specific examples of voice input recognition and information provision

[1203] The user speaks to the device, "What's the weather like today?" The device's microphone picks up this voice, and the voice data processing module converts the analog voice signal into digital data. The communication module sends the digital voice data to the server.

[1204] The server's speech recognition engine converts the voice data into text data, and the NLP module analyzes the text to understand the request, "I want weather information." The information provision engine calls the weather information API to obtain the required information. The emotion engine analyzes emotions such as "curiosity" from the voice data, and the feedback engine generates a positive message such as "The weather is sunny today. Have a nice day."

[1205] The speech synthesis engine converts this message into voice data and sends it to the terminal via the communication module, where the terminal's speaker plays the voice back to the user.

[1206] example:

[1207] User: "What's the weather like today?"

[1208] Server: "The weather is sunny today. Have a nice day."

[1209] Specific examples of support for health management

[1210] The user says, "Record my blood pressure today." The terminal receives the voice command, and the voice data processing module converts it into digital data and sends it to the server.

[1211] The server's speech recognition engine and NLP module analyze the voice data and understand the request to "record blood pressure data." The database is accessed and the user's blood pressure data is recorded. The emotion engine analyzes emotions such as "anxiety" and "worry," and the feedback engine generates a message saying, "Your blood pressure has been recorded. Please relax."

[1212] The speech synthesis engine converts this message into voice data, sends it to the device, and plays it back to the user through the device's speaker.

[1213] example:

[1214] User: "Log my blood pressure today."

[1215] Server: "Your blood pressure has been recorded. Please relax."

[1216] Specific examples of emergency response

[1217] The user shouts, "An emergency has occurred." The device immediately receives the voice command and sends it to the server as emergency data.

[1218] The server's speech recognition engine converts the voice data into text data, and the NLP module analyzes the emergency situation. Emergency contact information is retrieved from the database, and the automated notification system notifies the emergency contacts specified in the database. At the same time, the emotion engine analyzes emotions such as "fear" and "tension," and the feedback engine generates a message saying, "An emergency call has been made. Please wait until the ambulance arrives."

[1219] The speech synthesis engine converts this message into voice data, sends it to the device, and plays it back to the user through the device's speaker.

[1220] example:

[1221] User: "We have an emergency."

[1222] Server: "We've called emergency services. Please wait until the ambulance arrives."

[1223] In this way, this system executes a series of processes from acquiring voice commands to analyzing them, providing information, recognizing emotions, and responding to emergencies, enabling flexible responses according to the user's emotional state.

[1224] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1225] Step 1:

[1226] The user issues a voice command. The input is the user's voice command (e.g., "What's the weather today?"). The device's microphone and voice input module pick up this voice command and capture it as an analog voice signal. The output is an analog voice signal.

[1227] Step 2:

[1228] The terminal's voice data processing module converts the acquired analog voice signal into digital data. The input is an analog voice signal and the output is digital voice data. Specifically, the voice signal is sampled and encoded into a digital format (bit string).

[1229] Step 3:

[1230] The communication module of the device sends digital audio data to the server. The input is digital audio data, and the output is digital audio data sent to the server. Data transmission uses Wi-Fi or mobile data communication.

[1231] Step 4:

[1232] The server's speech recognition engine converts the received digital voice data into text data. The input is the digital voice data sent to the server, and the output is text data. Specifically, it uses a machine learning model to analyze phonemes and context and convert the data into text format.

[1233] Step 5:

[1234] The server's natural language processing (NLP) module analyzes the text data and understands the user's request. The input is text data, and the output is the analyzed request (e.g., "Get weather information"). Specifically, it analyzes the text data grammatically and extracts keywords and context.

[1235] Step 6:

[1236] The server's information engine retrieves information based on the user's request and generates a response. The input is the analyzed request (retrieving weather information), and the output is a response based on the retrieved information (e.g., "Today's weather is sunny"). Specifically, it accesses the weather information API and retrieves today's weather information.

[1237] Step 7:

[1238] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information. The input is the user's voice data, and the output is emotional information (e.g., "curiosity"). Specific operations include analyzing the voice tone and word usage to identify the emotional state.

[1239] Step 8:

[1240] The server's feedback engine generates appropriate feedback based on the emotional information. The input is the emotional information and acquired information (e.g., "Curiosity" and "The weather is sunny today"), and the output is a feedback message (e.g., "The weather is sunny today. Have a nice day."). Specific behavior is to adjust the tone and content of the message depending on the emotional state.

[1241] Step 9:

[1242] The server's speech synthesis engine converts the feedback message into speech data. The input is the feedback message (text) and the output is speech data. Specifically, the text information is converted into natural speech using speech synthesis technology.

[1243] Step 10:

[1244] The communication module of the server transmits the voice data to the terminal. The input is the voice data, and the output is the voice data transmitted to the terminal.

[1245] Step 11:

[1246] The device's speaker and audio output module plays the audio data received from the server and provides feedback to the user. The input is audio data, and the output is the audio being played. Specifically, the digital audio data is converted into an analog signal and played as sound through the speaker.

[1247] example:

[1248] User: "What's the weather like today?"

[1249] Server: "The weather is sunny today. Have a nice day."

[1250] (Application example 2)

[1251] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1252] Conventional voice recognition systems can analyze a user's voice commands and provide information, but they do not provide feedback or support that takes into account the user's emotional state. As a result, they are unable to respond flexibly to the user's emotional state, and there is a lack of appropriate support to alleviate the user's anxiety and fear, especially in emergencies. There is a need to solve this problem.

[1253] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1254] In this invention, the server includes emotion analysis means for analyzing emotions from the user's voice, feedback generation means for generating appropriate feedback based on the emotion information, support means for supporting the user's health management, and reporting means for quickly recognizing the user's voice command in an emergency and immediately reporting to an emergency service, thereby enabling flexible responses and appropriate support according to the user's voice command as well as their emotional state.

[1255] The "acquisition means" is a device or mechanism that acquires a user's voice command.

[1256] The "conversion means" is a device or mechanism that converts the acquired voice command into digital voice data.

[1257] The "transmitting means" is a device or mechanism that transmits the converted voice data to the server.

[1258] The "analysis means" is a device or mechanism that analyzes the voice data received on the server side and acquires the corresponding information.

[1259] The "synthesizing means" is a device or mechanism that converts the information acquired on the server side into voice data.

[1260] The "output means" is a device or mechanism that outputs synthesized voice data to the user.

[1261] The "emotion analysis means" is a device or mechanism that analyzes emotions from the user's voice.

[1262] The "feedback generating means" is a device or mechanism that generates appropriate feedback based on emotional information.

[1263] The "support means" is a device or mechanism that supports the user's health management using information acquired on the server side.

[1264] The "notification means" is a device or mechanism that quickly recognizes a user's voice command in an emergency and immediately makes an emergency call.

[1265] This invention adds emotion recognition functionality to a voice recognition IoT system that analyzes users' voice commands and provides information, health management, and emergency notification functions, thereby providing appropriate feedback and support according to the user's emotional state. This system consists of a terminal (edge ​​device) and a server (backend).

[1266] System Configuration

[1267] 1. Terminal (edge ​​device) side

[1268] Microphone and voice input module: A device that captures the user's voice commands, which are converted into a digital format by a conversion means, which will be described later.

[1269] Voice data processing module: A device that converts the captured voice commands into digital format and transmits them to the server.

[1270] Communication module: A device that transmits converted voice data to a server and receives responses from the server.

[1271] Speaker and audio output module: A device that plays back audio data received from the server and conveys information to the user.

[1272] 2. Server (backend) side

[1273] Speech recognition engine: Software that converts voice data sent by a user into text data, so that the user's commands can be analyzed.

[1274] Natural Language Processing (NLP) module: Software that analyzes text data and understands user requests, enabling it to provide appropriate information and take appropriate action.

[1275] Information provision engine: Software that obtains information based on user requests and generates the necessary responses.

[1276] Speech synthesis engine: Software that converts the generated response into voice data and sends it to the terminal.

[1277] Database: A device that stores and manages a user's health data and emergency contact information.

[1278] Emotion analysis means: A device that analyzes the user's emotions from voice data and generates emotional information.

[1279] Feedback generation means: Software that generates appropriate feedback to the user based on emotional information.

[1280] Monitoring system: A system for monitoring the user's health and mental state.

[1281] Support means: A device that uses the acquired information to support the user in managing their physical condition.

[1282] Notification means: A device that quickly recognizes the user's voice command in an emergency and immediately makes an emergency call.

[1283] System processing example

[1284] Voice input recognition and information provision

[1285] For example, when a user says, "What's the weather today?", the voice command is picked up by the microphone and converted into digital form by the voice data processing module. This digital voice data is sent to the server through the communication module. The server uses a voice recognition engine to convert the received voice data into text data. The NLP module then analyzes the text data and understands the user's request, "I want weather information." The information provision engine obtains the weather information, and the emotion analysis means analyzes the user's emotions to generate appropriate feedback and send it to the terminal as voice data. The terminal's speaker then tells the user, "The weather is sunny today. Have a nice day."

[1286] Support for health management

[1287] When a user says, "Record my blood pressure today," the voice command is captured and converted into digital voice data. This is sent to the server, where it is analyzed by a voice recognition engine and an NLP module to understand the user's request. The server accesses the database and records the blood pressure data. At the same time, an emotion analysis means analyzes the user's emotions and generates reassuring feedback. This notifies the user, "Your blood pressure has been recorded. Please relax."

[1288] Emergency response

[1289] When a user shouts "An emergency has occurred," the voice command is immediately captured and sent to the server as emergency data. The voice recognition engine and NLP module convert the voice command into text data, which is then analyzed to indicate an emergency. The server retrieves emergency contact information from the database and notifies the emergency contacts using the notification means. At the same time, the emotion analysis means analyzes the user's emotions and generates emotional information such as "fear" or "tension," thereby generating feedback to reassure the user. The message "An emergency call has been made. Please wait until the ambulance arrives" is converted into voice data, sent to the terminal, and conveyed to the user.

[1290] Prompt Sentence Examples

[1291] If the user says, "Did you lock the house?"

[1292] 1. The microphone picks up the voice and the speech recognition engine converts it into text: "Have you locked your house?"

[1293] 2. Emotion analysis determines that the user is "anxious."

[1294] 3. The command is parsed and a response is generated: "Check home security."

[1295] 4. The security status is queried from the server and the result is obtained.

[1296] 5. The feedback to the user, "The house is locked. Don't worry," is converted into audio data and played back.

[1297] This system can provide appropriate feedback based on the user's emotional state, improving the user's experience.

[1298] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1299] Step 1:

[1300] The user issues a voice command through a microphone. The microphone captures this voice. The input is the user's voice command, and the output is the captured analog voice data. Specifically, the user speaks a command such as "What's the weather like today?"

[1301] Step 2:

[1302] The terminal converts the captured analog voice data into a digital format using a voice data processing module. The input is analog voice data, and the output is digital voice data. Specifically, the analog data is converted into digital data through sampling and quantization.

[1303] Step 3:

[1304] The communication module of the terminal transmits the converted digital voice data to the server. The input is the digital voice data, and the output is data transmission to the server. The data is transferred using a communication protocol.

[1305] Step 4:

[1306] The server uses a speech recognition engine to convert the received digital voice data into text data. The input is digital voice data, and the output is text data. Specifically, the server analyzes the voice waveform and converts it into words.

[1307] Step 5:

[1308] The server's natural language processing (NLP) module analyzes text data and understands the user's request. The input is text data from the speech recognition engine, and the output is the user's request, which is the analysis result. Intent is identified using general-purpose deep learning models and rule-based analysis.

[1309] Step 6:

[1310] The server's information engine retrieves information based on the user's request and generates the necessary response. The input is the analysis result, and the output is the response information. For example, accessing a weather information API to retrieve current weather data.

[1311] Step 7:

[1312] The emotion analysis means of the server analyzes the user's emotion from the voice data. The input is the voice data, and the output is the user's emotion information. The emotion is identified by analyzing the voice tone, speed, and intonation.

[1313] Step 8:

[1314] The server's feedback generation means generates appropriate feedback for the user based on the emotion information. The input is emotion information and response information, and the output is adjusted feedback text. For example, a message such as "Today's weather is sunny. Have a nice day" is generated.

[1315] Step 9:

[1316] The server's speech synthesis engine converts the generated feedback text into speech data. The input is the feedback text, and the output is speech data. Text-to-speech (TTS) technology is used to convert the text into speech.

[1317] Step 10:

[1318] The communication module of the terminal receives the voice data received from the server, where the input is the voice data sent from the server and the output is the voice data received by the terminal.

[1319] Step 11:

[1320] The device's speaker and audio output module plays the received audio data and conveys information to the user. The input is audio data, and the output is the audio heard by the user. The audio is played on the device's built-in speaker.

[1321] Prompt Sentence Examples

[1322] If the user says, "Did you lock the house?"

[1323] 1. The microphone picks up the voice and the speech recognition engine converts it into text: "Have you locked your house?"

[1324] 2. Emotion analysis determines that the user is "anxious."

[1325] 3. The command is parsed and a response is generated: "Check home security."

[1326] 4. The security status is queried from the server and the result is obtained.

[1327] 5. The feedback to the user, "The house is locked. Don't worry," is converted into audio data and played back.

[1328] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1329] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1330] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1331] [Third embodiment]

[1332] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1333] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1334] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1335] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1336] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1337] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1338] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1339] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1340] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1341] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1342] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1343] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1344] This invention relates to a voice recognition IoT device and system that analyzes a user's voice commands and realizes functions such as providing information, managing health, and making emergency calls. This system includes an edge terminal and a back-end server that execute a series of processes from receiving a user's voice command to analyzing it, providing information, and responding to an emergency.

[1345] System Configuration

[1346] 1. Terminal (edge ​​device) side

[1347] Microphone and voice input module: to capture the user's voice commands.

[1348] Voice data processing module: converts voice data into digital format and sends it to the server.

[1349] Communication module: Sends voice data to the server and receives responses from the server.

[1350] Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[1351] 2. Server (backend) side

[1352] Speech recognition engine: Converts voice data sent by the user into text data.

[1353] Natural Language Processing (NLP) module: Analyzes text data and understands user requirements.

[1354] Information provision engine: Obtains information based on user requests and generates the required response.

[1355] Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[1356] Database: Stores and manages user health data and emergency contact information.

[1357] Program processing and specific examples

[1358] Voice input recognition and information provision

[1359] A user speaks to the terminal, "What's the weather like today?" This voice command is picked up by the microphone and digitized by the voice data processing module. This digital voice data is sent to the server through the communication module.

[1360] The server uses a speech recognition engine to convert the received voice data into text data. Next, the NLP module analyzes the text data and understands the user's request, "I want weather information." The server uses the information provision engine to call the weather information API and obtain the required weather information. The speech synthesis engine then converts the weather information into voice data, generating the message, "Today's weather is sunny." This voice data is sent to the device and is conveyed to the user through the device's speaker.

[1361] Support for health management

[1362] The user says, "Record my blood pressure today." The voice command is captured by the device and sent to the server. Similarly, the voice data is analyzed by the speech recognition engine and NLP module, and the request "record blood pressure data" is understood.

[1363] The server accesses the database and records the user's blood pressure data. Furthermore, it monitors the user's health condition as needed and generates a warning message if an abnormality is detected. The generated information is converted into voice data through a voice synthesis engine and notified to the user via the device.

[1364] Emergency response

[1365] The user shouts "An emergency has occurred." The voice command is immediately captured by the device and sent to the server as emergency data. The server uses a speech recognition engine to convert it into text data, and an NLP module analyzes it to indicate an emergency.

[1366] The server retrieves emergency contact information from the database and notifies the designated emergency contacts using an automated notification system. At the same time, the server generates feedback information such as "An emergency call has been made" and converts it into voice data using a voice synthesis engine. This voice data is sent to the device and is played back to the user through the device's speaker.

[1367] In this way, a comprehensive system of voice-recognition IoT devices to support users' daily lives is realized.

[1368] The processing flow will be explained below.

[1369] Voice input recognition and information provision

[1370] Step 1:

[1371] The user speaks the voice command "What's the weather like today?" into the device.

[1372] Step 2:

[1373] The device uses a built-in microphone to record voice commands.

[1374] Step 3:

[1375] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[1376] Step 4:

[1377] The terminal transmits the converted voice data to a server via the Internet.

[1378] Step 5:

[1379] The server's voice recognition engine converts the received voice data into text data.

[1380] Step 6:

[1381] The server's natural language processing (NLP) module analyzes the text data "What's the weather like today?" and understands the user's request.

[1382] Step 7:

[1383] The server calls the weather information API based on the analysis results and obtains current weather information.

[1384] Step 8:

[1385] The server's information engine generates a text message saying "Today's weather is sunny" based on the acquired weather information.

[1386] Step 9:

[1387] A speech synthesis engine on the server converts the generated text message into voice data.

[1388] Step 10:

[1389] The server then transmits the converted voice data to the terminal again via the Internet.

[1390] Step 11:

[1391] The audio output module of the terminal plays the received data over the speaker and tells the user aloud, "Today's weather is sunny."

[1392] Support for health management

[1393] Step 1:

[1394] The user speaks to the terminal, saying, "Record my blood pressure today."

[1395] Step 2:

[1396] The device will record your voice commands using the built-in microphone.

[1397] Step 3:

[1398] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[1399] Step 4:

[1400] The terminal transmits the converted voice data to a server via the Internet.

[1401] Step 5:

[1402] The server's voice recognition engine converts the received voice data into text data.

[1403] Step 6:

[1404] The server's natural language processing (NLP) module analyzes the text data "Record my blood pressure today" and understands the user's request.

[1405] Step 7:

[1406] The server accesses the user's physical condition database and records the blood pressure data.

[1407] Step 8:

[1408] The server generates a feedback message indicating that the recording was successful, and converts it into voice data using a voice synthesis engine.

[1409] Step 9:

[1410] The server then transmits the converted voice data to the terminal again via the Internet.

[1411] Step 10:

[1412] The device's audio output module plays the received data over the speaker and tells the user in a voice message, "Your blood pressure has been recorded. Take care."

[1413] Emergency response

[1414] Step 1:

[1415] The user shouts into the terminal, "An emergency has occurred."

[1416] Step 2:

[1417] The device will record emergency voice commands using the built-in microphone.

[1418] Step 3:

[1419] The voice data processing module of the terminal converts the recorded voice into digital data and passes it to the emergency processing module.

[1420] Step 4:

[1421] The terminal immediately transmits the formatted emergency voice data to the server.

[1422] Step 5:

[1423] The server's voice recognition engine converts the received emergency voice data into text data.

[1424] Step 6:

[1425] The server's natural language processing (NLP) module analyzes the text data "An emergency has occurred" and understands that it is an emergency.

[1426] Step 7:

[1427] A server retrieves emergency contact information from a database and sends emergency messages to the contacts using an automated notification system.

[1428] Step 8:

[1429] The server checks the status of the call transmission, generates a feedback message such as "An emergency call has been made. Please wait until the ambulance arrives," and converts it into voice data using a speech synthesis engine.

[1430] Step 9:

[1431] The server then transmits the converted voice data to the terminal again via the Internet.

[1432] Step 10:

[1433] The terminal's audio output module plays the received data over the speaker and tells the user in a voice message, "An emergency call has been made. Please wait until the ambulance arrives."

[1434] Example 1

[1435] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1436] Conventional speech recognition systems have had difficulty responding quickly and accurately to diverse user requests. Furthermore, there has been a lack of technology to accurately analyze user requests and provide appropriate information, particularly in the areas of health management and emergency response. This has led to problems such as a loss of convenience and safety for users.

[1437] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1438] In this invention, the server includes a conversion means for converting voice data into text data, an analysis means for analyzing the text data and understanding the user's request, and an information provision engine for acquiring information based on the request and generating the necessary response. This enables quick and accurate responses to various user requests. It also enables appropriate data analysis and information provision for health management and emergency response.

[1439] The "acquisition means" is a module for detecting a user's voice command and acquiring it as voice data.

[1440] "Conversion means" is a module for converting the acquired voice commands into a digital format.

[1441] The "transmitting means" is a module for transmitting the converted digital audio data to the server.

[1442] "Text data" refers to data that has been analyzed and converted into text information from voice data.

[1443] A "voice recognition engine" is software for converting received voice data into text data.

[1444] A "natural language processing (NLP) module" is software that analyzes text data and understands user requests.

[1445] An "information engine" is software that acquires information based on a user's request and generates the necessary response.

[1446] The "voice synthesis means" is a module for converting the generated response into voice data.

[1447] The "output means" is a module for outputting audio data to the user.

[1448] The "monitoring means" is a module for monitoring the user's health condition and issuing a warning if an abnormality is detected.

[1449] An "automatic reporting system" is a system that recognizes a user's voice command in an emergency and immediately makes an emergency call.

[1450] This invention relates to a voice recognition IoT device and system that analyzes a user's voice commands and realizes functions such as providing information, managing health, and making emergency calls. This system includes an edge terminal and a back-end server that execute a series of processes from receiving a user's voice command to analyzing it, providing information, and responding to an emergency.

[1451] System Configuration

[1452] Terminal (edge ​​device) configuration

[1453] 1. Microphone and voice input module: to capture the user's voice commands.

[1454] 2. Audio data processing module: converts the captured audio into digital form.

[1455] 3. Communication module: Sends digital voice data to the server and receives responses from the server.

[1456] 4. Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[1457] Server (backend) configuration

[1458] 1. Speech recognition engine: Converts voice data sent by the user into text data.

[1459] 2. Natural Language Processing (NLP) module: Analyzes text data and understands user requirements.

[1460] 3. Information engine: Obtains information based on user requests and generates the necessary response.

[1461] 4. Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[1462] 5. Database: Stores and manages user health data and emergency contact information.

[1463] Voice input recognition and information provision

[1464] A user speaks to the terminal, "What's the weather like today?" This voice command is picked up by the microphone and digitized by the voice data processing module. This digital voice data is sent to the server through the communication module.

[1465] The server uses a speech recognition engine to convert the received voice data into text data. Next, the NLP module analyzes the text data and understands the user's request, "I want weather information." The server's information provision engine calls the weather information API and obtains the required weather information. The speech synthesis engine then converts the weather information into voice data, generating the message, "Today's weather is sunny." This voice data is sent to the device and conveyed to the user through the device's speaker.

[1466] Example prompt:

[1467] Get the voice command "What's the weather today?" and query the server for weather information.

[1468] Support for health management

[1469] The user says, "Record my blood pressure today." The voice command is captured by the device and sent to the server. Similarly, the voice data is analyzed by the speech recognition engine and NLP module, and the request "record blood pressure data" is understood.

[1470] The server accesses the database and records the user's blood pressure data. Furthermore, it monitors the user's health condition as needed and generates a warning message if an abnormality is detected. The generated information is converted into voice data through a voice synthesis engine and notified to the user via the device.

[1471] Example prompt:

[1472] When you receive the voice command "Record today's blood pressure," save the blood pressure data on your server.

[1473] Emergency response

[1474] The user shouts "An emergency has occurred." The voice command is immediately captured by the device and sent to the server as emergency data. The server uses a speech recognition engine to convert it into text data, and an NLP module analyzes it to indicate an emergency.

[1475] The server retrieves emergency contact information from the database and notifies the designated emergency contacts using an automated notification system. At the same time, the server generates feedback information such as "An emergency call has been made" and converts it into voice data using a voice synthesis engine. This voice data is sent to the device and is played back to the user through the device's speaker.

[1476] Example prompt:

[1477] Instantly process the voice command "an emergency has occurred" and make an emergency call to your designated emergency contacts.

[1478] In this way, a comprehensive voice-recognition IoT device system is realized to support users' daily lives, allowing them to easily use a variety of functions through voice commands and also enabling rapid response in emergencies.

[1479] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1480] Voice input recognition and information provision

[1481] Processing steps and specific explanations

[1482] Step 1:

[1483] The user issues a voice command: "What's the weather like today?"

[1484] Input: Human speech.

[1485] Specific action: The user speaks clearly near the device.

[1486] Step 2:

[1487] The device picks up audio through a microphone.

[1488] Input: Human speech.

[1489] Output: Analog audio signal.

[1490] Specific operation: A microphone picks up an audio signal.

[1491] Step 3:

[1492] The terminal converts the voice into digital data using a voice data processing module.

[1493] Input: Analog audio signal.

[1494] Output: Digital audio data.

[1495] Specific operation: The audio data processing module digitizes the signal.

[1496] Step 4:

[1497] The terminal transmits digital audio data to the server via the communication module.

[1498] Input: Digital audio data.

[1499] Output: Digital audio data sent to the server.

[1500] Specific operation: The communication module transmits data over the Internet.

[1501] Step 5:

[1502] The server uses a voice recognition engine to convert the received voice data into text data.

[1503] Input: Digital audio data.

[1504] Output: Text data.

[1505] Specific operation: Speech recognition software analyzes the voice data and converts it into text.

[1506] Step 6:

[1507] The server analyzes the text data using a natural language processing (NLP) module to understand the user's request.

[1508] Input: Text data.

[1509] Output: User request information ("I want weather information").

[1510] What it does: The NLP module extracts intent from the text.

[1511] Step 7:

[1512] The server uses an information provision engine to obtain the required information from the weather information API.

[1513] Input: User requested information.

[1514] Output: Weather information data (e.g. "Sunny").

[1515] Specific operation: The information engine sends an API request and gets a response.

[1516] Step 8:

[1517] The server converts the weather information obtained using a voice synthesis engine into voice data and sends it to the terminal.

[1518] Input: Weather information data.

[1519] Output: Audio data.

[1520] Specific operation: The speech synthesis engine converts the text into speech and sends it to the terminal through the communication module.

[1521] Step 9:

[1522] The terminal uses a speaker to play audio data and convey information to the user.

[1523] Input: Audio data.

[1524] Output: Audio output ("The weather is sunny today").

[1525] Specific operation: The speaker plays audio data and provides weather information to the user.

[1526] Support for health management

[1527] Processing steps and specific explanations

[1528] Step 1:

[1529] The user issues a voice command such as "Record my blood pressure today."

[1530] Input: Human speech.

[1531] Specific action: The user speaks clearly near the device.

[1532] Step 2:

[1533] The device picks up audio through a microphone.

[1534] Input: Human speech.

[1535] Output: Analog audio signal.

[1536] Specific operation: A microphone picks up an audio signal.

[1537] Step 3:

[1538] The terminal converts the voice into digital data using a voice data processing module.

[1539] Input: Analog audio signal.

[1540] Output: Digital audio data.

[1541] Specific operation: The audio data processing module digitizes the signal.

[1542] Step 4:

[1543] The terminal transmits digital audio data to the server via the communication module.

[1544] Input: Digital audio data.

[1545] Output: Digital audio data sent to the server.

[1546] Specific operation: The communication module transmits data over the Internet.

[1547] Step 5:

[1548] The server uses a voice recognition engine to convert the received voice data into text data.

[1549] Input: Digital audio data.

[1550] Output: Text data.

[1551] Specific operation: Speech recognition software analyzes the voice data and converts it into text.

[1552] Step 6:

[1553] The server analyzes the text data using a natural language processing (NLP) module to understand the user's request.

[1554] Input: Text data.

[1555] Output: User requested information ("Record blood pressure data").

[1556] What it does: The NLP module extracts intent from the text.

[1557] Step 7:

[1558] The server accesses a database and records the user's blood pressure data.

[1559] Input: User requested information.

[1560] Output: Save results to database.

[1561] Specific operation: The server stores the blood pressure data in a database in the appropriate format.

[1562] Step 8:

[1563] The server uses a speech synthesis engine to convert the database update results into voice data and send it to the terminal.

[1564] Input: Database update results.

[1565] Output: Audio data.

[1566] Specific operation: The speech synthesis engine converts the text into speech and sends it to the terminal through the communication module.

[1567] Step 9:

[1568] The terminal uses a speaker to play audio data and convey information to the user.

[1569] Input: Audio data.

[1570] Output: Audio output ("Blood pressure data recorded").

[1571] Specific operation: The speaker plays the audio data and notifies the user.

[1572] Emergency response

[1573] Processing steps and specific explanations

[1574] Step 1:

[1575] The user shouts, "There's an emergency!"

[1576] Input: Human speech.

[1577] Specific action: The user shouts for help in an emergency.

[1578] Step 2:

[1579] The device picks up audio through a microphone.

[1580] Input: Human speech.

[1581] Output: Analog audio signal.

[1582] Specific action: The microphone picks up an emergency voice signal.

[1583] Step 3:

[1584] The terminal converts the voice into digital data using a voice data processing module.

[1585] Input: Analog audio signal.

[1586] Output: Digital audio data.

[1587] Specific operation: The audio data processing module digitizes the signal.

[1588] Step 4:

[1589] The terminal transmits digital audio data to the server via the communication module.

[1590] Input: Digital audio data.

[1591] Output: Digital audio data sent to the server.

[1592] Specific operation: The communication module transmits data over the Internet.

[1593] Step 5:

[1594] The server uses a voice recognition engine to convert the received voice data into text data.

[1595] Input: Digital audio data.

[1596] Output: Text data.

[1597] Specific operation: Speech recognition software analyzes the voice data and converts it into text.

[1598] Step 6:

[1599] The server analyzes the text data using a natural language processing (NLP) module and understands that it is an emergency.

[1600] Input: Text data.

[1601] Output: Emergency call request information.

[1602] Specific behavior: The NLP module extracts the emergency intent from the text.

[1603] Step 7:

[1604] The server retrieves the emergency contact information from the database and places an emergency call using an automated call system.

[1605] Input: Emergency call request information.

[1606] Output: Emergency call result.

[1607] Specific operation: The server automatically notifies emergency contacts.

[1608] Step 8:

[1609] The server converts the results of the emergency call into voice data using a voice synthesis engine and sends it to the terminal.

[1610] Input: Emergency call result.

[1611] Output: Audio data.

[1612] Specific operation: The speech synthesis engine converts the text into speech and sends it to the terminal through the communication module.

[1613] Step 9:

[1614] The terminal uses a speaker to play audio data and convey information to the user.

[1615] Input: Audio data.

[1616] Output: Audio output ("An emergency call has been made").

[1617] Specific action: The speaker plays audio data to notify the user that emergency measures have been taken.

[1618] Thus, the processing steps for quickly and accurately providing various functions based on the user's voice commands have been specifically described.

[1619] (Application example 1)

[1620] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1621] There are situations where it is difficult for consumers to quickly and accurately obtain information about products and services in physical stores, and where it is difficult to properly manage employee health and respond to emergencies. For these reasons, there is a need for a system that can provide information quickly and accurately, manage employee health, and effectively respond to emergencies.

[1622] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1623] In this invention, the server includes an acquisition means for acquiring a user's voice command, a conversion means for converting the acquired voice command into voice data, a transmission means for transmitting the converted voice data to the server, an analysis means for analyzing the voice data on the server side and acquiring corresponding information, a synthesis means for converting the acquired information on the server side into voice, an output means for outputting the synthesized voice data to the user, a means for providing product information and service information on a device installed in the physical store, a means for recording and managing employee health data, and a means for detecting and reporting an emergency within the store. This improves the speed and accuracy of information provision within the physical store, supports employee health management, and enables rapid and effective emergency response.

[1624] The "acquisition means" is a means for acquiring a voice command from a user.

[1625] The "conversion means" is a means for converting the acquired voice command into voice data.

[1626] The "transmitting means" is a means for transmitting the converted voice data to the server.

[1627] The "analysis means" is a means for analyzing the voice data on the server side and acquiring the corresponding information.

[1628] The "synthesis means" is a means for converting information acquired on the server side into voice.

[1629] The "output means" is a means for outputting synthesized voice data to the user.

[1630] "Means for providing product information and service information" refers to means for providing product information and service information using devices installed in physical stores.

[1631] "Means for recording and managing employee health data" means means for recording and managing employee health data.

[1632] "Means for detecting and reporting emergencies within the store" refers to means for detecting emergencies within a physical store and reporting them promptly.

[1633] This invention is a system that uses IoT devices that utilize voice recognition to provide information in a physical store, manage the health of employees, and make emergency calls. Below, we will explain how to specifically implement this system.

[1634] System Configuration

[1635] The system consists of both terminals (edge ​​devices) and servers (backends).

[1636] Terminal (edge ​​device) side

[1637] 1. Microphone and voice input module: to capture the user's voice commands.

[1638] 2. Voice data processing module: converts the captured voice commands into digital form.

[1639] 3. Communication module: Transmits the converted audio data to the server using wireless communication technologies such as Wi-Fi or 4G / 5G.

[1640] 4. Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[1641] Server (backend) side

[1642] 1. Speech recognition engine: Converts voice data sent from the device into text data. Google Speech Recognition API is used.

[1643] 2. Natural Language Processing (NLP) module: Analyzes text data and understands user requirements. This analysis is performed using a generative AI model.

[1644] 3. Information engine: Obtains information based on user requests and generates the necessary response.

[1645] 4. Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[1646] 5. Database: Stores and manages employee health data and emergency call information.

[1647] Program processing

[1648] The server performs a series of processes including the following steps.

[1649] 1. A means of acquiring the user's voice commands.

[1650] 2. A conversion means for converting the acquired voice command into voice data.

[1651] 3. A transmitting means for transmitting the converted audio data to the server.

[1652] 4. An analysis means for analyzing the voice data on the server side and obtaining the corresponding information.

[1653] 5. A synthesis means for converting the information obtained on the server side into speech.

[1654] 6. Output means for outputting the synthesized voice data to the user.

[1655] 7. A means of providing product and service information on devices installed in physical stores.

[1656] 8. Means for recording and managing employee health data.

[1657] 9. A means of detecting and reporting emergencies within the store.

[1658] This will improve the speed and accuracy of information provision in physical stores, support employee health management, and enable quick and effective response in emergencies.

[1659] Examples of concrete examples and prompts

[1660] Providing product information

[1661] When a user says, "Tell me more about this product," the system provides detailed information about the product.

[1662] Example prompt: "Tell me more about this product."

[1663] Health management

[1664] When an employee says, "Record my blood pressure today," the system records the blood pressure information in a database to help manage their health.

[1665] Example prompt: "Record your blood pressure today."

[1666] Emergency response

[1667] When a user says "an emergency has occurred," the system will quickly make an emergency call and prompt appropriate response.

[1668] Example prompt: "An emergency has occurred."

[1669] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1670] Step 1:

[1671] The user issues a voice command.

[1672] Input: User utterance

[1673] Output: Audio data

[1674] Step 2:

[1675] The terminal uses a microphone and a voice input module to obtain voice commands from the user.

[1676] Input: User's voice

[1677] Output: Analog audio signal

[1678] What it does: A microphone captures sound and generates an analog audio signal.

[1679] Step 3:

[1680] The voice data processing module of the terminal converts the analog voice signal into digital form.

[1681] Input: Analog audio signal

[1682] Output: Digital audio data

[1683] Specific operation: AD conversion (analog-to-digital conversion) processing is performed to convert analog signals into digital signals.

[1684] Step 4:

[1685] The communication module of the terminal transmits the digital voice data to the server.

[1686] Input: Digital audio data

[1687] Output: Audio data sent to the server

[1688] Specific operation: Digital audio data is uploaded to a server using a wireless network (Wi-Fi, 4G / 5G, etc.).

[1689] Step 5:

[1690] The server converts the received digital voice data into text data using a voice recognition engine.

[1691] Input: Digital audio data

[1692] Output: Text data

[1693] Specific behavior: Uses the Google Speech Recognition API to convert audio data into text.

[1694] Step 6:

[1695] The server's natural language processing (NLP) module analyzes the text data and understands the user's request.

[1696] Input: Text data

[1697] Output: Interpretation of user requests

[1698] Specific operation: Using a generative AI model, the text data is analyzed to identify the requested content (product information, health data recording, emergency call, etc.).

[1699] Step 7:

[1700] The server's information providing engine acquires the necessary information based on the user's request and generates a response.

[1701] Input: Interpretation of user requests

[1702] Output: Response data

[1703] Specific operation: For example, when providing product information, retrieve related information from the product database and generate a response.

[1704] Step 8:

[1705] The server's voice synthesis engine converts the response data into voice data and sends it to the terminal.

[1706] Input: Response data

[1707] Output: Audio data

[1708] Specific operation: Text-to-speech (TTS) is performed and the audio data is sent to the terminal.

[1709] Step 9:

[1710] The terminal receives the transmitted voice data and conveys information to the user using a speaker and a voice output module.

[1711] Input: Audio data

[1712] Output: Audio output

[1713] Specific operation: Plays the received audio data and outputs information to the user through the speaker.

[1714] This allows for a fast and accurate response to the user's voice commands.

[1715] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1716] This invention adds emotion recognition functionality to a voice recognition IoT device that analyzes a user's voice commands and provides information, health management, and emergency notification functions, providing appropriate feedback and support according to the user's emotional state. This system includes an edge terminal and a back-end server for executing a series of processes from voice command acquisition and analysis to information provision, emotion recognition, and emergency response.

[1717] System Configuration

[1718] 1. Terminal (edge ​​device) side

[1719] Microphone and voice input module: to capture the user's voice commands.

[1720] Voice data processing module: converts voice data into digital format and sends it to the server.

[1721] Communication module: Sends voice data to the server and receives responses from the server.

[1722] Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[1723] 2. Server (backend) side

[1724] Speech recognition engine: Converts voice data sent by the user into text data.

[1725] Natural Language Processing (NLP) module: Analyzes text data and understands user requirements.

[1726] Information provision engine: Obtains information based on user requests and generates the required response.

[1727] Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[1728] Database: Stores and manages user health data and emergency contact information.

[1729] Emotion engine: Analyzes the user's emotions from voice data and generates emotional information.

[1730] Feedback engine: Generates appropriate feedback to users based on emotional information.

[1731] Monitoring system: A system for monitoring the user's health and mental state.

[1732] Program processing and specific examples

[1733] Voice input recognition and information provision

[1734] A user speaks to the terminal, "What's the weather like today?" This voice command is picked up by the microphone and digitized by the voice data processing module. This digital voice data is sent to the server through the communication module.

[1735] The server uses a speech recognition engine to convert the received voice data into text data. Next, the NLP module analyzes the text data and understands the user's request, "I want weather information." The server uses the information provision engine to call the weather information API and obtain the required weather information. The emotion engine then analyzes the user's emotion from the voice data and generates emotion information such as "curiosity" or "confusion." Based on this, the feedback engine adjusts the text message and adds positive feedback such as "The weather is sunny today. Have a nice day." The message is then converted into voice data by the speech synthesis engine and sent to the device. It is then audibly conveyed to the user through the device's speaker.

[1736] Support for health management

[1737] The user says, "Record my blood pressure today." The voice command is captured by the device and sent to the server. Similarly, the voice data is analyzed by the speech recognition engine and NLP module, and the request "record blood pressure data" is understood.

[1738] The server accesses the database and records the user's blood pressure data. Furthermore, an emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "anxiety" or "worry." A feedback engine generates feedback messages to reassure the user. The generated information is converted into voice data through a voice synthesis engine, and the user is notified via their device that "Your blood pressure has been recorded. Please relax."

[1739] Emergency response

[1740] The user shouts "An emergency has occurred." The voice command is immediately captured by the device and sent to the server as emergency data. The server uses a speech recognition engine to convert it into text data, and an NLP module analyzes it to indicate an emergency.

[1741] The server retrieves emergency contact information from the database and uses an automatic notification system to notify the designated emergency contact. At the same time, an emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "fear" or "tension." Based on this, a feedback engine generates a feedback message to reassure the user. The message, "An emergency call has been made. Please wait until the ambulance arrives," is converted into voice data by a speech synthesis engine and sent to the terminal for transmission to the user.

[1742] In this way, adding emotion recognition functionality enables flexible responses according to the user's emotional state, improving the user experience.

[1743] The processing flow will be explained below.

[1744] Voice input recognition and information provision

[1745] Step 1:

[1746] The user speaks the voice command "What's the weather like today?" into the device.

[1747] Step 2:

[1748] The device uses a built-in microphone to record voice commands.

[1749] Step 3:

[1750] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[1751] Step 4:

[1752] The terminal transmits the converted voice data to a server via the Internet.

[1753] Step 5:

[1754] The server's voice recognition engine converts the received voice data into text data.

[1755] Step 6:

[1756] The server's natural language processing (NLP) module analyzes the text data "What's the weather like today?" and understands the user's request.

[1757] Step 7:

[1758] The server calls the weather information API based on the analysis results and obtains current weather information.

[1759] Step 8:

[1760] The server's information engine generates a text message saying "Today's weather is sunny" based on the acquired weather information.

[1761] Step 9:

[1762] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "curiosity" or "confusion."

[1763] Step 10:

[1764] The server's feedback engine generates feedback appropriate for the user based on the emotional information, generating a message such as "The weather is sunny today. Have a nice day."

[1765] Step 11:

[1766] A speech synthesis engine on the server converts the generated text message into voice data.

[1767] Step 12:

[1768] The server transmits the converted audio data to the terminal via the Internet.

[1769] Step 13:

[1770] The terminal's audio output module plays the received data over the speaker and tells the user in a voice message, "Today's weather is sunny. Have a nice day."

[1771] Support for health management

[1772] Step 1:

[1773] The user speaks to the terminal, saying, "Record my blood pressure today."

[1774] Step 2:

[1775] The device will record your voice commands using the built-in microphone.

[1776] Step 3:

[1777] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[1778] Step 4:

[1779] The terminal transmits the converted voice data to a server via the Internet.

[1780] Step 5:

[1781] The server's voice recognition engine converts the received voice data into text data.

[1782] Step 6:

[1783] The server's natural language processing (NLP) module analyzes the text data "Record my blood pressure today" and understands the user's request.

[1784] Step 7:

[1785] The server accesses the user's physical condition database and records the blood pressure data.

[1786] Step 8:

[1787] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "anxiety" or "worry."

[1788] Step 9:

[1789] The server's feedback engine generates a feedback message based on the emotional information to reassure the user, such as "Your blood pressure has been recorded. Please relax."

[1790] Step 10:

[1791] A speech synthesis engine on the server converts the generated text message into voice data.

[1792] Step 11:

[1793] The server transmits the converted audio data to the terminal via the Internet.

[1794] Step 12:

[1795] The device's audio output module plays the received data over the speaker and tells the user in a voice message, "Your blood pressure has been recorded. Please relax."

[1796] Emergency response

[1797] Step 1:

[1798] The user shouts into the terminal, "An emergency has occurred."

[1799] Step 2:

[1800] The device will record emergency voice commands using the built-in microphone.

[1801] Step 3:

[1802] The voice data processing module of the terminal converts the recorded voice into digital data and passes it to the emergency processing module.

[1803] Step 4:

[1804] The terminal immediately transmits the formatted emergency voice data to the server.

[1805] Step 5:

[1806] The server's voice recognition engine converts the received emergency voice data into text data.

[1807] Step 6:

[1808] The server's natural language processing (NLP) module analyzes the text data "An emergency has occurred" and understands that it is an emergency.

[1809] Step 7:

[1810] The server retrieves the emergency contact information from the database and sends an emergency message to the emergency contact using an automated notification system.

[1811] Step 8:

[1812] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "fear" or "tension."

[1813] Step 9:

[1814] The server's feedback engine generates a reassuring feedback message based on the emotional information, such as "An emergency call has been made. Please wait until the ambulance arrives."

[1815] Step 10:

[1816] A speech synthesis engine on the server converts the generated feedback message into voice data.

[1817] Step 11:

[1818] The server transmits the converted audio data to the terminal via the Internet.

[1819] Step 12:

[1820] The terminal's audio output module plays the received data over the speaker and tells the user in a voice message, "An emergency call has been made. Please wait until the ambulance arrives."

[1821] Example 2

[1822] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1823] Conventional voice recognition systems only analyze users' voice commands and provide information, but are unable to provide appropriate feedback or support based on the user's emotional state. This makes it difficult to improve the user experience or respond quickly and appropriately in emergencies.

[1824] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1825] In this invention, the server includes a speech recognition means for converting voice data into text data, a natural language processing means for analyzing the text data and understanding the request, and an information providing means for acquiring information based on the request and generating a response. This enables not only the provision of information based on the user's voice command, but also appropriate feedback based on emotion recognition and a rapid response in emergencies.

[1826] The "acquisition means" is a device or module for acquiring a user's voice command.

[1827] The "conversion means" is a device or module for converting the acquired audio data into digital data.

[1828] The "transmitting means" is a communication device or module for transmitting the converted voice data to the server.

[1829] "Speech recognition means" refers to a device or algorithm that converts voice data into text data on the server side.

[1830] "Natural language processing means" refers to devices and algorithms that analyze text data and understand user requests.

[1831] An "information providing means" is a device or algorithm for obtaining information based on a user request and generating a response.

[1832] A "voice synthesis means" is a device or algorithm for converting a generated response into voice data.

[1833] "Output means" refers to a device or module for outputting audio data to the user.

[1834] "Emotion recognition means" refers to a device or algorithm for analyzing a user's emotions from voice data.

[1835] The "feedback means" refers to a device or algorithm for generating appropriate feedback to the user based on emotional information.

[1836] The "support means" is a device or system that supports the user's health management using the health data and emotion information acquired on the server side.

[1837] The "reporting means" refers to a device or system that quickly recognizes a user's voice command in an emergency and immediately makes an emergency call.

[1838] This invention is a system that adds emotion recognition functionality to a voice recognition IoT device that analyzes a user's voice commands and realizes functions such as providing information, managing one's health, and making an emergency call, and provides appropriate feedback and support according to the user's emotional state. Here, an embodiment of the present invention will be specifically described.

[1839] System Configuration

[1840] Terminal (edge ​​device) side

[1841] Microphone and voice input module: Acquires the user's voice commands by capturing the user's voice as an analog signal.

[1842] Audio data processing module: Converts the captured analog audio signal into digital format. Samples the audio signal and encodes it into digital data.

[1843] Communication module: Sends digital audio data to the server and receives responses from the server. Uses Wi-Fi or mobile data.

[1844] Speaker and audio output module: Plays back audio data received from the server and conveys information to the user. Converts digital audio data into an analog signal and plays it as sound through a speaker.

[1845] Server (backend) side

[1846] Speech recognition engine: Converts voice data sent from the device into text data. It uses machine learning models to analyze the voice data and convert it into text format.

[1847] Natural Language Processing (NLP) module: Analyzes text data to understand user requirements, analyzes text data grammatically, and extracts keywords and context.

[1848] Information engine: Obtains information based on user requests and generates responses. Specifically, it accesses weather information APIs and other sources to obtain the necessary information.

[1849] Speech synthesis engine: Converts the generated response into voice data and sends it to the device. It converts text information into natural-sounding voice using speech synthesis technology.

[1850] Database: Stores and manages user health data and emergency contact information.

[1851] Emotion Engine: Analyzes user emotions from voice data and generates emotional information. Analyzes voice tone and word usage to identify emotional state.

[1852] Feedback Engine: Generates appropriate feedback based on emotional information. Adjusts the tone and content of messages depending on the emotional state.

[1853] Monitoring system: A system that monitors the user's health and mental state.

[1854] Specific examples

[1855] Specific examples of voice input recognition and information provision

[1856] The user speaks to the device, "What's the weather like today?" The device's microphone picks up this voice, and the voice data processing module converts the analog voice signal into digital data. The communication module sends the digital voice data to the server.

[1857] The server's speech recognition engine converts the voice data into text data, and the NLP module analyzes the text to understand the request, "I want weather information." The information provision engine calls the weather information API to obtain the required information. The emotion engine analyzes emotions such as "curiosity" from the voice data, and the feedback engine generates a positive message such as "The weather is sunny today. Have a nice day."

[1858] The speech synthesis engine converts this message into voice data and sends it to the terminal via the communication module, where the terminal's speaker plays the voice back to the user.

[1859] example:

[1860] User: "What's the weather like today?"

[1861] Server: "The weather is sunny today. Have a nice day."

[1862] Specific examples of support for health management

[1863] The user says, "Record my blood pressure today." The terminal receives the voice command, and the voice data processing module converts it into digital data and sends it to the server.

[1864] The server's speech recognition engine and NLP module analyze the voice data and understand the request to "record blood pressure data." The database is accessed and the user's blood pressure data is recorded. The emotion engine analyzes emotions such as "anxiety" and "worry," and the feedback engine generates a message saying, "Your blood pressure has been recorded. Please relax."

[1865] The speech synthesis engine converts this message into voice data, sends it to the device, and plays it back to the user through the device's speaker.

[1866] example:

[1867] User: "Log my blood pressure today."

[1868] Server: "Your blood pressure has been recorded. Please relax."

[1869] Specific examples of emergency response

[1870] The user shouts, "An emergency has occurred." The device immediately receives the voice command and sends it to the server as emergency data.

[1871] The server's speech recognition engine converts the voice data into text data, and the NLP module analyzes the emergency situation. Emergency contact information is retrieved from the database, and the automated notification system notifies the emergency contacts specified in the database. At the same time, the emotion engine analyzes emotions such as "fear" and "tension," and the feedback engine generates a message saying, "An emergency call has been made. Please wait until the ambulance arrives."

[1872] The speech synthesis engine converts this message into voice data, sends it to the device, and plays it back to the user through the device's speaker.

[1873] example:

[1874] User: "We have an emergency."

[1875] Server: "We've called emergency services. Please wait until the ambulance arrives."

[1876] In this way, this system executes a series of processes from acquiring voice commands to analyzing them, providing information, recognizing emotions, and responding to emergencies, enabling flexible responses according to the user's emotional state.

[1877] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1878] Step 1:

[1879] The user issues a voice command. The input is the user's voice command (e.g., "What's the weather today?"). The device's microphone and voice input module pick up this voice command and capture it as an analog voice signal. The output is an analog voice signal.

[1880] Step 2:

[1881] The terminal's voice data processing module converts the acquired analog voice signal into digital data. The input is an analog voice signal and the output is digital voice data. Specifically, the voice signal is sampled and encoded into a digital format (bit string).

[1882] Step 3:

[1883] The communication module of the device sends digital audio data to the server. The input is digital audio data, and the output is digital audio data sent to the server. Data transmission uses Wi-Fi or mobile data communication.

[1884] Step 4:

[1885] The server's speech recognition engine converts the received digital voice data into text data. The input is the digital voice data sent to the server, and the output is text data. Specifically, it uses a machine learning model to analyze phonemes and context and convert the data into text format.

[1886] Step 5:

[1887] The server's natural language processing (NLP) module analyzes the text data and understands the user's request. The input is text data, and the output is the analyzed request (e.g., "Get weather information"). Specifically, it analyzes the text data grammatically and extracts keywords and context.

[1888] Step 6:

[1889] The server's information engine retrieves information based on the user's request and generates a response. The input is the analyzed request (retrieving weather information), and the output is a response based on the retrieved information (e.g., "Today's weather is sunny"). Specifically, it accesses the weather information API and retrieves today's weather information.

[1890] Step 7:

[1891] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information. The input is the user's voice data, and the output is emotional information (e.g., "curiosity"). Specific operations include analyzing the voice tone and word usage to identify the emotional state.

[1892] Step 8:

[1893] The server's feedback engine generates appropriate feedback based on the emotional information. The input is the emotional information and acquired information (e.g., "Curiosity" and "The weather is sunny today"), and the output is a feedback message (e.g., "The weather is sunny today. Have a nice day."). Specific behavior is to adjust the tone and content of the message depending on the emotional state.

[1894] Step 9:

[1895] The server's speech synthesis engine converts the feedback message into speech data. The input is the feedback message (text) and the output is speech data. Specifically, the text information is converted into natural speech using speech synthesis technology.

[1896] Step 10:

[1897] The communication module of the server transmits the voice data to the terminal. The input is the voice data, and the output is the voice data transmitted to the terminal.

[1898] Step 11:

[1899] The device's speaker and audio output module plays the audio data received from the server and provides feedback to the user. The input is audio data, and the output is the audio being played. Specifically, the digital audio data is converted into an analog signal and played as sound through the speaker.

[1900] example:

[1901] User: "What's the weather like today?"

[1902] Server: "The weather is sunny today. Have a nice day."

[1903] (Application example 2)

[1904] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1905] Conventional voice recognition systems can analyze a user's voice commands and provide information, but they do not provide feedback or support that takes into account the user's emotional state. As a result, they are unable to respond flexibly to the user's emotional state, and there is a lack of appropriate support to alleviate the user's anxiety and fear, especially in emergencies. There is a need to solve this problem.

[1906] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1907] In this invention, the server includes emotion analysis means for analyzing emotions from the user's voice, feedback generation means for generating appropriate feedback based on the emotion information, support means for supporting the user's health management, and reporting means for quickly recognizing the user's voice command in an emergency and immediately reporting to an emergency service, thereby enabling flexible responses and appropriate support according to the user's voice command as well as their emotional state.

[1908] The "acquisition means" is a device or mechanism that acquires a user's voice command.

[1909] The "conversion means" is a device or mechanism that converts the acquired voice command into digital voice data.

[1910] The "transmitting means" is a device or mechanism that transmits the converted voice data to the server.

[1911] The "analysis means" is a device or mechanism that analyzes the voice data received on the server side and acquires the corresponding information.

[1912] The "synthesizing means" is a device or mechanism that converts the information acquired on the server side into voice data.

[1913] The "output means" is a device or mechanism that outputs synthesized voice data to the user.

[1914] The "emotion analysis means" is a device or mechanism that analyzes emotions from the user's voice.

[1915] The "feedback generating means" is a device or mechanism that generates appropriate feedback based on emotional information.

[1916] The "support means" is a device or mechanism that supports the user's health management using information acquired on the server side.

[1917] The "notification means" is a device or mechanism that quickly recognizes a user's voice command in an emergency and immediately makes an emergency call.

[1918] This invention adds emotion recognition functionality to a voice recognition IoT system that analyzes users' voice commands and provides information, health management, and emergency notification functions, thereby providing appropriate feedback and support according to the user's emotional state. This system consists of a terminal (edge ​​device) and a server (backend).

[1919] System Configuration

[1920] 1. Terminal (edge ​​device) side

[1921] Microphone and voice input module: A device that captures the user's voice commands, which are converted into a digital format by a conversion means, which will be described later.

[1922] Voice data processing module: A device that converts the captured voice commands into digital format and transmits them to the server.

[1923] Communication module: A device that transmits converted voice data to a server and receives responses from the server.

[1924] Speaker and audio output module: A device that plays back audio data received from the server and conveys information to the user.

[1925] 2. Server (backend) side

[1926] Speech recognition engine: Software that converts voice data sent by a user into text data, so that the user's commands can be analyzed.

[1927] Natural Language Processing (NLP) module: Software that analyzes text data and understands user requests, enabling it to provide appropriate information and take appropriate action.

[1928] Information provision engine: Software that obtains information based on user requests and generates the necessary responses.

[1929] Speech synthesis engine: Software that converts the generated response into voice data and sends it to the terminal.

[1930] Database: A device that stores and manages a user's health data and emergency contact information.

[1931] Emotion analysis means: A device that analyzes the user's emotions from voice data and generates emotional information.

[1932] Feedback generation means: Software that generates appropriate feedback to the user based on emotional information.

[1933] Monitoring system: A system for monitoring the user's health and mental state.

[1934] Support means: A device that uses the acquired information to support the user in managing their physical condition.

[1935] Notification means: A device that quickly recognizes the user's voice command in an emergency and immediately makes an emergency call.

[1936] System processing example

[1937] Voice input recognition and information provision

[1938] For example, when a user says, "What's the weather today?", the voice command is picked up by the microphone and converted into digital form by the voice data processing module. This digital voice data is sent to the server through the communication module. The server uses a voice recognition engine to convert the received voice data into text data. The NLP module then analyzes the text data and understands the user's request, "I want weather information." The information provision engine obtains the weather information, and the emotion analysis means analyzes the user's emotions to generate appropriate feedback and send it to the terminal as voice data. The terminal's speaker then tells the user, "The weather is sunny today. Have a nice day."

[1939] Support for health management

[1940] When a user says, "Record my blood pressure today," the voice command is captured and converted into digital voice data. This is sent to the server, where it is analyzed by a voice recognition engine and an NLP module to understand the user's request. The server accesses the database and records the blood pressure data. At the same time, an emotion analysis means analyzes the user's emotions and generates reassuring feedback. This notifies the user, "Your blood pressure has been recorded. Please relax."

[1941] Emergency response

[1942] When a user shouts "An emergency has occurred," the voice command is immediately captured and sent to the server as emergency data. The voice recognition engine and NLP module convert the voice command into text data, which is then analyzed to indicate an emergency. The server retrieves emergency contact information from the database and notifies the emergency contacts using the notification means. At the same time, the emotion analysis means analyzes the user's emotions and generates emotional information such as "fear" or "tension," thereby generating feedback to reassure the user. The message "An emergency call has been made. Please wait until the ambulance arrives" is converted into voice data, sent to the terminal, and conveyed to the user.

[1943] Prompt Sentence Examples

[1944] If the user says, "Did you lock the house?"

[1945] 1. The microphone picks up the voice and the speech recognition engine converts it into text: "Have you locked your house?"

[1946] 2. Emotion analysis determines that the user is "anxious."

[1947] 3. The command is parsed and a response is generated: "Check home security."

[1948] 4. The security status is queried from the server and the result is obtained.

[1949] 5. The feedback to the user, "The house is locked. Don't worry," is converted into audio data and played back.

[1950] This system can provide appropriate feedback based on the user's emotional state, improving the user's experience.

[1951] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1952] Step 1:

[1953] The user issues a voice command through a microphone. The microphone captures this voice. The input is the user's voice command, and the output is the captured analog voice data. Specifically, the user speaks a command such as "What's the weather like today?"

[1954] Step 2:

[1955] The terminal converts the captured analog voice data into a digital format using a voice data processing module. The input is analog voice data, and the output is digital voice data. Specifically, the analog data is converted into digital data through sampling and quantization.

[1956] Step 3:

[1957] The communication module of the terminal transmits the converted digital voice data to the server. The input is the digital voice data, and the output is data transmission to the server. The data is transferred using a communication protocol.

[1958] Step 4:

[1959] The server uses a speech recognition engine to convert the received digital voice data into text data. The input is digital voice data, and the output is text data. Specifically, the server analyzes the voice waveform and converts it into words.

[1960] Step 5:

[1961] The server's natural language processing (NLP) module analyzes text data and understands the user's request. The input is text data from the speech recognition engine, and the output is the user's request, which is the analysis result. Intent is identified using general-purpose deep learning models and rule-based analysis.

[1962] Step 6:

[1963] The server's information engine retrieves information based on the user's request and generates the necessary response. The input is the analysis result, and the output is the response information. For example, accessing a weather information API to retrieve current weather data.

[1964] Step 7:

[1965] The emotion analysis means of the server analyzes the user's emotion from the voice data. The input is the voice data, and the output is the user's emotion information. The emotion is identified by analyzing the voice tone, speed, and intonation.

[1966] Step 8:

[1967] The server's feedback generation means generates appropriate feedback for the user based on the emotion information. The input is emotion information and response information, and the output is adjusted feedback text. For example, a message such as "Today's weather is sunny. Have a nice day" is generated.

[1968] Step 9:

[1969] The server's speech synthesis engine converts the generated feedback text into speech data. The input is the feedback text, and the output is speech data. Text-to-speech (TTS) technology is used to convert the text into speech.

[1970] Step 10:

[1971] The communication module of the terminal receives the voice data received from the server, where the input is the voice data sent from the server and the output is the voice data received by the terminal.

[1972] Step 11:

[1973] The device's speaker and audio output module plays the received audio data and conveys information to the user. The input is audio data, and the output is the audio heard by the user. The audio is played on the device's built-in speaker.

[1974] Prompt Sentence Examples

[1975] If the user says, "Did you lock the house?"

[1976] 1. The microphone picks up the voice and the speech recognition engine converts it into text: "Have you locked your house?"

[1977] 2. Emotion analysis determines that the user is "anxious."

[1978] 3. The command is parsed and a response is generated: "Check home security."

[1979] 4. The security status is queried from the server and the result is obtained.

[1980] 5. The feedback to the user, "The house is locked. Don't worry," is converted into audio data and played back.

[1981] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1982] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1983] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1984] [Fourth embodiment]

[1985] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1986] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1987] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1988] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1989] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1990] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1991] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1992] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1993] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1994] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1995] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1996] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1997] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1998] This invention relates to a voice recognition IoT device and system that analyzes a user's voice commands and realizes functions such as providing information, managing health, and making emergency calls. This system includes an edge terminal and a back-end server that execute a series of processes from receiving a user's voice command to analyzing it, providing information, and responding to an emergency.

[1999] System Configuration

[2000] 1. Terminal (edge ​​device) side

[2001] Microphone and voice input module: to capture the user's voice commands.

[2002] Voice data processing module: converts voice data into digital format and sends it to the server.

[2003] Communication module: Sends voice data to the server and receives responses from the server.

[2004] Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[2005] 2. Server (backend) side

[2006] Speech recognition engine: Converts voice data sent by the user into text data.

[2007] Natural Language Processing (NLP) module: Analyzes text data and understands user requirements.

[2008] Information provision engine: Obtains information based on user requests and generates the required response.

[2009] Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[2010] Database: Stores and manages user health data and emergency contact information.

[2011] Program processing and specific examples

[2012] Voice input recognition and information provision

[2013] A user speaks to the terminal, "What's the weather like today?" This voice command is picked up by the microphone and digitized by the voice data processing module. This digital voice data is sent to the server through the communication module.

[2014] The server uses a speech recognition engine to convert the received voice data into text data. Next, the NLP module analyzes the text data and understands the user's request, "I want weather information." The server uses the information provision engine to call the weather information API and obtain the required weather information. The speech synthesis engine then converts the weather information into voice data, generating the message, "Today's weather is sunny." This voice data is sent to the device and is conveyed to the user through the device's speaker.

[2015] Support for health management

[2016] The user says, "Record my blood pressure today." The voice command is captured by the device and sent to the server. Similarly, the voice data is analyzed by the speech recognition engine and NLP module, and the request "record blood pressure data" is understood.

[2017] The server accesses the database and records the user's blood pressure data. Furthermore, it monitors the user's health condition as needed and generates a warning message if an abnormality is detected. The generated information is converted into voice data through a voice synthesis engine and notified to the user via the device.

[2018] Emergency response

[2019] The user shouts "An emergency has occurred." The voice command is immediately captured by the device and sent to the server as emergency data. The server uses a speech recognition engine to convert it into text data, and an NLP module analyzes it to indicate an emergency.

[2020] The server retrieves emergency contact information from the database and notifies the designated emergency contacts using an automated notification system. At the same time, the server generates feedback information such as "An emergency call has been made" and converts it into voice data using a voice synthesis engine. This voice data is sent to the device and is played back to the user through the device's speaker.

[2021] In this way, a comprehensive system of voice-recognition IoT devices to support users' daily lives is realized.

[2022] The processing flow will be explained below.

[2023] Voice input recognition and information provision

[2024] Step 1:

[2025] The user speaks the voice command "What's the weather like today?" into the device.

[2026] Step 2:

[2027] The device uses a built-in microphone to record voice commands.

[2028] Step 3:

[2029] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[2030] Step 4:

[2031] The terminal transmits the converted voice data to a server via the Internet.

[2032] Step 5:

[2033] The server's voice recognition engine converts the received voice data into text data.

[2034] Step 6:

[2035] The server's natural language processing (NLP) module analyzes the text data "What's the weather like today?" and understands the user's request.

[2036] Step 7:

[2037] The server calls the weather information API based on the analysis results and obtains current weather information.

[2038] Step 8:

[2039] The server's information engine generates a text message saying "Today's weather is sunny" based on the acquired weather information.

[2040] Step 9:

[2041] A speech synthesis engine on the server converts the generated text message into voice data.

[2042] Step 10:

[2043] The server then transmits the converted voice data to the terminal again via the Internet.

[2044] Step 11:

[2045] The audio output module of the terminal plays the received data over the speaker and tells the user aloud, "Today's weather is sunny."

[2046] Support for health management

[2047] Step 1:

[2048] The user speaks to the terminal, saying, "Record my blood pressure today."

[2049] Step 2:

[2050] The device will record your voice commands using the built-in microphone.

[2051] Step 3:

[2052] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[2053] Step 4:

[2054] The terminal transmits the converted voice data to a server via the Internet.

[2055] Step 5:

[2056] The server's voice recognition engine converts the received voice data into text data.

[2057] Step 6:

[2058] The server's natural language processing (NLP) module analyzes the text data "Record my blood pressure today" and understands the user's request.

[2059] Step 7:

[2060] The server accesses the user's physical condition database and records the blood pressure data.

[2061] Step 8:

[2062] The server generates a feedback message indicating that the recording was successful, and converts it into voice data using a voice synthesis engine.

[2063] Step 9:

[2064] The server then transmits the converted voice data to the terminal again via the Internet.

[2065] Step 10:

[2066] The device's audio output module plays the received data over the speaker and tells the user in a voice message, "Your blood pressure has been recorded. Take care."

[2067] Emergency response

[2068] Step 1:

[2069] The user shouts into the terminal, "An emergency has occurred."

[2070] Step 2:

[2071] The device will record emergency voice commands using the built-in microphone.

[2072] Step 3:

[2073] The voice data processing module of the terminal converts the recorded voice into digital data and passes it to the emergency processing module.

[2074] Step 4:

[2075] The terminal immediately transmits the formatted emergency voice data to the server.

[2076] Step 5:

[2077] The server's voice recognition engine converts the received emergency voice data into text data.

[2078] Step 6:

[2079] The server's natural language processing (NLP) module analyzes the text data "An emergency has occurred" and understands that it is an emergency.

[2080] Step 7:

[2081] A server retrieves emergency contact information from a database and sends emergency messages to the contacts using an automated notification system.

[2082] Step 8:

[2083] The server checks the status of the call transmission, generates a feedback message such as "An emergency call has been made. Please wait until the ambulance arrives," and converts it into voice data using a speech synthesis engine.

[2084] Step 9:

[2085] The server then transmits the converted voice data to the terminal again via the Internet.

[2086] Step 10:

[2087] The terminal's audio output module plays the received data over the speaker and tells the user in a voice message, "An emergency call has been made. Please wait until the ambulance arrives."

[2088] Example 1

[2089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2090] Conventional speech recognition systems have had difficulty responding quickly and accurately to diverse user requests. Furthermore, there has been a lack of technology to accurately analyze user requests and provide appropriate information, particularly in the areas of health management and emergency response. This has led to problems such as a loss of convenience and safety for users.

[2091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2092] In this invention, the server includes a conversion means for converting voice data into text data, an analysis means for analyzing the text data and understanding the user's request, and an information provision engine for acquiring information based on the request and generating the necessary response. This enables quick and accurate responses to various user requests. It also enables appropriate data analysis and information provision for health management and emergency response.

[2093] The "acquisition means" is a module for detecting a user's voice command and acquiring it as voice data.

[2094] "Conversion means" is a module for converting the acquired voice commands into a digital format.

[2095] The "transmitting means" is a module for transmitting the converted digital audio data to the server.

[2096] "Text data" refers to data that has been analyzed and converted into text information from voice data.

[2097] A "voice recognition engine" is software for converting received voice data into text data.

[2098] A "natural language processing (NLP) module" is software that analyzes text data and understands user requests.

[2099] An "information engine" is software that acquires information based on a user's request and generates the necessary response.

[2100] The "voice synthesis means" is a module for converting the generated response into voice data.

[2101] The "output means" is a module for outputting audio data to the user.

[2102] The "monitoring means" is a module for monitoring the user's health condition and issuing a warning if an abnormality is detected.

[2103] An "automatic reporting system" is a system that recognizes a user's voice command in an emergency and immediately makes an emergency call.

[2104] This invention relates to a voice recognition IoT device and system that analyzes a user's voice commands and realizes functions such as providing information, managing health, and making emergency calls. This system includes an edge terminal and a back-end server that execute a series of processes from receiving a user's voice command to analyzing it, providing information, and responding to an emergency.

[2105] System Configuration

[2106] Terminal (edge ​​device) configuration

[2107] 1. Microphone and voice input module: to capture the user's voice commands.

[2108] 2. Audio data processing module: converts the captured audio into digital form.

[2109] 3. Communication module: Sends digital voice data to the server and receives responses from the server.

[2110] 4. Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[2111] Server (backend) configuration

[2112] 1. Speech recognition engine: Converts voice data sent by the user into text data.

[2113] 2. Natural Language Processing (NLP) module: Analyzes text data and understands user requirements.

[2114] 3. Information engine: Obtains information based on user requests and generates the necessary response.

[2115] 4. Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[2116] 5. Database: Stores and manages user health data and emergency contact information.

[2117] Voice input recognition and information provision

[2118] A user speaks to the terminal, "What's the weather like today?" This voice command is picked up by the microphone and digitized by the voice data processing module. This digital voice data is sent to the server through the communication module.

[2119] The server uses a speech recognition engine to convert the received voice data into text data. Next, the NLP module analyzes the text data and understands the user's request, "I want weather information." The server's information provision engine calls the weather information API and obtains the required weather information. The speech synthesis engine then converts the weather information into voice data, generating the message, "Today's weather is sunny." This voice data is sent to the device and conveyed to the user through the device's speaker.

[2120] Example prompt:

[2121] Get the voice command "What's the weather today?" and query the server for weather information.

[2122] Support for health management

[2123] The user says, "Record my blood pressure today." The voice command is captured by the device and sent to the server. Similarly, the voice data is analyzed by the speech recognition engine and NLP module, and the request "record blood pressure data" is understood.

[2124] The server accesses the database and records the user's blood pressure data. Furthermore, it monitors the user's health condition as needed and generates a warning message if an abnormality is detected. The generated information is converted into voice data through a voice synthesis engine and notified to the user via the device.

[2125] Example prompt:

[2126] When you receive the voice command "Record today's blood pressure," save the blood pressure data on your server.

[2127] Emergency response

[2128] The user shouts "An emergency has occurred." The voice command is immediately captured by the device and sent to the server as emergency data. The server uses a speech recognition engine to convert it into text data, and an NLP module analyzes it to indicate an emergency.

[2129] The server retrieves emergency contact information from the database and notifies the designated emergency contacts using an automated notification system. At the same time, the server generates feedback information such as "An emergency call has been made" and converts it into voice data using a voice synthesis engine. This voice data is sent to the device and is played back to the user through the device's speaker.

[2130] Example prompt:

[2131] Instantly process the voice command "an emergency has occurred" and make an emergency call to your designated emergency contacts.

[2132] In this way, a comprehensive voice-recognition IoT device system is realized to support users' daily lives, allowing them to easily use a variety of functions through voice commands and also enabling rapid response in emergencies.

[2133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2134] Voice input recognition and information provision

[2135] Processing steps and specific explanations

[2136] Step 1:

[2137] The user issues a voice command: "What's the weather like today?"

[2138] Input: Human speech.

[2139] Specific action: The user speaks clearly near the device.

[2140] Step 2:

[2141] The device picks up audio through a microphone.

[2142] Input: Human speech.

[2143] Output: Analog audio signal.

[2144] Specific operation: A microphone picks up an audio signal.

[2145] Step 3:

[2146] The terminal converts the voice into digital data using a voice data processing module.

[2147] Input: Analog audio signal.

[2148] Output: Digital audio data.

[2149] Specific operation: The audio data processing module digitizes the signal.

[2150] Step 4:

[2151] The terminal transmits digital audio data to the server via the communication module.

[2152] Input: Digital audio data.

[2153] Output: Digital audio data sent to the server.

[2154] Specific operation: The communication module transmits data over the Internet.

[2155] Step 5:

[2156] The server uses a voice recognition engine to convert the received voice data into text data.

[2157] Input: Digital audio data.

[2158] Output: Text data.

[2159] Specific operation: Speech recognition software analyzes the voice data and converts it into text.

[2160] Step 6:

[2161] The server analyzes the text data using a natural language processing (NLP) module to understand the user's request.

[2162] Input: Text data.

[2163] Output: User request information ("I want weather information").

[2164] What it does: The NLP module extracts intent from the text.

[2165] Step 7:

[2166] The server uses an information provision engine to obtain the required information from the weather information API.

[2167] Input: User requested information.

[2168] Output: Weather information data (e.g. "Sunny").

[2169] Specific operation: The information engine sends an API request and gets a response.

[2170] Step 8:

[2171] The server converts the weather information obtained using a voice synthesis engine into voice data and sends it to the terminal.

[2172] Input: Weather information data.

[2173] Output: Audio data.

[2174] Specific operation: The speech synthesis engine converts the text into speech and sends it to the terminal through the communication module.

[2175] Step 9:

[2176] The terminal uses a speaker to play audio data and convey information to the user.

[2177] Input: Audio data.

[2178] Output: Audio output ("The weather is sunny today").

[2179] Specific operation: The speaker plays audio data and provides weather information to the user.

[2180] Support for health management

[2181] Processing steps and specific explanations

[2182] Step 1:

[2183] The user issues a voice command such as "Record my blood pressure today."

[2184] Input: Human speech.

[2185] Specific action: The user speaks clearly near the device.

[2186] Step 2:

[2187] The device picks up audio through a microphone.

[2188] Input: Human speech.

[2189] Output: Analog audio signal.

[2190] Specific operation: A microphone picks up an audio signal.

[2191] Step 3:

[2192] The terminal converts the voice into digital data using a voice data processing module.

[2193] Input: Analog audio signal.

[2194] Output: Digital audio data.

[2195] Specific operation: The audio data processing module digitizes the signal.

[2196] Step 4:

[2197] The terminal transmits digital audio data to the server via the communication module.

[2198] Input: Digital audio data.

[2199] Output: Digital audio data sent to the server.

[2200] Specific operation: The communication module transmits data over the Internet.

[2201] Step 5:

[2202] The server uses a voice recognition engine to convert the received voice data into text data.

[2203] Input: Digital audio data.

[2204] Output: Text data.

[2205] Specific operation: Speech recognition software analyzes the voice data and converts it into text.

[2206] Step 6:

[2207] The server analyzes the text data using a natural language processing (NLP) module to understand the user's request.

[2208] Input: Text data.

[2209] Output: User requested information ("Record blood pressure data").

[2210] What it does: The NLP module extracts intent from the text.

[2211] Step 7:

[2212] The server accesses a database and records the user's blood pressure data.

[2213] Input: User requested information.

[2214] Output: Save results to database.

[2215] Specific operation: The server stores the blood pressure data in a database in the appropriate format.

[2216] Step 8:

[2217] The server uses a speech synthesis engine to convert the database update results into voice data and send it to the terminal.

[2218] Input: Database update results.

[2219] Output: Audio data.

[2220] Specific operation: The speech synthesis engine converts the text into speech and sends it to the terminal through the communication module.

[2221] Step 9:

[2222] The terminal uses a speaker to play audio data and convey information to the user.

[2223] Input: Audio data.

[2224] Output: Audio output ("Blood pressure data recorded").

[2225] Specific operation: The speaker plays the audio data and notifies the user.

[2226] Emergency response

[2227] Processing steps and specific explanations

[2228] Step 1:

[2229] The user shouts, "There's an emergency!"

[2230] Input: Human speech.

[2231] Specific action: The user shouts for help in an emergency.

[2232] Step 2:

[2233] The device picks up audio through a microphone.

[2234] Input: Human speech.

[2235] Output: Analog audio signal.

[2236] Specific action: The microphone picks up an emergency voice signal.

[2237] Step 3:

[2238] The terminal converts the voice into digital data using a voice data processing module.

[2239] Input: Analog audio signal.

[2240] Output: Digital audio data.

[2241] Specific operation: The audio data processing module digitizes the signal.

[2242] Step 4:

[2243] The terminal transmits digital audio data to the server via the communication module.

[2244] Input: Digital audio data.

[2245] Output: Digital audio data sent to the server.

[2246] Specific operation: The communication module transmits data over the Internet.

[2247] Step 5:

[2248] The server uses a voice recognition engine to convert the received voice data into text data.

[2249] Input: Digital audio data.

[2250] Output: Text data.

[2251] Specific operation: Speech recognition software analyzes the voice data and converts it into text.

[2252] Step 6:

[2253] The server analyzes the text data using a natural language processing (NLP) module and understands that it is an emergency.

[2254] Input: Text data.

[2255] Output: Emergency call request information.

[2256] Specific behavior: The NLP module extracts the emergency intent from the text.

[2257] Step 7:

[2258] The server retrieves the emergency contact information from the database and places an emergency call using an automated call system.

[2259] Input: Emergency call request information.

[2260] Output: Emergency call result.

[2261] Specific operation: The server automatically notifies emergency contacts.

[2262] Step 8:

[2263] The server converts the results of the emergency call into voice data using a voice synthesis engine and sends it to the terminal.

[2264] Input: Emergency call result.

[2265] Output: Audio data.

[2266] Specific operation: The speech synthesis engine converts the text into speech and sends it to the terminal through the communication module.

[2267] Step 9:

[2268] The terminal uses a speaker to play audio data and convey information to the user.

[2269] Input: Audio data.

[2270] Output: Audio output ("An emergency call has been made").

[2271] Specific action: The speaker plays audio data to notify the user that emergency measures have been taken.

[2272] Thus, the processing steps for quickly and accurately providing various functions based on the user's voice commands have been specifically described.

[2273] (Application example 1)

[2274] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2275] There are situations where it is difficult for consumers to quickly and accurately obtain information about products and services in physical stores, and where it is difficult to properly manage employee health and respond to emergencies. For these reasons, there is a need for a system that can provide information quickly and accurately, manage employee health, and effectively respond to emergencies.

[2276] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2277] In this invention, the server includes an acquisition means for acquiring a user's voice command, a conversion means for converting the acquired voice command into voice data, a transmission means for transmitting the converted voice data to the server, an analysis means for analyzing the voice data on the server side and acquiring corresponding information, a synthesis means for converting the acquired information on the server side into voice, an output means for outputting the synthesized voice data to the user, a means for providing product information and service information on a device installed in the physical store, a means for recording and managing employee health data, and a means for detecting and reporting an emergency within the store. This improves the speed and accuracy of information provision within the physical store, supports employee health management, and enables rapid and effective emergency response.

[2278] The "acquisition means" is a means for acquiring a voice command from a user.

[2279] The "conversion means" is a means for converting the acquired voice command into voice data.

[2280] The "transmitting means" is a means for transmitting the converted voice data to the server.

[2281] The "analysis means" is a means for analyzing the voice data on the server side and acquiring the corresponding information.

[2282] The "synthesis means" is a means for converting information acquired on the server side into voice.

[2283] The "output means" is a means for outputting synthesized voice data to the user.

[2284] "Means for providing product information and service information" refers to means for providing product information and service information using devices installed in physical stores.

[2285] "Means for recording and managing employee health data" means means for recording and managing employee health data.

[2286] "Means for detecting and reporting emergencies within the store" refers to means for detecting emergencies within a physical store and reporting them promptly.

[2287] This invention is a system that uses IoT devices that utilize voice recognition to provide information in a physical store, manage the health of employees, and make emergency calls. Below, we will explain how to specifically implement this system.

[2288] System Configuration

[2289] The system consists of both terminals (edge ​​devices) and servers (backends).

[2290] Terminal (edge ​​device) side

[2291] 1. Microphone and voice input module: to capture the user's voice commands.

[2292] 2. Voice data processing module: converts the captured voice commands into digital form.

[2293] 3. Communication module: Transmits the converted audio data to the server using wireless communication technologies such as Wi-Fi or 4G / 5G.

[2294] 4. Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[2295] Server (backend) side

[2296] 1. Speech recognition engine: Converts voice data sent from the device into text data. Google Speech Recognition API is used.

[2297] 2. Natural Language Processing (NLP) module: Analyzes text data and understands user requirements. This analysis is performed using a generative AI model.

[2298] 3. Information engine: Obtains information based on user requests and generates the necessary response.

[2299] 4. Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[2300] 5. Database: Stores and manages employee health data and emergency call information.

[2301] Program processing

[2302] The server performs a series of processes including the following steps.

[2303] 1. A means of acquiring the user's voice commands.

[2304] 2. A conversion means for converting the acquired voice command into voice data.

[2305] 3. A transmitting means for transmitting the converted audio data to the server.

[2306] 4. An analysis means for analyzing the voice data on the server side and obtaining the corresponding information.

[2307] 5. A synthesis means for converting the information obtained on the server side into speech.

[2308] 6. Output means for outputting the synthesized voice data to the user.

[2309] 7. A means of providing product and service information on devices installed in physical stores.

[2310] 8. Means for recording and managing employee health data.

[2311] 9. A means of detecting and reporting emergencies within the store.

[2312] This will improve the speed and accuracy of information provision in physical stores, support employee health management, and enable quick and effective response in emergencies.

[2313] Examples of concrete examples and prompts

[2314] Providing product information

[2315] When a user says, "Tell me more about this product," the system provides detailed information about the product.

[2316] Example prompt: "Tell me more about this product."

[2317] Health management

[2318] When an employee says, "Record my blood pressure today," the system records the blood pressure information in a database to help manage their health.

[2319] Example prompt: "Record your blood pressure today."

[2320] Emergency response

[2321] When a user says "an emergency has occurred," the system will quickly make an emergency call and prompt appropriate response.

[2322] Example prompt: "An emergency has occurred."

[2323] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2324] Step 1:

[2325] The user issues a voice command.

[2326] Input: User utterance

[2327] Output: Audio data

[2328] Step 2:

[2329] The terminal uses a microphone and a voice input module to obtain voice commands from the user.

[2330] Input: User's voice

[2331] Output: Analog audio signal

[2332] What it does: A microphone captures sound and generates an analog audio signal.

[2333] Step 3:

[2334] The voice data processing module of the terminal converts the analog voice signal into digital form.

[2335] Input: Analog audio signal

[2336] Output: Digital audio data

[2337] Specific operation: AD conversion (analog-to-digital conversion) processing is performed to convert analog signals into digital signals.

[2338] Step 4:

[2339] The communication module of the terminal transmits the digital voice data to the server.

[2340] Input: Digital audio data

[2341] Output: Audio data sent to the server

[2342] Specific operation: Digital audio data is uploaded to a server using a wireless network (Wi-Fi, 4G / 5G, etc.).

[2343] Step 5:

[2344] The server converts the received digital voice data into text data using a voice recognition engine.

[2345] Input: Digital audio data

[2346] Output: Text data

[2347] Specific behavior: Uses the Google Speech Recognition API to convert audio data into text.

[2348] Step 6:

[2349] The server's natural language processing (NLP) module analyzes the text data and understands the user's request.

[2350] Input: Text data

[2351] Output: Interpretation of user requests

[2352] Specific operation: Using a generative AI model, the text data is analyzed to identify the requested content (product information, health data recording, emergency call, etc.).

[2353] Step 7:

[2354] The server's information providing engine acquires the necessary information based on the user's request and generates a response.

[2355] Input: Interpretation of user requests

[2356] Output: Response data

[2357] Specific operation: For example, when providing product information, retrieve related information from the product database and generate a response.

[2358] Step 8:

[2359] The server's voice synthesis engine converts the response data into voice data and sends it to the terminal.

[2360] Input: Response data

[2361] Output: Audio data

[2362] Specific operation: Text-to-speech (TTS) is performed and the audio data is sent to the terminal.

[2363] Step 9:

[2364] The terminal receives the transmitted voice data and conveys information to the user using a speaker and a voice output module.

[2365] Input: Audio data

[2366] Output: Audio output

[2367] Specific operation: Plays the received audio data and outputs information to the user through the speaker.

[2368] This allows for a fast and accurate response to the user's voice commands.

[2369] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2370] This invention adds emotion recognition functionality to a voice recognition IoT device that analyzes a user's voice commands and provides information, health management, and emergency notification functions, providing appropriate feedback and support according to the user's emotional state. This system includes an edge terminal and a back-end server for executing a series of processes from voice command acquisition and analysis to information provision, emotion recognition, and emergency response.

[2371] System Configuration

[2372] 1. Terminal (edge ​​device) side

[2373] Microphone and voice input module: to capture the user's voice commands.

[2374] Voice data processing module: converts voice data into digital format and sends it to the server.

[2375] Communication module: Sends voice data to the server and receives responses from the server.

[2376] Speaker and audio output module: Plays back audio data received from the server and conveys information to the user.

[2377] 2. Server (backend) side

[2378] Speech recognition engine: Converts voice data sent by the user into text data.

[2379] Natural Language Processing (NLP) module: Analyzes text data and understands user requirements.

[2380] Information provision engine: Obtains information based on user requests and generates the required response.

[2381] Speech synthesis engine: Converts the generated response into voice data and sends it to the device.

[2382] Database: Stores and manages user health data and emergency contact information.

[2383] Emotion engine: Analyzes the user's emotions from voice data and generates emotional information.

[2384] Feedback engine: Generates appropriate feedback to users based on emotional information.

[2385] Monitoring system: A system for monitoring the user's health and mental state.

[2386] Program processing and specific examples

[2387] Voice input recognition and information provision

[2388] A user speaks to the terminal, "What's the weather like today?" This voice command is picked up by the microphone and digitized by the voice data processing module. This digital voice data is sent to the server through the communication module.

[2389] The server uses a speech recognition engine to convert the received voice data into text data. Next, the NLP module analyzes the text data and understands the user's request, "I want weather information." The server uses the information provision engine to call the weather information API and obtain the required weather information. The emotion engine then analyzes the user's emotion from the voice data and generates emotion information such as "curiosity" or "confusion." Based on this, the feedback engine adjusts the text message and adds positive feedback such as "The weather is sunny today. Have a nice day." The message is then converted into voice data by the speech synthesis engine and sent to the device. It is then audibly conveyed to the user through the device's speaker.

[2390] Support for health management

[2391] The user says, "Record my blood pressure today." The voice command is captured by the device and sent to the server. Similarly, the voice data is analyzed by the speech recognition engine and NLP module, and the request "record blood pressure data" is understood.

[2392] The server accesses the database and records the user's blood pressure data. Furthermore, an emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "anxiety" or "worry." A feedback engine generates feedback messages to reassure the user. The generated information is converted into voice data through a voice synthesis engine, and the user is notified via their device that "Your blood pressure has been recorded. Please relax."

[2393] Emergency response

[2394] The user shouts "An emergency has occurred." The voice command is immediately captured by the device and sent to the server as emergency data. The server uses a speech recognition engine to convert it into text data, and an NLP module analyzes it to indicate an emergency.

[2395] The server retrieves emergency contact information from the database and uses an automatic notification system to notify the designated emergency contact. At the same time, an emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "fear" or "tension." Based on this, a feedback engine generates a feedback message to reassure the user. The message, "An emergency call has been made. Please wait until the ambulance arrives," is converted into voice data by a speech synthesis engine and sent to the terminal for transmission to the user.

[2396] In this way, adding emotion recognition functionality enables flexible responses according to the user's emotional state, improving the user experience.

[2397] The processing flow will be explained below.

[2398] Voice input recognition and information provision

[2399] Step 1:

[2400] The user speaks the voice command "What's the weather like today?" into the device.

[2401] Step 2:

[2402] The device uses a built-in microphone to record voice commands.

[2403] Step 3:

[2404] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[2405] Step 4:

[2406] The terminal transmits the converted voice data to a server via the Internet.

[2407] Step 5:

[2408] The server's voice recognition engine converts the received voice data into text data.

[2409] Step 6:

[2410] The server's natural language processing (NLP) module analyzes the text data "What's the weather like today?" and understands the user's request.

[2411] Step 7:

[2412] The server calls the weather information API based on the analysis results and obtains current weather information.

[2413] Step 8:

[2414] The server's information engine generates a text message saying "Today's weather is sunny" based on the acquired weather information.

[2415] Step 9:

[2416] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "curiosity" or "confusion."

[2417] Step 10:

[2418] The server's feedback engine generates feedback appropriate for the user based on the emotional information, generating a message such as "The weather is sunny today. Have a nice day."

[2419] Step 11:

[2420] A speech synthesis engine on the server converts the generated text message into voice data.

[2421] Step 12:

[2422] The server transmits the converted audio data to the terminal via the Internet.

[2423] Step 13:

[2424] The terminal's audio output module plays the received data over the speaker and tells the user in a voice message, "Today's weather is sunny. Have a nice day."

[2425] Support for health management

[2426] Step 1:

[2427] The user speaks to the terminal, saying, "Record my blood pressure today."

[2428] Step 2:

[2429] The device will record your voice commands using the built-in microphone.

[2430] Step 3:

[2431] The terminal's voice data processing module converts the recorded voice into digital data and passes it to the communication module.

[2432] Step 4:

[2433] The terminal transmits the converted voice data to a server via the Internet.

[2434] Step 5:

[2435] The server's voice recognition engine converts the received voice data into text data.

[2436] Step 6:

[2437] The server's natural language processing (NLP) module analyzes the text data "Record my blood pressure today" and understands the user's request.

[2438] Step 7:

[2439] The server accesses the user's physical condition database and records the blood pressure data.

[2440] Step 8:

[2441] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "anxiety" or "worry."

[2442] Step 9:

[2443] The server's feedback engine generates a feedback message based on the emotional information to reassure the user, such as "Your blood pressure has been recorded. Please relax."

[2444] Step 10:

[2445] A speech synthesis engine on the server converts the generated text message into voice data.

[2446] Step 11:

[2447] The server transmits the converted audio data to the terminal via the Internet.

[2448] Step 12:

[2449] The device's audio output module plays the received data over the speaker and tells the user in a voice message, "Your blood pressure has been recorded. Please relax."

[2450] Emergency response

[2451] Step 1:

[2452] The user shouts into the terminal, "An emergency has occurred."

[2453] Step 2:

[2454] The device will record emergency voice commands using the built-in microphone.

[2455] Step 3:

[2456] The voice data processing module of the terminal converts the recorded voice into digital data and passes it to the emergency processing module.

[2457] Step 4:

[2458] The terminal immediately transmits the formatted emergency voice data to the server.

[2459] Step 5:

[2460] The server's voice recognition engine converts the received emergency voice data into text data.

[2461] Step 6:

[2462] The server's natural language processing (NLP) module analyzes the text data "An emergency has occurred" and understands that it is an emergency.

[2463] Step 7:

[2464] The server retrieves the emergency contact information from the database and sends an emergency message to the emergency contact using an automated notification system.

[2465] Step 8:

[2466] The server's emotion engine analyzes the user's emotions from the voice data and generates emotional information such as "fear" or "tension."

[2467] Step 9:

[2468] The server's feedback engine generates a reassuring feedback message based on the emotional information, such as "An emergency call has been made. Please wait until the ambulance arrives."

[2469] Step 10:

[2470] A speech synthesis engine on the server converts the generated feedback message into voice data.

[2471] Step 11:

[2472] The server transmits the converted audio data to the terminal via the Internet.

[2473] Step 12:

[2474] The terminal's audio output module plays the received data over the speaker and tells the user in a voice message, "An emergency call has been made. Please wait until the ambulance arrives."

[2475] Example 2

[2476] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2477] Conventional voice recognition systems only analyze users' voice commands and provide information, but are unable to provide appropriate feedback or support based on the user's emotional state. This makes it difficult to improve the user experience or respond quickly and appropriately in emergencies.

[2478] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2479] In this invention, the server includes a speech recognition means for converting voice data into text data, a natural language processing means for analyzing the text data and understanding the request, and an information providing means for acquiring information based on the request and generating a response. This enables not only the provision of information based on the user's voice command, but also appropriate feedback based on emotion recognition and a rapid response in emergencies.

[2480] The "acquisition means" is a device or module for acquiring a user's voice command.

[2481] The "conversion means" is a device or module for converting the acquired audio data into digital data.

[2482] The "transmitting means" is a communication device or module for transmitting the converted voice data to the server.

[2483] "Speech recognition means" refers to a device or algorithm that converts voice data into text data on the server side.

[2484] "Natural language processing means" refers to devices and algorithms that analyze text data and understand user requests.

[2485] An "information providing means" is a device or algorithm for obtaining information based on a user request and generating a response.

[2486] A "voice synthesis means" is a device or algorithm for converting a generated response into voice data.

[2487] "Output means" refers to a device or module for outputting audio data to the user.

[2488] "Emotion recognition means" refers to a device or algorithm for analyzing a user's emotions from voice data.

[2489] The "feedback means" refers to a device or algorithm for generating appropriate feedback to the user based on emotional information.

[2490] The "support means" is a device or system that supports the user's health management using the health data and emotion information acquired on the server side.

[2491] The "reporting means" refers to a device or system that quickly recognizes a user's voice command in an emergency and immediately makes an emergency call.

[2492] This invention is a system that adds emotion recognition functionality to a voice recognition IoT device that analyzes a user's voice commands and realizes functions such as providing information, managing one's health, and making an emergency call, and provides appropriate feedback and support according to the user's emotional state. Here, an embodiment of the present invention will be specifically described.

[2493] System Configuration

[2494] Terminal (edge ​​device) side

[2495] Microphone and voice input module: Acquires the user's voice commands by capturing the user's voice as an analog signal.

[2496] Audio data processing module: Converts the captured analog audio signal into digital format. Samples the audio signal and encodes it into digital data.

[2497] Communication module: Sends digital audio data to the server and receives responses from the server. Uses Wi-Fi or mobile data.

[2498] Speaker and audio output module: Plays back audio data received from the server and conveys information to the user. Converts digital audio data into an analog signal and plays it as sound through a speaker.

[2499] Server (backend) side

[2500] Speech recognition engine: Converts voice data sent from the device into text data. It uses machine learning models to analyze the voice data and convert it into text format.

[2501] Natural Language Processing (NLP) module: Analyzes text data to understand user requirements, analyzes text data grammatically, and extracts keywords and context.

[2502] Information engine: Obtains information based on user requests and generates responses. Specifically, it accesses weather information APIs and other sources to obtain the necessary information.

[2503] Speech synthesis engine: Converts the generated response into voice data and sends it to the device. It converts text information into natural-sounding voice using speech synthesis technology.

[2504] Database: Stores and manages user health data and emergency contact information.

[2505] Emotion Engine: Analyzes user emotions from voice data and generates emotional information. Analyzes voice tone and word usage to identify emotional state.

[2506] Feedback Engine: Generates appropriate feedback based on emotional information. Adjusts the tone and content of messages depending on the emotional state.

[2507] Monitoring system: A system that monitors the user's health and mental state.

[2508] Specific examples

[2509] Specific examples of voice input recognition and information provision

[2510] The user speaks to the device, "What's the weather like today?" The device's microphone picks up this voice, and the voice data processing module converts the analog voice signal into digital data. The communication module sends the digital voice data to the server.

[2511] The server's speech recognition engine converts the voice data into text data, and the NLP module analyzes the text to understand the request, "I want weather information." The information provision engine calls the weather information API to obtain the required information. The emotion engine analyzes emotions such as "curiosity" from the voice data, and the feedback engine generates a positive message such as "The weather is sunny today. Have a nice day."

[2512] The speech synthesis engine converts this message into voice data and sends it to the terminal via the communication module, where the terminal's speaker plays the voice back to the user.

[2513] example:

[2514] User: "What's the weather like today?"

[2515] Server: "The weather is sunny today. Have a nice day."

[2516] Specific examples of support for health management

[2517] The user says, "Record my blood pressure today." The terminal receives the voice command, and the voice data processing module converts it into digital data and sends it to the server.

[2518] The server's speech recognition engine and NLP module analyze the voice data and understand the request to "record blood pressure data." The database is accessed and the user's blood pressure data is recorded. The emotion engine analyzes emotions such as "anxiety" and "worry," and the feedback engine generates a message saying, "Your blood pressure has been recorded. Please relax."

[2519] The speech synthesis engine converts this message into voice data, sends it to the device, and plays it back to the user through the device's speaker.

[2520] example:

[2521] User: "Log my blood pressure today."

[2522] Server: "Your blood pressure has been recorded. Please relax."

[2523] Specific examples of emergency response

[2524] The user shouts, "An emergency has occurred." The device immediately receives the voice command and sends it to the server as emergency data.

[2525] The server's speech recognition engine converts the voice data into text data, and the NLP module analyzes the emergency situation. Emergency contact information is retrieved from the database, and the automated notification system notifies the emergency contacts specified in the database. At the same time, the emotion engine analyzes emotions such as "fear" and "tension," and the feedback engine generates a message saying, "An emergency call has been made. Please wait until the ambulance arrives."

[2526] The speech synthesis engine converts this message into voice data, sends it to the device, and plays it back to the user through the device's speaker.

[2527] example:

[2528] User: "We have an emergency."

[2529] Server: "We've called emergency services. Please wait until the ambulance arrives."

[2530] In this way, this system executes a series of processes from acquiring voice commands to analyzing them, providing information, recognizing emotions, and responding to emergencies, enabling flexible responses according to the user's emotional state.

[2531] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2532] Step 1:

[2533] The user issues a voice command. The input is the user's voice command (e.g., "What's the weather today?"). The device's microphone and voice input module pick up this voice command and capture it as an analog voice signal. The output is an analog voice signal.

[2534] Step 2:

[2535] The terminal's voice data processing module converts the acquired analog voice signal into digital data. The input is an analog voice signal and the output is digital voice data. Specifically, the voice signal is sampled and encoded into a digital format (bit string).

[2536] Step 3:

[2537] The communication module of the device sends digital audio data to the server. The input is digital audio data, and the output is digital audio data sent to the server. Data transmission uses Wi-Fi or mobile data communication.

[2538] Step 4:

[2539] The server's speech recognition e...

Claims

1. an acquisition means for acquiring a voice command of a user; a conversion means for converting the acquired voice command into voice data; a transmitting means for transmitting the converted voice data to a server; an analysis means for analyzing the voice data on the server side and obtaining corresponding information; A synthesis means for converting the acquired information into speech on the server side; output means for outputting the synthesized voice data to a user; A system including:

2. 2. The system according to claim 1, further comprising support means for supporting the user's health management using the information acquired on the server side.

3. The system according to claim 1, further comprising a notification means for quickly recognizing a user's voice command in an emergency and immediately making an emergency call.

Images