System

Abstract

A system is provided.SOLUTION: The system includes a means for receiving an input by voice, a means for converting the input voice into a text, a means for transmitting the voice-converted text to a server through a network, a means for reconstituting the text received by the server with an easy-to-understand expression, a means for converting the reconstituted text into voice data, and a means for receiving and reproducing the voice data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As our society ages, many elderly people are forced to live lonely lives. For those elderly who have difficulty using smartphones or computers, smooth communication with family and relatives who live far away is a particular challenge. Obtaining information for administrative procedures and daily life is also difficult, contributing to a decline in the quality of life for the elderly. Therefore, there is a need for a means for elderly people to easily obtain information using only their voice and to smoothly communicate with family and relatives. [Means for solving the problem]

[0005] The present invention provides a system including a means for accepting voice input, a means for converting the input voice into text, a means for transmitting the converted voice text to a server via a network, a generating means for the server to reconstruct the received text into easy-to-understand expressions, a means for converting the reconstructed text into voice data, and a means for receiving and playing the voice data. Furthermore, the generating means has a function for automatically adjusting the text depending on the user's age group and information literacy. Furthermore, the means for accepting voice input has a function for performing noise removal and signal normalization. This enables elderly people to easily obtain necessary information using only their voice and to smoothly communicate with their family and relatives.

[0006] ---

[0007] "Means for accepting input by voice" means having the ability to capture the user's voice using a microphone or other voice input device.

[0008] The "means for converting input voice into text" has the function of converting captured voice signals into corresponding text data using voice recognition technology.

[0009] The "means for transmitting the converted text to a server via a network" means a means that has the function of transmitting the converted text data to a remote server via the Internet or other network.

[0010] "A means for generating text received by the server in an easy-to-understand format" refers to an artificial intelligence or algorithm that converts the text data received by the server into an easy-to-understand format depending on the user's age group and information literacy.

[0011] The "means for converting the reconstructed text into voice data" has a function of converting the reconstructed text data into corresponding voice data using voice synthesis technology.

[0012] "Means for receiving and playing audio data" means a device that has the function of receiving audio data transmitted via a network and playing it using a speaker or other audio output device.

[0013] The "function of automatically adjusting text according to the user's age group and information literacy" means that the generation means has the function of adaptively generating text data based on the user's profile information and expressing it in an easy-to-understand manner.

[0014] The "noise removal and signal normalization function" refers to a technology that removes unnecessary noise from the audio signal captured during audio input and adjusts the audio signal to an optimal state. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The system of the present invention includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, a voice synthesis engine, and a voice output device, which allows elderly people to obtain information using only their voice and facilitate communication with family and relatives in remote locations.

[0037] Voice input process

[0038] 1. A user speaks a question or command to a smart speaker, such as "What's the weather like today?" or "Tell me how to apply for a pension."

[0039] 2. The microphone on the device (smart speaker) captures the user's voice. At this time, the device performs noise reduction and signal normalization to optimize the voice.

[0040] Voice recognition and transmission to the server

[0041] 3. The device uses a speech recognition engine to convert the captured speech into corresponding text, for example, converting the speech "What's the weather today?" into the text "What's the weather today?"

[0042] 4. The device sends this text data to the server via the network, along with meta-information such as the user's age group and information literacy.

[0043] Information generation by generative AI

[0044] 5. The server analyzes the received text data and passes it to the generative AI engine, which generates information in easy-to-understand language according to the user's age and literacy level.

[0045] 6. For example, if the received text is "Please tell me how to apply for a pension," the generation AI will generate the response "To apply for a pension, first get the necessary documents from city hall."

[0046] Audio generation and playback

[0047] 7. The server passes the generated response text to the speech synthesis engine and converts it into voice data. The speech synthesis engine converts the text "To apply for a pension, first get the necessary documents from city hall" into voice data.

[0048] 8. The server sends the audio data back to the terminal via the network.

[0049] Providing voice responses

[0050] 9. The device plays back the received voice data and communicates the response to the user. For example, a voice may be heard from the speaker saying, "To apply for a pension, first obtain the necessary documents from city hall."

[0051] 10. The user listens to this and takes the necessary steps or actions.

[0052] Specific examples

[0053] For example, consider the case where a user speaks to a smart speaker saying, "Tell me what the weather is today." At this time, the device captures the voice, converts it into text, and sends the message "What's the weather today?" to the server. The server passes this text to a generative AI engine, which generates a response saying, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." The generated response text is then converted into voice data and sent back to the device. Finally, the device plays back this voice data, telling the user, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out."

[0054] In this way, the elderly can easily obtain the necessary information by voice, and can also receive easy-to-understand responses according to specific conditions, thereby improving the quality of life for the elderly.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] The user speaks a question or command to the smart speaker, for example, "What's the weather like today?" or "How do I apply for a pension?"

[0058] Step 2:

[0059] The microphone on the device (smart speaker) captures the user's voice, which includes collecting and pre-processing the voice signal (noise removal and signal normalization).

[0060] Step 3:

[0061] The device uses a speech recognition engine to convert the captured speech into text data, for example converting the speech "What's the weather today?" into text "What's the weather today?"

[0062] Step 4:

[0063] The device sends the converted text data and user meta-information (such as age group and information literacy) to the server as an HTTP request. The request includes the text content as well as profile information for each user.

[0064] Step 5:

[0065] The server analyzes the received HTTP request and extracts text data, which it then passes to the AI ​​generation engine.

[0066] Step 6:

[0067] The generative AI analyzes the text data and generates a response in easy-to-understand language based on the user's profile information. For example, in response to the text "Please tell me how to apply for a pension," it creates the response "To apply for a pension, first obtain the necessary documents from city hall."

[0068] Step 7:

[0069] The server passes the generated response text to a speech synthesis engine, which converts the text into speech data that can be read aloud to the user.

[0070] Step 8:

[0071] The server sends the generated audio data to the terminal as an HTTP response.

[0072] Step 9:

[0073] The device uses its function to play back the received voice data and provides a response to the user as voice. For example, it tells the user, "To apply for a pension, first get the necessary documents from your city hall."

[0074] Step 10:

[0075] The user hears a voice response from the smart speaker and takes the necessary action, for example, preparing to go to city hall.

[0076] The above processing flow allows users, especially elderly people, to easily obtain information using voice and to smoothly communicate with family and relatives.

[0077] Example 1

[0078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0079] Elderly people and users with low information literacy need a system that makes it easier to obtain information and allows smooth communication with family and relatives in remote locations using only voice. However, previous voice recognition systems have had difficulty providing appropriate information according to the user's age and information literacy, and have also had problems with noise removal and signal normalization.

[0080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0081] In this invention, the server includes means for accepting voice input, means for converting the input voice into text, means for transmitting the converted voice text to the server via a network, means for the server to analyze the received text and input it into the generative AI model, means for converting the response generated by the generative AI model into voice data, and means for receiving and playing the voice data. This enables the provision of easy-to-understand information according to the user's age group and information literacy, as well as noise removal and signal normalization during voice input.

[0082] "Means for receiving input by voice" refers to a device or component that has the function of capturing voice uttered by a user and inputting it into the system.

[0083] "Means for converting input voice into text" refers to a device or software that has the function of appropriately analyzing the captured voice signal and converting its content into text data.

[0084] "Means for transmitting speech-converted text to a server via a network" refers to devices or software that have the function of transmitting text data converted from speech to a server via a network such as the Internet.

[0085] "Means for analyzing text received by the server and inputting it into the generative AI model" refers to devices or software that have the function of analyzing text data received via a network and inputting appropriate prompt sentences into the generative AI model based on the analysis results.

[0086] "Means for converting responses generated by a generative AI model into voice data" refers to devices or software that have the function of converting text-format responses generated by a generative AI model into voice data.

[0087] "Means for receiving and reproducing audio data" refers to a device or component that has the function of reproducing audio data received via a network and providing information to the user by audio.

[0088] "Noise reduction" refers to technology that removes unnecessary background sounds and noises that are included in voice input, and clearly captures only the voice spoken by the user.

[0089] "Signal normalization" refers to a technique for adjusting the amplitude and volume of an audio signal to a certain range in order to improve the accuracy of voice recognition.

[0090] "Automatically adjusting responses according to the user's age group and information literacy" refers to the ability of the generative AI model to generate responses with optimal expressions and content that are tailored to the user's characteristics.

[0091] MODE FOR CARRYING OUT THE INVENTION

[0092] The system of the present invention combines a voice input device, a voice recognition engine, a network communication module, a generative AI engine, a voice synthesis engine, and a voice output device, allowing elderly people to obtain information using only their voice and facilitating communication with family and relatives in remote locations.

[0093] Hardware and Software Configuration

[0094] The server has a generative AI engine (e.g., OpenAI® GPT-4®) and a speech synthesis engine (e.g., Google® Text-to-Speech or Amazon Polly). The terminal has a voice input device (e.g., a smart speaker), a speech recognition engine (e.g., Google Cloud Speech-to-Text or Amazon Transcribe), and a network communication module (e.g., Wi-Fi or LTE). The user uses these devices to obtain information by voice and receive voice responses.

[0095] Audio Input and Preprocessing

[0096] The user speaks questions or commands to the smart speaker, such as "What's the weather like today?" or "How do I apply for a pension?" The device uses a voice input device to capture the user's voice, then performs noise reduction and signal normalization to obtain an optimal voice signal.

[0097] Voice recognition and data transmission

[0098] The device uses a speech recognition engine to convert the captured voice into text. For example, it converts the voice "What's the weather today?" into text "What's the weather today?" The device then sends the converted text data to the server via the network communication module.

[0099] Text analysis and response generation

[0100] The server analyzes the received text data and inputs it as a prompt to the generative AI engine. Based on this prompt, the generative AI engine generates an appropriate response according to the user's age group and information literacy. For example, in response to the text "Please tell me how to apply for a pension," it generates the response "To apply for a pension, first obtain the necessary documents from city hall."

[0101] Speech synthesis and data transmission

[0102] The server passes the generated response text to a speech synthesis engine, which converts it into voice data. For example, the text "To apply for a pension, first get the necessary documents at city hall" is converted into voice data. The converted voice data is then sent back to the terminal via the network.

[0103] Playing a voice response

[0104] The device plays the received voice data and communicates the response to the user. For example, a voice may come from the speaker saying, "To apply for a pension, first get the necessary documents from your city hall." The user listens to this and takes the necessary steps or actions.

[0105] Specific examples

[0106] Specifically, imagine a case where a user asks a smart speaker, "What's the weather today?" At this time, the device captures the voice, converts it into text, "What's the weather today?", and sends it to the server. The server inputs the prompt text into a generation AI engine, which generates a response, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." This response is converted into voice data and sent back to the device. Finally, the device plays back this voice data, telling the user, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out."

[0107] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0108] Step 1:

[0109] A user speaks a question or command into a smart speaker, and the input audio signal is captured by the microphone. For example, "What's the weather like today?"

[0110] Step 2:

[0111] The device's audio input device receives the captured audio signal and performs noise reduction and signal normalization. Noise reduction removes background noise from the audio signal, and signal normalization adjusts the amplitude of the audio signal to an appropriate range, thereby making the input audio clearer.

[0112] Step 3:

[0113] The device uses a speech recognition engine to convert the preprocessed voice signal into text data. Specifically, the speech recognition engine analyzes the voice signal and generates text data such as "What's the weather like today?". This text data is output.

[0114] Step 4:

[0115] The terminal sends the converted text data to the server via the network communication module. When sending, meta-information such as the user's age group and information literacy level is also sent along with the text data. The text data and meta-information are sent to the server as output.

[0116] Step 5:

[0117] The server analyzes the received text data and meta information and inputs it as a prompt to the generative AI model. Specifically, the received text "What's the weather like today?" is processed as a prompt for the generative AI model, and an easy-to-understand response is generated that takes into account the user's age group and literacy level. "What's the weather like today?"

[0118] Step 6:

[0119] The generative AI model responds by generating text such as, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." This text data is output on the server.

[0120] Step 7:

[0121] The server passes the generated response text to the speech synthesis engine and converts it into voice data. The speech synthesis engine converts the text "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." into voice data. This voice data is generated.

[0122] Step 8:

[0123] The server then transmits the converted audio data to the terminal via the network, using a network protocol for transmission.

[0124] Step 9:

[0125] The device plays the received voice data and conveys the response to the user. The smart speaker's speaker will say, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." The user listens to this voice response and takes the necessary action.

[0126] Through the above steps, the user can easily obtain information and receive appropriate responses using only voice.

[0127] (Application example 1)

[0128] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0129] The challenge is to enable elderly people to easily obtain necessary information using only their voice, and to be able to quickly request assistance in an emergency. There is also a need to provide an environment where elderly people can easily check the safety of their homes and live with peace of mind.

[0130] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0131] In this invention, the server includes means for accepting voice input, means for converting the input voice into text, means for transmitting the converted text to the server via a network, means for reconstructing the text received by the server in an easy-to-understand manner, means for converting the reconstructed text into voice data, means for receiving and playing the voice data, and means for sending a notification to a contact specified based on the voice input in an emergency. This enables elderly people to obtain information, confirm their safety, and quickly request assistance in an emergency using only their voice.

[0132] "Means for accepting input via voice" refers to devices or technologies that have the function of capturing voices uttered by a user and inputting them into the system.

[0133] "Means for converting input speech into text" refers to technology or devices that analyze speech signals and convert them into corresponding text data.

[0134] "Means for transmitting the converted text to a server via a network" refers to a technology for transmitting text data converted from speech to a server via a network such as the Internet.

[0135] "Means for generating text received by the server in an easy-to-understand format" refers to technology or devices that automatically process and reconstruct text data so that it is easy for users to understand.

[0136] The "means for converting reconstructed text into speech data" refers to a technique or device that converts text data into a speech signal.

[0137] "Means for receiving and playing audio data" refers to a technique for playing audio data received from a server on a device such as a speaker.

[0138] "Means for sending notifications to designated contacts based on voice input in an emergency" refers to technology or devices that automatically send notifications to pre-set contacts when a user inputs a voice command indicating an emergency.

[0139] The present invention provides a home security system that allows elderly people to obtain information using only their voice, check the safety of their home, and use emergency notification functions. The system includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, a voice synthesis engine, and a voice output device.

[0140] System configuration and usage

[0141] 1. Voice Input Device

[0142] A voice input device is a device that receives voice instructions from a user, such as a smart speaker. The device has the ability to perform noise reduction and signal normalization. The user speaks questions or commands into the device.

[0143] 2. Speech Recognition Engine

[0144] The speech captured by the voice input device is converted into text by a speech recognition engine, which can use the speech_recognition package. This engine analyzes the speech and converts it into text data.

[0145] 3. Network Communication Module

[0146] The text generated by the speech recognition engine is sent to the server via the network communication module, which may also include user meta-information such as age and information literacy.

[0147] 4. Generative AI Engine

[0148] The server passes the received text data to a generative AI engine, which generates information in easy-to-understand language according to the user's age group and information literacy. This engine is realized using a generative AI model such as ChatGPT (registered trademark).

[0149] 5. Speech synthesis engine

[0150] The generated response text is converted into voice data by a speech synthesis engine. The gTTS package can be used for this voice synthesis. The server then transmits the generated voice data back to the terminal via the network.

[0151] 6. Audio Output Device

[0152] The terminal plays the received voice data and communicates the response to the user, where information is provided using a speaker.

[0153] 7. Emergency notification function

[0154] In the event of an emergency, the user can enter a designated voice command (e.g., "Help me") and the system will automatically send a notification to pre-registered contacts, enabling a quick response.

[0155] Specific examples

[0156] For example, suppose a user speaks to a smart speaker saying, "Tell me the lock status of the windows." At this time, the voice input device captures the voice, and the voice recognition engine converts it into text, "Tell me the lock status of the windows." This text is then sent to the server via the network. The server passes the text to the generative AI engine, which generates the response, "All windows are locked." This response text is converted into voice data by the speech synthesis engine and sent back to the terminal, allowing the user to hear the response, "All windows are locked," on the speaker.

[0157] Additionally, if the user says "help me," the system will immediately send a notification to designated emergency contacts, allowing seniors to quickly seek assistance in an emergency.

[0158] Prompt Sentence Examples

[0159] "Tell me if the windows are locked."

[0160] "help me."

[0161] This allows the system of the present invention to allow elderly people to obtain information using only their voice, check the safety of their homes, and request assistance in emergencies.

[0162] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0163] Step 1:

[0164] Users speak questions and commands to the smart speaker.

[0165] Input: Example spoken question: "How are the windows locked?"

[0166] Data processing: The microphone built into the smart speaker captures the audio and performs noise reduction and signal normalization.

[0167] Output: Optimized audio signal

[0168] Step 2:

[0169] The device uses a speech recognition engine to convert the captured speech into text.

[0170] Input: Optimized audio signal

[0171] Data calculation: A speech recognition engine (e.g., speech_recognition) analyzes the audio signal and converts it into corresponding text data.

[0172] Output: Text data example: "Please tell me if the windows are locked."

[0173] Step 3:

[0174] The terminal transmits the converted text to the server via the network communication module.

[0175] Input: Text data, user meta information (e.g., age group, information literacy)

[0176] Data processing: Text data and meta information are packaged into packets and sent over the network.

[0177] Output: Receives text data and meta information to the server

[0178] Step 4:

[0179] The server analyzes the received text and meta information and passes it to the generative AI engine.

[0180] Input: Text data, meta information

[0181] Data calculation: A generative AI engine reconstructs text into easy-to-understand expressions according to the user's age group and information literacy. Example: Using a generative AI model

[0182] Output: Reconstructed text data Example: "All windows are locked."

[0183] Step 5:

[0184] The server passes the reconstructed text to a speech synthesis engine, which converts it into speech data.

[0185] Input: Reconstructed text data

[0186] Data processing: A speech synthesis engine (e.g., gTTS) converts text data into speech signals and generates an audio file.

[0187] Output: Example speech data: "All windows are locked."

[0188] Step 6:

[0189] The server transmits the voice data again to the terminal via the network.

[0190] Input: Audio data

[0191] Data processing: Packetizing the audio data and sending it over the network.

[0192] Output: Receiving audio data to the device

[0193] Step 7:

[0194] The terminal plays back the received voice data and conveys the response to the user.

[0195] Input: Audio data

[0196] Specific behavior: Plays audio data through the speaker, telling the user, "All windows are locked."

[0197] Output: Providing a voice response to the user

[0198] Step 8:

[0199] The user can enter further voice commands as needed, including emergency commands such as "help."

[0200] Input: Example user voice command: "Help me"

[0201] Data calculation: The smart speaker captures the voice again and converts it into text using the recognition engine. The server receives the text "Help" and sends an emergency notification to the specified contacts.

[0202] Output: Sending an emergency notification and acknowledging it

[0203] In this way, the system allows seniors to check the safety of their homes, obtain necessary information, and quickly call for help in emergencies using only their voice.

[0204] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0205] The system of the present invention includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, an emotion engine, a voice synthesis engine, and a voice output device, which allows elderly people to obtain information solely through voice and facilitate communication with family and relatives based on emotion analysis.

[0206] Voice input process

[0207] 1. A user speaks questions, commands, and everyday conversations into a smart speaker. For example, they can say, "I'm in a bad mood today, so I don't want to call my friends."

[0208] 2. The microphone on the device (smart speaker) captures the user's voice. At this time, the device performs noise reduction and signal normalization to optimize the voice.

[0209] 3. The device uses an emotion engine to analyze parameters such as tone, tempo, and emphasis from the captured voice to identify the user's emotion. For example, if the voice tone is low and the tempo is slow, it will be determined that the user is in a bad mood.

[0210] Voice recognition and transmission to the server

[0211] 4. The device uses a speech recognition engine to convert the captured speech into corresponding text data, for example, converting the speech "I'm in a bad mood today, so I don't want to call my friends" into text "I'm in a bad mood today, so I don't want to call my friends."

[0212] 5. The device sends the converted text data and user meta-information (such as age group, information literacy, and emotional information) to the server as an HTTP request. The request includes the text content as well as profile information for each user.

[0213] Information generation by generative AI

[0214] 6. The server analyzes the received HTTP request, extracts text data and emotional information, and passes this data to the generative AI engine.

[0215] 7. The generation AI analyzes the text data and generates a response in easy-to-understand language based on the user's profile information and emotional information. For example, if the received text reads, "I'm in a bad mood today, so I don't want to contact my friends," and the emotional information is bad, the generation AI will generate a response like, "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[0216] Audio generation and playback

[0217] 8. The server passes the generated response text to a speech synthesis engine to convert the text into speech data that can be read aloud to the user.

[0218] 9. The server sends the generated audio data to the device as an HTTP response.

[0219] Providing voice responses

[0220] 10. Use the device's function to play back the received voice data and provide a response to the user as voice. For example, tell the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself."

[0221] 11. The user listens to the voice response from the smart speaker and takes the necessary action, for example, taking action to relax.

[0222] Specific examples

[0223] For example, consider the case where a user says, "I'm in a bad mood today, so I don't want to contact my friends." At this time, the device captures the voice, converts it into text, and sends the content "I'm in a bad mood today, so I don't want to contact my friends." to the server. The emotion engine then analyzes the user's emotion as "bad mood." The server passes this text and emotion information to the generation AI engine, which generates a response saying, "You seem to be feeling unwell, but it's important to take it easy and get some rest." The generated response text is then converted into voice data and sent back to the device. Finally, the device plays back this voice data, telling the user, "You seem to be feeling unwell, but it's important to take it easy and get some rest."

[0224] In this way, the elderly can easily obtain the information they need through voice, and receive easy-to-understand responses according to their specific emotional state, which not only improves their quality of life and alleviates their sense of loneliness, but also provides them with appropriate support.

[0225] The processing flow will be explained below.

[0226] Step 1:

[0227] A user speaks a question or command containing emotion into a smart speaker, for example, "I'm in a bad mood today, so I don't want to contact my friends."

[0228] Step 2:

[0229] The microphone on the device (smart speaker) captures the user's voice, and performs noise reduction and signal normalization during capture to optimize the voice data.

[0230] Step 3:

[0231] The device uses a speech recognition engine to convert the captured speech into corresponding text, for example, converting the speech "I'm in a bad mood today, so I don't want to call my friends" into the text "I'm in a bad mood today, so I don't want to call my friends."

[0232] Step 4:

[0233] The device uses an emotion engine to analyze parameters such as tone, tempo, and emphasis from the voice data to identify the user's emotion. For example, if the voice tone is low and the tempo is slow, the user's emotion is identified as "unhappy."

[0234] Step 5:

[0235] The device sends the converted text data, analyzed emotional information, and user meta-information to the server as an HTTP request. The request includes the text content, the user's age group, information literacy, and emotional information.

[0236] Step 6:

[0237] The server analyzes the received HTTP request and extracts text data and emotional information.

[0238] Step 7:

[0239] The server passes the extracted text data and emotional information to the generative AI engine.

[0240] Step 8:

[0241] The generation AI analyzes the text data and generates a response in easy-to-understand and appropriate language based on the user's profile and emotional information. For example, if the received text is "I'm in a bad mood today, so I don't want to contact my friends," and the emotional information is "bad mood," the generation AI will generate a response like, "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[0242] Step 9:

[0243] The server passes the generated response text to a speech synthesis engine and converts it into voice data.

[0244] Step 10:

[0245] The server sends the generated audio data to the terminal as an HTTP response.

[0246] Step 11:

[0247] The device uses its function to play back the received voice data and provides a response to the user as voice, for example, telling the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself."

[0248] Step 12:

[0249] The user hears a voice response from the smart speaker and takes necessary action based on it, for example, taking action to relax.

[0250] Example 2

[0251] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0252] Currently, there are voice recognition systems that allow elderly people to obtain information and easily communicate, but they lack the ability to accurately analyze emotional states and provide optimal responses based on those states. Furthermore, they are unable to generate responses that correspond to the elderly person's own profile information (age group and information literacy), resulting in a poor user experience. Furthermore, voice recognition accuracy decreases in noisy environments, resulting in situations where satisfactory performance cannot be achieved.

[0253] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0254] In this invention, the server includes means for removing noise from the input speech and normalizing the signal, means for analyzing emotions from the input speech, and means for converting the input speech to text. This makes it possible to accurately analyze the emotional state of an elderly person from their input speech and generate an optimal response based on that state. Furthermore, noise removal and signal normalization enable highly accurate speech recognition even in a variety of environments, improving the user experience.

[0255] "Means for accepting input by voice" refers to a function for capturing voice from a user using a voice input device.

[0256] The "means for removing noise from input audio and normalizing the signal" refers to a function for removing unnecessary noise from captured audio data and performing processing to obtain a stable audio signal.

[0257] "Means for analyzing emotions from input voice" refers to a function that analyzes parameters such as tone, tempo, and emphasis contained in voice data to identify the user's emotional state.

[0258] "Means for converting input speech to text" refers to a function that uses a speech recognition engine to convert captured speech data into corresponding text data.

[0259] "Means for transmitting the converted text and user profile information to a server via a network" refers to a function for transmitting data including the converted text data and user meta information to a server via a network.

[0260] "Means for analyzing text and emotional information received by the server" refers to a function for analyzing text data and emotional information received by the server and extracting necessary information.

[0261] "Means for generating a response using generative AI based on analyzed information" refers to the function of using a generative AI engine to create an optimal response based on the generated text data and emotional information.

[0262] The "means for converting the generated text into voice data" refers to a function for converting the generated response text into voice data using a voice synthesis engine and conveying it to the user.

[0263] "Means for receiving and playing back audio data" refers to the function of receiving audio data sent from the server and playing back that audio data.

[0264] The system of the present invention includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, an emotion engine, a voice synthesis engine, and a voice output device. The following describes the details of each component and how they work together.

[0265] First, the user speaks into a smart speaker. A smart speaker is a piece of hardware used for voice capture, collecting the user's voice using a built-in microphone. The system acts as a voice input device to accept the user's voice. This voice input device includes a noise reduction and signal normalization function, which normalizes the signal while removing ambient noise.

[0266] Next, while the device is capturing the audio, it uses an emotion engine to analyze the audio for emotional information. The emotion engine determines the user's emotional state based on parameters such as voice tone, tempo, and emphasis. This analysis can determine whether the user is feeling sad, happy, angry, etc.

[0267] After the voice is captured and the emotion is analyzed, the device uses a speech recognition engine to convert the voice into text data. For example, "I'm in a bad mood today, so I don't want to contact my friends." The voice is converted into text "I'm in a bad mood today, so I don't want to contact my friends." This text data and the user's profile information (such as age group, information literacy, and emotional information) are sent to the server as an HTTP request.

[0268] The server analyzes the received text data and emotional information and passes the necessary information to the generative AI engine. The generative AI engine generates an appropriate response based on the user's profile information and emotional information. For example, based on the text "I'm in a bad mood today, so I don't want to contact my friends," and the emotional information "bad mood," the response generated is "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[0269] The generated response text is again passed to the speech synthesis engine by the server and converted into voice data, which is finally sent to the terminal as an HTTP response.

[0270] The device plays back the received voice data and provides a response to the user as voice. For example, by telling the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself," the user can take appropriate action.

[0271] Specific examples

[0272] For example, if a user says, "I'm in a bad mood today, so I don't want to contact my friends," the system works as follows:

[0273] 1. The device captures the audio and performs noise reduction and signal normalization.

[0274] 2. The device analyzes the emotional information of "bad mood" using an emotion engine.

[0275] 3. The device converts the speech into text data such as "I'm in a bad mood today, so I don't want to contact my friends," and sends this text data and profile information to the server.

[0276] 4. The server analyzes the received data, and the AI ​​engine generates a response saying, "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[0277] 5. The server converts the response text into voice data and sends it to the device.

[0278] 6. The device plays audio data and tells the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself."

[0279] This process allows seniors to easily obtain the information they need through voice and provides easy-to-understand responses tailored to their specific emotional states, thereby improving their quality of life, reducing loneliness, and providing appropriate support.

[0280] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0281] Step 1:

[0282] A user speaks to a smart speaker, for example, "I'm in a bad mood today, so I don't want to contact my friends." This speech is the input for the entire system, and it is the starting point for processing.

[0283] Step 2:

[0284] The smart speaker of the device captures the user's voice using the smart speaker's microphone. The captured voice data is accompanied by noise data, so noise removal and signal normalization are required in the next processing steps.

[0285] Step 3:

[0286] The device performs noise reduction and signal normalization on the audio data. At this stage, background noise is removed from the captured audio data and the amplitude of the audio signal is adjusted to obtain clear audio data. This clear audio data is the input for the next step.

[0287] Step 4:

[0288] The device uses an emotion engine to analyze emotional information from the normalized voice data. The emotion engine analyzes the tone, tempo, and emphasis of the voice to determine the emotion the user is feeling when speaking. In this case, the analyzed emotion is displeased. The output is the voice data before it is converted into text and the emotional information (displeased).

[0289] Step 5:

[0290] The device uses a speech recognition engine to convert the normalized voice data into text data. This conversion process generates the text, "I'm in a bad mood today, so I don't want to contact my friends." The output is this text data.

[0291] Step 6:

[0292] The device sends an HTTP request to the server, including the converted text data and the user's profile information (such as age group, information literacy, and emotional information). The server receives the text data and the user profile information.

[0293] Step 7:

[0294] The server analyzes the received HTTP request and extracts text data and emotional information. In this example, the text data "I'm in a bad mood today, so I don't want to contact my friends." and emotional information (bad mood) are extracted within the server. The output is the analyzed text data and emotional information.

[0295] Step 8:

[0296] The server passes the extracted text data and emotional information to the generative AI engine. The generative AI engine generates the optimal response based on the user's profile information and emotional information. In this case, the response generated is, "You seem to be feeling unwell, but it's important to take it easy and get some rest." The output is the generated response text.

[0297] Step 9:

[0298] The server passes the generated response text to the speech synthesis engine, which converts the text into speech data. The speech synthesis engine generates speech data saying, "You seem to be feeling unwell, but it's important to take a rest and not push yourself." The output is speech data.

[0299] Step 10:

[0300] The server sends the generated audio data to the terminal as an HTTP response, and the terminal receives this audio data.

[0301] Step 11:

[0302] The device plays the received voice data and provides a response to the user. For example, a smart speaker might say to the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself." The user can listen to this voice response and take appropriate action.

[0303] (Application example 2)

[0304] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0305] When elderly people order food delivery services by voice, there is a need for a system that can provide appropriate menu suggestions and allow them to proceed with the ordering process comfortably and without stress. In particular, it is a challenge to increase user satisfaction and improve the quality of life by making suggestions that correspond to the emotional state of the elderly.

[0306] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0307] In this invention, the server includes means for accepting input by voice, means for converting the input voice into text, means for transmitting the converted text to the server via a network, means for analyzing the text and emotional information received by the server and generating easy-to-understand expressions and suggestions according to the user's emotional state, means for converting the reconstructed text and suggestions into voice data, and means for receiving and playing the voice data. This enables elderly people to receive appropriate meal suggestions according to their emotional state when ordering food delivery by voice.

[0308] "Means for accepting voice input" refers to a device that detects voices emitted by the user and inputs them into the system, such as a microphone or the voice input function of a smartphone.

[0309] The "means for converting input voice into text" refers to a voice recognition engine that converts voice data into character data, and has the function of converting voice into text using voice analysis technology.

[0310] The "means for transmitting the converted text to a server via a network" is a communication module for transmitting the converted text data to a server via the Internet or another network.

[0311] "Generation means that analyzes the text and emotional information received by the server and generates easy-to-understand expressions and suggestions according to the user's emotional state" refers to a generative AI engine or emotion analysis engine that generates appropriate responses and suggestions based on the received text and the user's emotional information.

[0312] The "means for converting the reconstructed text and suggestions into speech data" refers to a speech synthesis engine that converts the generated text and suggestions into speech data, and has the function of converting the generated information into easy-to-listen speech.

[0313] "Means for receiving and playing audio data" refers to a device that receives the converted audio data and plays it in a form that the user can hear, such as a speaker or the audio output function of a smartphone.

[0314] This invention realizes a system that allows elderly people to easily order food delivery by voice by using a voice input device, a voice recognition engine, a network communication module, a generative AI engine, an emotion engine, a voice synthesis engine, and a voice output device.

[0315] When using a smartphone, the user expresses their order preference using the smartphone's voice input function. For example, they might say, "I'm tired, please give me something to eat that will energize me." This voice is captured by the smartphone's microphone. The captured voice data is converted into text data by a speech recognition engine. Specifically, the voice is converted into text using the Google Cloud Speech-to-Text API.

[0316] The converted text is then sent to a server via a network. This network communication module uses HTTP communication. The server receives the received text and the user's emotional information, which is analyzed by an emotion engine. This emotion analysis uses IBM Watson (registered trademark) Tone Analyzer.

[0317] The server then uses a generative AI engine (such as the OpenAI GPT-4 API) to generate easy-to-understand expressions and suggestions based on the emotional state of the user based on the received text and emotional information. For example, if the user is tired, a suggestion such as "Would you like some nutritious chicken soup and fresh fruit?" is generated. This generated text is then converted into speech data using a speech synthesis engine. Amazon Polly is used for this speech synthesis engine.

[0318] The converted voice data is then sent back to the smartphone via the network and played back through the smartphone's speaker, allowing the user to listen to the menu of voice suggestions and make a selection from them.

[0319] As a specific example, if a user says, "I'm tired, please give me something to eat that will give me energy," the following prompt sentence will be sent to the generation AI.

[0320] Example prompt sentence:

[0321] User said: I'm tired, please give me something to eat to cheer me up. Tone: Sad. Suggest a meal.

[0322] Here's an example of a response suggested by the generative AI:

[0323] How about a nutritious chicken soup with vegetables and a side of fresh fruit to boost your energy?

[0324] In this way, a system has been realized that allows elderly people to order food delivery by voice without stress, and suggestions based on emotion analysis improve satisfaction. This also improves the quality of life for elderly people and makes their daily lives more convenient.

[0325] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0326] Step 1:

[0327] The user uses the smartphone's voice input function to verbally state their order. At this time, the smartphone's microphone captures the user's voice and stores it as voice data. The input is the user's voice, and the output is the captured voice data.

[0328] Step 2:

[0329] The device converts the captured voice data into text data using the Google Cloud Speech-to-Text API. The input is the voice data, and the output is the converted text data. The speech recognition engine analyzes the voice and generates the corresponding text.

[0330] Step 3:

[0331] The terminal sends the converted text data to the server via the network communication module. The input is text data, and the output is text data as an HTTP request. The network communication module is responsible for sending.

[0332] Step 4:

[0333] The server uses IBM Watson Tone Analyzer to analyze the received text data and the user's emotional information. The input is the received text data and the emotional parameters extracted from the user's voice, and the output is the analyzed emotional information. The emotional analysis engine identifies the emotional state.

[0334] Step 5:

[0335] The server passes the analyzed emotional information and text data to the OpenAI GPT-4 API, and the generative AI engine generates appropriate suggestions based on this information. The input is text data and emotional information, and the output is the generated suggested text. The generative AI engine generates the suggestions.

[0336] Step 6:

[0337] The server converts the generated suggested text into speech data using Amazon Polly. The input is the suggested text and the output is the generated speech data. The speech synthesis engine converts the text into speech.

[0338] Step 7:

[0339] The server sends the generated voice data to the terminal via the network. The input is the voice data, and the output is the voice data as an HTTP response. The network communication module is responsible for sending the voice data.

[0340] Step 8:

[0341] The device plays the received voice data and conveys the suggestion to the user. The input is the received voice data, and the output is a voice response to the user. The smartphone speaker performs the specific operation of playing the voice data.

[0342] Step 9:

[0343] The user listens to the voice response played from the terminal, selects a suggested menu from the list, and proceeds with the ordering process. For example, the user confirms the suggested menu and then issues another voice command such as "I would like to order this menu item," thereby completing the order. The input is the voice response from the terminal, and the output is the execution of the order based on the user's selection.

[0344] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0345] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0346] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0347] [Second embodiment]

[0348] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0349] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0350] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0351] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0352] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0353] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0354] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0355] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0356] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0357] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0358] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0359] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0360] The system of the present invention includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, a voice synthesis engine, and a voice output device, which allows elderly people to obtain information using only their voice and facilitate communication with family and relatives in remote locations.

[0361] Voice input process

[0362] 1. A user speaks a question or command to a smart speaker, such as "What's the weather like today?" or "Tell me how to apply for a pension."

[0363] 2. The microphone on the device (smart speaker) captures the user's voice. At this time, the device performs noise reduction and signal normalization to optimize the voice.

[0364] Voice recognition and transmission to the server

[0365] 3. The device uses a speech recognition engine to convert the captured speech into corresponding text, for example, converting the speech "What's the weather today?" into the text "What's the weather today?"

[0366] 4. The device sends this text data to the server via the network, along with meta-information such as the user's age group and information literacy.

[0367] Information generation by generative AI

[0368] 5. The server analyzes the received text data and passes it to the generative AI engine, which generates information in easy-to-understand language according to the user's age and literacy level.

[0369] 6. For example, if the received text is "Please tell me how to apply for a pension," the generation AI will generate the response "To apply for a pension, first get the necessary documents from city hall."

[0370] Audio generation and playback

[0371] 7. The server passes the generated response text to the speech synthesis engine and converts it into voice data. The speech synthesis engine converts the text "To apply for a pension, first get the necessary documents from city hall" into voice data.

[0372] 8. The server sends the audio data back to the terminal via the network.

[0373] Providing voice responses

[0374] 9. The device plays back the received voice data and communicates the response to the user. For example, a voice may be heard from the speaker saying, "To apply for a pension, first obtain the necessary documents from city hall."

[0375] 10. The user listens to this and takes the necessary steps or actions.

[0376] Specific examples

[0377] For example, consider the case where a user speaks to a smart speaker saying, "Tell me what the weather is today." At this time, the device captures the voice, converts it into text, and sends the message "What's the weather today?" to the server. The server passes this text to a generative AI engine, which generates a response saying, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." The generated response text is then converted into voice data and sent back to the device. Finally, the device plays back this voice data, telling the user, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out."

[0378] In this way, the elderly can easily obtain the necessary information by voice, and can also receive easy-to-understand responses according to specific conditions, thereby improving the quality of life for the elderly.

[0379] The processing flow will be explained below.

[0380] Step 1:

[0381] The user speaks a question or command to the smart speaker, for example, "What's the weather like today?" or "How do I apply for a pension?"

[0382] Step 2:

[0383] The microphone on the device (smart speaker) captures the user's voice, which includes collecting and pre-processing the voice signal (noise removal and signal normalization).

[0384] Step 3:

[0385] The device uses a speech recognition engine to convert the captured speech into text data, for example converting the speech "What's the weather today?" into text "What's the weather today?"

[0386] Step 4:

[0387] The device sends the converted text data and user meta-information (such as age group and information literacy) to the server as an HTTP request. The request includes the text content as well as profile information for each user.

[0388] Step 5:

[0389] The server analyzes the received HTTP request and extracts text data, which it then passes to the AI ​​generation engine.

[0390] Step 6:

[0391] The generative AI analyzes the text data and generates a response in easy-to-understand language based on the user's profile information. For example, in response to the text "Please tell me how to apply for a pension," it creates the response "To apply for a pension, first obtain the necessary documents from city hall."

[0392] Step 7:

[0393] The server passes the generated response text to a speech synthesis engine, which converts the text into speech data that can be read aloud to the user.

[0394] Step 8:

[0395] The server sends the generated audio data to the terminal as an HTTP response.

[0396] Step 9:

[0397] The device uses its function to play back the received voice data and provides a response to the user as voice. For example, it tells the user, "To apply for a pension, first get the necessary documents from your city hall."

[0398] Step 10:

[0399] The user hears a voice response from the smart speaker and takes the necessary action, for example, preparing to go to city hall.

[0400] The above processing flow allows users, especially elderly people, to easily obtain information using voice and to smoothly communicate with family and relatives.

[0401] Example 1

[0402] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0403] Elderly people and users with low information literacy need a system that makes it easier to obtain information and allows smooth communication with family and relatives in remote locations using only voice. However, previous voice recognition systems have had difficulty providing appropriate information according to the user's age and information literacy, and have also had problems with noise removal and signal normalization.

[0404] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0405] In this invention, the server includes means for accepting voice input, means for converting the input voice into text, means for transmitting the converted voice text to the server via a network, means for the server to analyze the received text and input it into the generative AI model, means for converting the response generated by the generative AI model into voice data, and means for receiving and playing the voice data. This enables the provision of easy-to-understand information according to the user's age group and information literacy, as well as noise removal and signal normalization during voice input.

[0406] "Means for receiving input by voice" refers to a device or component that has the function of capturing voice uttered by a user and inputting it into the system.

[0407] "Means for converting input voice into text" refers to a device or software that has the function of appropriately analyzing the captured voice signal and converting its content into text data.

[0408] "Means for transmitting speech-converted text to a server via a network" refers to devices or software that have the function of transmitting text data converted from speech to a server via a network such as the Internet.

[0409] "Means for analyzing text received by the server and inputting it into the generative AI model" refers to devices or software that have the function of analyzing text data received via a network and inputting appropriate prompt sentences into the generative AI model based on the analysis results.

[0410] "Means for converting responses generated by a generative AI model into voice data" refers to devices or software that have the function of converting text-format responses generated by a generative AI model into voice data.

[0411] "Means for receiving and reproducing audio data" refers to a device or component that has the function of reproducing audio data received via a network and providing information to the user by audio.

[0412] "Noise reduction" refers to technology that removes unnecessary background sounds and noises that are included in voice input, and clearly captures only the voice spoken by the user.

[0413] "Signal normalization" refers to a technique for adjusting the amplitude and volume of an audio signal to a certain range in order to improve the accuracy of voice recognition.

[0414] "Automatically adjusting responses according to the user's age group and information literacy" refers to the ability of the generative AI model to generate responses with optimal expressions and content that are tailored to the user's characteristics.

[0415] MODE FOR CARRYING OUT THE INVENTION

[0416] The system of the present invention combines a voice input device, a voice recognition engine, a network communication module, a generative AI engine, a voice synthesis engine, and a voice output device, allowing elderly people to obtain information using only their voice and facilitating communication with family and relatives in remote locations.

[0417] Hardware and Software Configuration

[0418] The server has a generative AI engine (e.g., OpenAI GPT-4) and a speech synthesis engine (e.g., Google Text-to-Speech or Amazon Polly). The terminal has a voice input device (e.g., a smart speaker), a speech recognition engine (e.g., Google Cloud Speech-to-Text or Amazon Transcribe), and a network communication module (e.g., Wi-Fi or LTE). The user uses these devices to obtain information by voice and receive voice responses.

[0419] Audio Input and Preprocessing

[0420] The user speaks questions or commands to the smart speaker, such as "What's the weather like today?" or "How do I apply for a pension?" The device uses a voice input device to capture the user's voice, then performs noise reduction and signal normalization to obtain an optimal voice signal.

[0421] Voice recognition and data transmission

[0422] The device uses a speech recognition engine to convert the captured voice into text. For example, it converts the voice "What's the weather today?" into text "What's the weather today?" The device then sends the converted text data to the server via the network communication module.

[0423] Text analysis and response generation

[0424] The server analyzes the received text data and inputs it as a prompt to the generative AI engine. Based on this prompt, the generative AI engine generates an appropriate response according to the user's age group and information literacy. For example, in response to the text "Please tell me how to apply for a pension," it generates the response "To apply for a pension, first obtain the necessary documents from city hall."

[0425] Speech synthesis and data transmission

[0426] The server passes the generated response text to a speech synthesis engine, which converts it into voice data. For example, the text "To apply for a pension, first get the necessary documents at city hall" is converted into voice data. The converted voice data is then sent back to the terminal via the network.

[0427] Playing a voice response

[0428] The device plays the received voice data and communicates the response to the user. For example, a voice may come from the speaker saying, "To apply for a pension, first get the necessary documents from your city hall." The user listens to this and takes the necessary steps or actions.

[0429] Specific examples

[0430] Specifically, imagine a case where a user asks a smart speaker, "What's the weather today?" At this time, the device captures the voice, converts it into text, "What's the weather today?", and sends it to the server. The server inputs the prompt text into a generation AI engine, which generates a response, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." This response is converted into voice data and sent back to the device. Finally, the device plays back this voice data, telling the user, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out."

[0431] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0432] Step 1:

[0433] A user speaks a question or command into a smart speaker, and the input audio signal is captured by the microphone. For example, "What's the weather like today?"

[0434] Step 2:

[0435] The device's audio input device receives the captured audio signal and performs noise reduction and signal normalization. Noise reduction removes background noise from the audio signal, and signal normalization adjusts the amplitude of the audio signal to an appropriate range, thereby making the input audio clearer.

[0436] Step 3:

[0437] The device uses a speech recognition engine to convert the preprocessed voice signal into text data. Specifically, the speech recognition engine analyzes the voice signal and generates text data such as "What's the weather like today?". This text data is output.

[0438] Step 4:

[0439] The terminal sends the converted text data to the server via the network communication module. When sending, meta-information such as the user's age group and information literacy level is also sent along with the text data. The text data and meta-information are sent to the server as output.

[0440] Step 5:

[0441] The server analyzes the received text data and meta information and inputs it as a prompt to the generative AI model. Specifically, the received text "What's the weather like today?" is processed as a prompt for the generative AI model, and an easy-to-understand response is generated that takes into account the user's age group and literacy level. "What's the weather like today?"

[0442] Step 6:

[0443] The generative AI model responds by generating text such as, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." This text data is output on the server.

[0444] Step 7:

[0445] The server passes the generated response text to the speech synthesis engine and converts it into voice data. The speech synthesis engine converts the text "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." into voice data. This voice data is generated.

[0446] Step 8:

[0447] The server then transmits the converted audio data to the terminal via the network, using a network protocol for transmission.

[0448] Step 9:

[0449] The device plays the received voice data and conveys the response to the user. The smart speaker's speaker will say, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." The user listens to this voice response and takes the necessary action.

[0450] Through the above steps, the user can easily obtain information and receive appropriate responses using only voice.

[0451] (Application example 1)

[0452] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0453] The challenge is to enable elderly people to easily obtain necessary information using only their voice, and to be able to quickly request assistance in an emergency. There is also a need to provide an environment where elderly people can easily check the safety of their homes and live with peace of mind.

[0454] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0455] In this invention, the server includes means for accepting voice input, means for converting the input voice into text, means for transmitting the converted text to the server via a network, means for reconstructing the text received by the server in an easy-to-understand manner, means for converting the reconstructed text into voice data, means for receiving and playing the voice data, and means for sending a notification to a contact specified based on the voice input in an emergency. This enables elderly people to obtain information, confirm their safety, and quickly request assistance in an emergency using only their voice.

[0456] "Means for accepting input via voice" refers to devices or technologies that have the function of capturing voices uttered by a user and inputting them into the system.

[0457] "Means for converting input speech into text" refers to technology or devices that analyze speech signals and convert them into corresponding text data.

[0458] "Means for transmitting the converted text to a server via a network" refers to a technology for transmitting text data converted from speech to a server via a network such as the Internet.

[0459] "Means for generating text received by the server in an easy-to-understand format" refers to technology or devices that automatically process and reconstruct text data so that it is easy for users to understand.

[0460] The "means for converting reconstructed text into speech data" refers to a technique or device that converts text data into a speech signal.

[0461] "Means for receiving and playing audio data" refers to a technique for playing audio data received from a server on a device such as a speaker.

[0462] "Means for sending notifications to designated contacts based on voice input in an emergency" refers to technology or devices that automatically send notifications to pre-set contacts when a user inputs a voice command indicating an emergency.

[0463] The present invention provides a home security system that allows elderly people to obtain information using only their voice, check the safety of their home, and use emergency notification functions. The system includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, a voice synthesis engine, and a voice output device.

[0464] System configuration and usage

[0465] 1. Voice Input Device

[0466] A voice input device is a device that receives voice instructions from a user, such as a smart speaker. The device has the ability to perform noise reduction and signal normalization. The user speaks questions or commands into the device.

[0467] 2. Speech Recognition Engine

[0468] The speech captured by the voice input device is converted into text by a speech recognition engine, which can use the speech_recognition package. This engine analyzes the speech and converts it into text data.

[0469] 3. Network Communication Module

[0470] The text generated by the speech recognition engine is sent to the server via the network communication module, which may also include user meta-information such as age and information literacy.

[0471] 4. Generative AI Engine

[0472] The server passes the received text data to a generative AI engine, which generates information in easy-to-understand language according to the user's age group and information literacy. This engine is realized using a generative AI model such as ChatGPT.

[0473] 5. Speech synthesis engine

[0474] The generated response text is converted into voice data by a speech synthesis engine. The gTTS package can be used for this voice synthesis. The server then transmits the generated voice data back to the terminal via the network.

[0475] 6. Audio Output Device

[0476] The terminal plays the received voice data and communicates the response to the user, where information is provided using a speaker.

[0477] 7. Emergency notification function

[0478] In the event of an emergency, the user can enter a designated voice command (e.g., "Help me") and the system will automatically send a notification to pre-registered contacts, enabling a quick response.

[0479] Specific examples

[0480] For example, suppose a user speaks to a smart speaker saying, "Tell me the lock status of the windows." At this time, the voice input device captures the voice, and the voice recognition engine converts it into text, "Tell me the lock status of the windows." This text is then sent to the server via the network. The server passes the text to the generative AI engine, which generates the response, "All windows are locked." This response text is converted into voice data by the speech synthesis engine and sent back to the terminal, allowing the user to hear the response, "All windows are locked," on the speaker.

[0481] Additionally, if the user says "help me," the system will immediately send a notification to designated emergency contacts, allowing seniors to quickly seek assistance in an emergency.

[0482] Prompt Sentence Examples

[0483] "Tell me if the windows are locked."

[0484] "help me."

[0485] This allows the system of the present invention to allow elderly people to obtain information using only their voice, check the safety of their homes, and request assistance in emergencies.

[0486] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0487] Step 1:

[0488] Users speak questions and commands to the smart speaker.

[0489] Input: Example spoken question: "How are the windows locked?"

[0490] Data processing: The microphone built into the smart speaker captures the audio and performs noise reduction and signal normalization.

[0491] Output: Optimized audio signal

[0492] Step 2:

[0493] The device uses a speech recognition engine to convert the captured speech into text.

[0494] Input: Optimized audio signal

[0495] Data calculation: A speech recognition engine (e.g., speech_recognition) analyzes the audio signal and converts it into corresponding text data.

[0496] Output: Text data example: "Please tell me if the windows are locked."

[0497] Step 3:

[0498] The terminal transmits the converted text to the server via the network communication module.

[0499] Input: Text data, user meta information (e.g., age group, information literacy)

[0500] Data processing: Text data and meta information are packaged into packets and sent over the network.

[0501] Output: Receives text data and meta information to the server

[0502] Step 4:

[0503] The server analyzes the received text and meta information and passes it to the generative AI engine.

[0504] Input: Text data, meta information

[0505] Data calculation: A generative AI engine reconstructs text into easy-to-understand expressions according to the user's age group and information literacy. Example: Using a generative AI model

[0506] Output: Reconstructed text data Example: "All windows are locked."

[0507] Step 5:

[0508] The server passes the reconstructed text to a speech synthesis engine, which converts it into speech data.

[0509] Input: Reconstructed text data

[0510] Data processing: A speech synthesis engine (e.g., gTTS) converts text data into speech signals and generates an audio file.

[0511] Output: Example speech data: "All windows are locked."

[0512] Step 6:

[0513] The server transmits the voice data again to the terminal via the network.

[0514] Input: Audio data

[0515] Data processing: Packetizing the audio data and sending it over the network.

[0516] Output: Receiving audio data to the device

[0517] Step 7:

[0518] The terminal plays back the received voice data and conveys the response to the user.

[0519] Input: Audio data

[0520] Specific behavior: Plays audio data through the speaker, telling the user, "All windows are locked."

[0521] Output: Providing a voice response to the user

[0522] Step 8:

[0523] The user can enter further voice commands as needed, including emergency commands such as "help."

[0524] Input: Example user voice command: "Help me"

[0525] Data calculation: The smart speaker captures the voice again and converts it into text using the recognition engine. The server receives the text "Help" and sends an emergency notification to the specified contacts.

[0526] Output: Sending an emergency notification and acknowledging it

[0527] In this way, the system allows seniors to check the safety of their homes, obtain necessary information, and quickly call for help in emergencies using only their voice.

[0528] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0529] The system of the present invention includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, an emotion engine, a voice synthesis engine, and a voice output device, which allows elderly people to obtain information solely through voice and facilitate communication with family and relatives based on emotion analysis.

[0530] Voice input process

[0531] 1. A user speaks questions, commands, and everyday conversations into a smart speaker. For example, they can say, "I'm in a bad mood today, so I don't want to call my friends."

[0532] 2. The microphone on the device (smart speaker) captures the user's voice. At this time, the device performs noise reduction and signal normalization to optimize the voice.

[0533] 3. The device uses an emotion engine to analyze parameters such as tone, tempo, and emphasis from the captured voice to identify the user's emotion. For example, if the voice tone is low and the tempo is slow, it will be determined that the user is in a bad mood.

[0534] Voice recognition and transmission to the server

[0535] 4. The device uses a speech recognition engine to convert the captured speech into corresponding text data, for example, converting the speech "I'm in a bad mood today, so I don't want to call my friends" into text "I'm in a bad mood today, so I don't want to call my friends."

[0536] 5. The device sends the converted text data and user meta-information (such as age group, information literacy, and emotional information) to the server as an HTTP request. The request includes the text content as well as profile information for each user.

[0537] Information generation by generative AI

[0538] 6. The server analyzes the received HTTP request, extracts text data and emotional information, and passes this data to the generative AI engine.

[0539] 7. The generation AI analyzes the text data and generates a response in easy-to-understand language based on the user's profile information and emotional information. For example, if the received text reads, "I'm in a bad mood today, so I don't want to contact my friends," and the emotional information is bad, the generation AI will generate a response like, "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[0540] Audio generation and playback

[0541] 8. The server passes the generated response text to a speech synthesis engine to convert the text into speech data that can be read aloud to the user.

[0542] 9. The server sends the generated audio data to the device as an HTTP response.

[0543] Providing voice responses

[0544] 10. Use the device's function to play back the received voice data and provide a response to the user as voice. For example, tell the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself."

[0545] 11. The user listens to the voice response from the smart speaker and takes the necessary action, for example, taking action to relax.

[0546] Specific examples

[0547] For example, consider the case where a user says, "I'm in a bad mood today, so I don't want to contact my friends." At this time, the device captures the voice, converts it into text, and sends the content "I'm in a bad mood today, so I don't want to contact my friends." to the server. The emotion engine then analyzes the user's emotion as "bad mood." The server passes this text and emotion information to the generation AI engine, which generates a response saying, "You seem to be feeling unwell, but it's important to take it easy and get some rest." The generated response text is then converted into voice data and sent back to the device. Finally, the device plays back this voice data, telling the user, "You seem to be feeling unwell, but it's important to take it easy and get some rest."

[0548] In this way, the elderly can easily obtain the information they need through voice, and receive easy-to-understand responses according to their specific emotional state, which not only improves their quality of life and alleviates their sense of loneliness, but also provides them with appropriate support.

[0549] The processing flow will be explained below.

[0550] Step 1:

[0551] A user speaks a question or command containing emotion into a smart speaker, for example, "I'm in a bad mood today, so I don't want to contact my friends."

[0552] Step 2:

[0553] The microphone on the device (smart speaker) captures the user's voice, and performs noise reduction and signal normalization during capture to optimize the voice data.

[0554] Step 3:

[0555] The device uses a speech recognition engine to convert the captured speech into corresponding text, for example, converting the speech "I'm in a bad mood today, so I don't want to call my friends" into the text "I'm in a bad mood today, so I don't want to call my friends."

[0556] Step 4:

[0557] The device uses an emotion engine to analyze parameters such as tone, tempo, and emphasis from the voice data to identify the user's emotion. For example, if the voice tone is low and the tempo is slow, the user's emotion is identified as "unhappy."

[0558] Step 5:

[0559] The device sends the converted text data, analyzed emotional information, and user meta-information to the server as an HTTP request. The request includes the text content, the user's age group, information literacy, and emotional information.

[0560] Step 6:

[0561] The server analyzes the received HTTP request and extracts text data and emotional information.

[0562] Step 7:

[0563] The server passes the extracted text data and emotional information to the generative AI engine.

[0564] Step 8:

[0565] The generation AI analyzes the text data and generates a response in easy-to-understand and appropriate language based on the user's profile and emotional information. For example, if the received text is "I'm in a bad mood today, so I don't want to contact my friends," and the emotional information is "bad mood," the generation AI will generate a response like, "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[0566] Step 9:

[0567] The server passes the generated response text to a speech synthesis engine and converts it into voice data.

[0568] Step 10:

[0569] The server sends the generated audio data to the terminal as an HTTP response.

[0570] Step 11:

[0571] The device uses its function to play back the received voice data and provides a response to the user as voice, for example, telling the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself."

[0572] Step 12:

[0573] The user hears a voice response from the smart speaker and takes necessary action based on it, for example, taking action to relax.

[0574] Example 2

[0575] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0576] Currently, there are voice recognition systems that allow elderly people to obtain information and easily communicate, but they lack the ability to accurately analyze emotional states and provide optimal responses based on those states. Furthermore, they are unable to generate responses that correspond to the elderly person's own profile information (age group and information literacy), resulting in a poor user experience. Furthermore, voice recognition accuracy decreases in noisy environments, resulting in situations where satisfactory performance cannot be achieved.

[0577] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0578] In this invention, the server includes means for removing noise from the input speech and normalizing the signal, means for analyzing emotions from the input speech, and means for converting the input speech to text. This makes it possible to accurately analyze the emotional state of an elderly person from their input speech and generate an optimal response based on that state. Furthermore, noise removal and signal normalization enable highly accurate speech recognition even in a variety of environments, improving the user experience.

[0579] "Means for accepting input by voice" refers to a function for capturing voice from a user using a voice input device.

[0580] The "means for removing noise from input audio and normalizing the signal" refers to a function for removing unnecessary noise from captured audio data and performing processing to obtain a stable audio signal.

[0581] "Means for analyzing emotions from input voice" refers to a function that analyzes parameters such as tone, tempo, and emphasis contained in voice data to identify the user's emotional state.

[0582] "Means for converting input speech to text" refers to a function that uses a speech recognition engine to convert captured speech data into corresponding text data.

[0583] "Means for transmitting the converted text and user profile information to a server via a network" refers to a function for transmitting data including the converted text data and user meta information to a server via a network.

[0584] "Means for analyzing text and emotional information received by the server" refers to a function for analyzing text data and emotional information received by the server and extracting necessary information.

[0585] "Means for generating a response using generative AI based on analyzed information" refers to the function of using a generative AI engine to create an optimal response based on the generated text data and emotional information.

[0586] The "means for converting the generated text into voice data" refers to a function for converting the generated response text into voice data using a voice synthesis engine and conveying it to the user.

[0587] "Means for receiving and playing back audio data" refers to the function of receiving audio data sent from the server and playing back that audio data.

[0588] The system of the present invention includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, an emotion engine, a voice synthesis engine, and a voice output device. The following describes the details of each component and how they work together.

[0589] First, the user speaks into a smart speaker. A smart speaker is a piece of hardware used for voice capture, collecting the user's voice using a built-in microphone. The system acts as a voice input device to accept the user's voice. This voice input device includes a noise reduction and signal normalization function, which normalizes the signal while removing ambient noise.

[0590] Next, while the device is capturing the audio, it uses an emotion engine to analyze the audio for emotional information. The emotion engine determines the user's emotional state based on parameters such as voice tone, tempo, and emphasis. This analysis can determine whether the user is feeling sad, happy, angry, etc.

[0591] After the voice is captured and the emotion is analyzed, the device uses a speech recognition engine to convert the voice into text data. For example, "I'm in a bad mood today, so I don't want to contact my friends." The voice is converted into text "I'm in a bad mood today, so I don't want to contact my friends." This text data and the user's profile information (such as age group, information literacy, and emotional information) are sent to the server as an HTTP request.

[0592] The server analyzes the received text data and emotional information and passes the necessary information to the generative AI engine. The generative AI engine generates an appropriate response based on the user's profile information and emotional information. For example, based on the text "I'm in a bad mood today, so I don't want to contact my friends," and the emotional information "bad mood," the response generated is "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[0593] The generated response text is again passed to the speech synthesis engine by the server and converted into voice data, which is finally sent to the terminal as an HTTP response.

[0594] The device plays back the received voice data and provides a response to the user as voice. For example, by telling the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself," the user can take appropriate action.

[0595] Specific examples

[0596] For example, if a user says, "I'm in a bad mood today, so I don't want to contact my friends," the system works as follows:

[0597] 1. The device captures the audio and performs noise reduction and signal normalization.

[0598] 2. The device analyzes the emotional information of "bad mood" using an emotion engine.

[0599] 3. The device converts the speech into text data such as "I'm in a bad mood today, so I don't want to contact my friends," and sends this text data and profile information to the server.

[0600] 4. The server analyzes the received data, and the AI ​​engine generates a response saying, "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[0601] 5. The server converts the response text into voice data and sends it to the device.

[0602] 6. The device plays audio data and tells the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself."

[0603] This process allows seniors to easily obtain the information they need through voice and provides easy-to-understand responses tailored to their specific emotional states, thereby improving their quality of life, reducing loneliness, and providing appropriate support.

[0604] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0605] Step 1:

[0606] A user speaks to a smart speaker, for example, "I'm in a bad mood today, so I don't want to contact my friends." This speech is the input for the entire system, and it is the starting point for processing.

[0607] Step 2:

[0608] The smart speaker of the device captures the user's voice using the smart speaker's microphone. The captured voice data is accompanied by noise data, so noise removal and signal normalization are required in the next processing steps.

[0609] Step 3:

[0610] The device performs noise reduction and signal normalization on the audio data. At this stage, background noise is removed from the captured audio data and the amplitude of the audio signal is adjusted to obtain clear audio data. This clear audio data is the input for the next step.

[0611] Step 4:

[0612] The device uses an emotion engine to analyze emotional information from the normalized voice data. The emotion engine analyzes the tone, tempo, and emphasis of the voice to determine the emotion the user is feeling when speaking. In this case, the analyzed emotion is displeased. The output is the voice data before it is converted into text and the emotional information (displeased).

[0613] Step 5:

[0614] The device uses a speech recognition engine to convert the normalized voice data into text data. This conversion process generates the text, "I'm in a bad mood today, so I don't want to contact my friends." The output is this text data.

[0615] Step 6:

[0616] The device sends an HTTP request to the server, including the converted text data and the user's profile information (such as age group, information literacy, and emotional information). The server receives the text data and the user profile information.

[0617] Step 7:

[0618] The server analyzes the received HTTP request and extracts text data and emotional information. In this example, the text data "I'm in a bad mood today, so I don't want to contact my friends." and emotional information (bad mood) are extracted within the server. The output is the analyzed text data and emotional information.

[0619] Step 8:

[0620] The server passes the extracted text data and emotional information to the generative AI engine. The generative AI engine generates the optimal response based on the user's profile information and emotional information. In this case, the response generated is, "You seem to be feeling unwell, but it's important to take it easy and get some rest." The output is the generated response text.

[0621] Step 9:

[0622] The server passes the generated response text to the speech synthesis engine, which converts the text into speech data. The speech synthesis engine generates speech data saying, "You seem to be feeling unwell, but it's important to take a rest and not push yourself." The output is speech data.

[0623] Step 10:

[0624] The server sends the generated audio data to the terminal as an HTTP response, and the terminal receives this audio data.

[0625] Step 11:

[0626] The device plays the received voice data and provides a response to the user. For example, a smart speaker might say to the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself." The user can listen to this voice response and take appropriate action.

[0627] (Application example 2)

[0628] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0629] When elderly people order food delivery services by voice, there is a need for a system that can provide appropriate menu suggestions and allow them to proceed with the ordering process comfortably and without stress. In particular, it is a challenge to increase user satisfaction and improve the quality of life by making suggestions that correspond to the emotional state of the elderly.

[0630] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0631] In this invention, the server includes means for accepting input by voice, means for converting the input voice into text, means for transmitting the converted text to the server via a network, means for analyzing the text and emotional information received by the server and generating easy-to-understand expressions and suggestions according to the user's emotional state, means for converting the reconstructed text and suggestions into voice data, and means for receiving and playing the voice data. This enables elderly people to receive appropriate meal suggestions according to their emotional state when ordering food delivery by voice.

[0632] "Means for accepting voice input" refers to a device that detects voices emitted by the user and inputs them into the system, such as a microphone or the voice input function of a smartphone.

[0633] The "means for converting input voice into text" refers to a voice recognition engine that converts voice data into character data, and has the function of converting voice into text using voice analysis technology.

[0634] The "means for transmitting the converted text to a server via a network" is a communication module for transmitting the converted text data to a server via the Internet or another network.

[0635] "Generation means that analyzes the text and emotional information received by the server and generates easy-to-understand expressions and suggestions according to the user's emotional state" refers to a generative AI engine or emotion analysis engine that generates appropriate responses and suggestions based on the received text and the user's emotional information.

[0636] The "means for converting the reconstructed text and suggestions into speech data" refers to a speech synthesis engine that converts the generated text and suggestions into speech data, and has the function of converting the generated information into easy-to-listen speech.

[0637] "Means for receiving and playing audio data" refers to a device that receives the converted audio data and plays it in a form that the user can hear, such as a speaker or the audio output function of a smartphone.

[0638] This invention realizes a system that allows elderly people to easily order food delivery by voice by using a voice input device, a voice recognition engine, a network communication module, a generative AI engine, an emotion engine, a voice synthesis engine, and a voice output device.

[0639] When using a smartphone, the user expresses their order preference using the smartphone's voice input function. For example, they might say, "I'm tired, please give me something to eat that will energize me." This voice is captured by the smartphone's microphone. The captured voice data is converted into text data by a speech recognition engine. Specifically, the voice is converted into text using the Google Cloud Speech-to-Text API.

[0640] The converted text is then sent to the server via the network. This network communication module uses HTTP. The server receives the received text and the user's emotional information, which is analyzed by the emotion engine. This emotion analysis uses IBM Watson Tone Analyzer.

[0641] The server then uses a generative AI engine (such as the OpenAI GPT-4 API) to generate easy-to-understand expressions and suggestions based on the emotional state of the user based on the received text and emotional information. For example, if the user is tired, a suggestion such as "Would you like some nutritious chicken soup and fresh fruit?" is generated. This generated text is then converted into speech data using a speech synthesis engine. Amazon Polly is used for this speech synthesis engine.

[0642] The converted voice data is then sent back to the smartphone via the network and played back through the smartphone's speaker, allowing the user to listen to the menu of voice suggestions and make a selection from them.

[0643] As a specific example, if a user says, "I'm tired, please give me something to eat that will give me energy," the following prompt sentence will be sent to the generation AI.

[0644] Example prompt sentence:

[0645] User said: I'm tired, please give me something to eat to cheer me up. Tone: Sad. Suggest a meal.

[0646] Here's an example of a response suggested by the generative AI:

[0647] How about a nutritious chicken soup with vegetables and a side of fresh fruit to boost your energy?

[0648] In this way, a system has been realized that allows elderly people to order food delivery by voice without stress, and suggestions based on emotion analysis improve satisfaction. This also improves the quality of life for elderly people and makes their daily lives more convenient.

[0649] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0650] Step 1:

[0651] The user uses the smartphone's voice input function to verbally state their order. At this time, the smartphone's microphone captures the user's voice and stores it as voice data. The input is the user's voice, and the output is the captured voice data.

[0652] Step 2:

[0653] The device converts the captured voice data into text data using the Google Cloud Speech-to-Text API. The input is the voice data, and the output is the converted text data. The speech recognition engine analyzes the voice and generates the corresponding text.

[0654] Step 3:

[0655] The terminal sends the converted text data to the server via the network communication module. The input is text data, and the output is text data as an HTTP request. The network communication module is responsible for sending.

[0656] Step 4:

[0657] The server uses IBM Watson Tone Analyzer to analyze the received text data and the user's emotional information. The input is the received text data and the emotional parameters extracted from the user's voice, and the output is the analyzed emotional information. The emotional analysis engine identifies the emotional state.

[0658] Step 5:

[0659] The server passes the analyzed emotional information and text data to the OpenAI GPT-4 API, and the generative AI engine generates appropriate suggestions based on this information. The input is text data and emotional information, and the output is the generated suggested text. The generative AI engine generates the suggestions.

[0660] Step 6:

[0661] The server converts the generated suggested text into speech data using Amazon Polly. The input is the suggested text and the output is the generated speech data. The speech synthesis engine converts the text into speech.

[0662] Step 7:

[0663] The server sends the generated voice data to the terminal via the network. The input is the voice data, and the output is the voice data as an HTTP response. The network communication module is responsible for sending the voice data.

[0664] Step 8:

[0665] The device plays the received voice data and conveys the suggestion to the user. The input is the received voice data, and the output is a voice response to the user. The smartphone speaker performs the specific operation of playing the voice data.

[0666] Step 9:

[0667] The user listens to the voice response played from the terminal, selects a suggested menu from the list, and proceeds with the ordering process. For example, the user confirms the suggested menu and then issues another voice command such as "I would like to order this menu item," thereby completing the order. The input is the voice response from the terminal, and the output is the execution of the order based on the user's selection.

[0668] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0669] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0670] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0671] [Third embodiment]

[0672] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0673] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0674] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0675] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0676] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0677] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0678] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0679] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0680] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0681] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0682] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0683] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0684] The system of the present invention includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, a voice synthesis engine, and a voice output device, which allows elderly people to obtain information using only their voice and facilitate communication with family and relatives in remote locations.

[0685] Voice input process

[0686] 1. A user speaks a question or command to a smart speaker, such as "What's the weather like today?" or "Tell me how to apply for a pension."

[0687] 2. The microphone on the device (smart speaker) captures the user's voice. At this time, the device performs noise reduction and signal normalization to optimize the voice.

[0688] Voice recognition and transmission to the server

[0689] 3. The device uses a speech recognition engine to convert the captured speech into corresponding text, for example, converting the speech "What's the weather today?" into the text "What's the weather today?"

[0690] 4. The device sends this text data to the server via the network, along with meta-information such as the user's age group and information literacy.

[0691] Information generation by generative AI

[0692] 5. The server analyzes the received text data and passes it to the generative AI engine, which generates information in easy-to-understand language according to the user's age and literacy level.

[0693] 6. For example, if the received text is "Please tell me how to apply for a pension," the generation AI will generate the response "To apply for a pension, first get the necessary documents from city hall."

[0694] Audio generation and playback

[0695] 7. The server passes the generated response text to the speech synthesis engine and converts it into voice data. The speech synthesis engine converts the text "To apply for a pension, first get the necessary documents from city hall" into voice data.

[0696] 8. The server sends the audio data back to the terminal via the network.

[0697] Providing voice responses

[0698] 9. The device plays back the received voice data and communicates the response to the user. For example, a voice may be heard from the speaker saying, "To apply for a pension, first obtain the necessary documents from city hall."

[0699] 10. The user listens to this and takes the necessary steps or actions.

[0700] Specific examples

[0701] For example, consider the case where a user speaks to a smart speaker saying, "Tell me what the weather is today." At this time, the device captures the voice, converts it into text, and sends the message "What's the weather today?" to the server. The server passes this text to a generative AI engine, which generates a response saying, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." The generated response text is then converted into voice data and sent back to the device. Finally, the device plays back this voice data, telling the user, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out."

[0702] In this way, the elderly can easily obtain the necessary information by voice, and can also receive easy-to-understand responses according to specific conditions, thereby improving the quality of life for the elderly.

[0703] The processing flow will be explained below.

[0704] Step 1:

[0705] The user speaks a question or command to the smart speaker, for example, "What's the weather like today?" or "How do I apply for a pension?"

[0706] Step 2:

[0707] The microphone on the device (smart speaker) captures the user's voice, which includes collecting and pre-processing the voice signal (noise removal and signal normalization).

[0708] Step 3:

[0709] The device uses a speech recognition engine to convert the captured speech into text data, for example converting the speech "What's the weather today?" into text "What's the weather today?"

[0710] Step 4:

[0711] The device sends the converted text data and user meta-information (such as age group and information literacy) to the server as an HTTP request. The request includes the text content as well as profile information for each user.

[0712] Step 5:

[0713] The server analyzes the received HTTP request and extracts text data, which it then passes to the AI ​​generation engine.

[0714] Step 6:

[0715] The generative AI analyzes the text data and generates a response in easy-to-understand language based on the user's profile information. For example, in response to the text "Please tell me how to apply for a pension," it creates the response "To apply for a pension, first obtain the necessary documents from city hall."

[0716] Step 7:

[0717] The server passes the generated response text to a speech synthesis engine, which converts the text into speech data that can be read aloud to the user.

[0718] Step 8:

[0719] The server sends the generated audio data to the terminal as an HTTP response.

[0720] Step 9:

[0721] The device uses its function to play back the received voice data and provides a response to the user as voice. For example, it tells the user, "To apply for a pension, first get the necessary documents from your city hall."

[0722] Step 10:

[0723] The user hears a voice response from the smart speaker and takes the necessary action, for example, preparing to go to city hall.

[0724] The above processing flow allows users, especially elderly people, to easily obtain information using voice and to smoothly communicate with family and relatives.

[0725] Example 1

[0726] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0727] Elderly people and users with low information literacy need a system that makes it easier to obtain information and allows smooth communication with family and relatives in remote locations using only voice. However, previous voice recognition systems have had difficulty providing appropriate information according to the user's age and information literacy, and have also had problems with noise removal and signal normalization.

[0728] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0729] In this invention, the server includes means for accepting voice input, means for converting the input voice into text, means for transmitting the converted voice text to the server via a network, means for the server to analyze the received text and input it into the generative AI model, means for converting the response generated by the generative AI model into voice data, and means for receiving and playing the voice data. This enables the provision of easy-to-understand information according to the user's age group and information literacy, as well as noise removal and signal normalization during voice input.

[0730] "Means for receiving input by voice" refers to a device or component that has the function of capturing voice uttered by a user and inputting it into the system.

[0731] "Means for converting input voice into text" refers to a device or software that has the function of appropriately analyzing the captured voice signal and converting its content into text data.

[0732] "Means for transmitting speech-converted text to a server via a network" refers to devices or software that have the function of transmitting text data converted from speech to a server via a network such as the Internet.

[0733] "Means for analyzing text received by the server and inputting it into the generative AI model" refers to devices or software that have the function of analyzing text data received via a network and inputting appropriate prompt sentences into the generative AI model based on the analysis results.

[0734] "Means for converting responses generated by a generative AI model into voice data" refers to devices or software that have the function of converting text-format responses generated by a generative AI model into voice data.

[0735] "Means for receiving and reproducing audio data" refers to a device or component that has the function of reproducing audio data received via a network and providing information to the user by audio.

[0736] "Noise reduction" refers to technology that removes unnecessary background sounds and noises that are included in voice input, and clearly captures only the voice spoken by the user.

[0737] "Signal normalization" refers to a technique for adjusting the amplitude and volume of an audio signal to a certain range in order to improve the accuracy of voice recognition.

[0738] "Automatically adjusting responses according to the user's age group and information literacy" refers to the ability of the generative AI model to generate responses with optimal expressions and content that are tailored to the user's characteristics.

[0739] MODE FOR CARRYING OUT THE INVENTION

[0740] The system of the present invention combines a voice input device, a voice recognition engine, a network communication module, a generative AI engine, a voice synthesis engine, and a voice output device, allowing elderly people to obtain information using only their voice and facilitating communication with family and relatives in remote locations.

[0741] Hardware and Software Configuration

[0742] The server has a generative AI engine (e.g., OpenAI GPT-4) and a speech synthesis engine (e.g., Google Text-to-Speech or Amazon Polly). The terminal has a voice input device (e.g., a smart speaker), a speech recognition engine (e.g., Google Cloud Speech-to-Text or Amazon Transcribe), and a network communication module (e.g., Wi-Fi or LTE). The user uses these devices to obtain information by voice and receive voice responses.

[0743] Audio Input and Preprocessing

[0744] The user speaks questions or commands to the smart speaker, such as "What's the weather like today?" or "How do I apply for a pension?" The device uses a voice input device to capture the user's voice, then performs noise reduction and signal normalization to obtain an optimal voice signal.

[0745] Voice recognition and data transmission

[0746] The device uses a speech recognition engine to convert the captured voice into text. For example, it converts the voice "What's the weather today?" into text "What's the weather today?" The device then sends the converted text data to the server via the network communication module.

[0747] Text analysis and response generation

[0748] The server analyzes the received text data and inputs it as a prompt to the generative AI engine. Based on this prompt, the generative AI engine generates an appropriate response according to the user's age group and information literacy. For example, in response to the text "Please tell me how to apply for a pension," it generates the response "To apply for a pension, first obtain the necessary documents from city hall."

[0749] Speech synthesis and data transmission

[0750] The server passes the generated response text to a speech synthesis engine, which converts it into voice data. For example, the text "To apply for a pension, first get the necessary documents at city hall" is converted into voice data. The converted voice data is then sent back to the terminal via the network.

[0751] Playing a voice response

[0752] The device plays the received voice data and communicates the response to the user. For example, a voice may come from the speaker saying, "To apply for a pension, first get the necessary documents from your city hall." The user listens to this and takes the necessary steps or actions.

[0753] Specific examples

[0754] Specifically, imagine a case where a user asks a smart speaker, "What's the weather today?" At this time, the device captures the voice, converts it into text, "What's the weather today?", and sends it to the server. The server inputs the prompt text into a generation AI engine, which generates a response, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." This response is converted into voice data and sent back to the device. Finally, the device plays back this voice data, telling the user, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out."

[0755] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0756] Step 1:

[0757] A user speaks a question or command into a smart speaker, and the input audio signal is captured by the microphone. For example, "What's the weather like today?"

[0758] Step 2:

[0759] The device's audio input device receives the captured audio signal and performs noise reduction and signal normalization. Noise reduction removes background noise from the audio signal, and signal normalization adjusts the amplitude of the audio signal to an appropriate range, thereby making the input audio clearer.

[0760] Step 3:

[0761] The device uses a speech recognition engine to convert the preprocessed voice signal into text data. Specifically, the speech recognition engine analyzes the voice signal and generates text data such as "What's the weather like today?". This text data is output.

[0762] Step 4:

[0763] The terminal sends the converted text data to the server via the network communication module. When sending, meta-information such as the user's age group and information literacy level is also sent along with the text data. The text data and meta-information are sent to the server as output.

[0764] Step 5:

[0765] The server analyzes the received text data and meta information and inputs it as a prompt to the generative AI model. Specifically, the received text "What's the weather like today?" is processed as a prompt for the generative AI model, and an easy-to-understand response is generated that takes into account the user's age group and literacy level. "What's the weather like today?"

[0766] Step 6:

[0767] The generative AI model responds by generating text such as, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." This text data is output on the server.

[0768] Step 7:

[0769] The server passes the generated response text to the speech synthesis engine and converts it into voice data. The speech synthesis engine converts the text "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." into voice data. This voice data is generated.

[0770] Step 8:

[0771] The server then transmits the converted audio data to the terminal via the network, using a network protocol for transmission.

[0772] Step 9:

[0773] The device plays the received voice data and conveys the response to the user. The smart speaker's speaker will say, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." The user listens to this voice response and takes the necessary action.

[0774] Through the above steps, the user can easily obtain information and receive appropriate responses using only voice.

[0775] (Application example 1)

[0776] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0777] The challenge is to enable elderly people to easily obtain necessary information using only their voice, and to be able to quickly request assistance in an emergency. There is also a need to provide an environment where elderly people can easily check the safety of their homes and live with peace of mind.

[0778] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0779] In this invention, the server includes means for accepting voice input, means for converting the input voice into text, means for transmitting the converted text to the server via a network, means for reconstructing the text received by the server in an easy-to-understand manner, means for converting the reconstructed text into voice data, means for receiving and playing the voice data, and means for sending a notification to a contact specified based on the voice input in an emergency. This enables elderly people to obtain information, confirm their safety, and quickly request assistance in an emergency using only their voice.

[0780] "Means for accepting input via voice" refers to devices or technologies that have the function of capturing voices uttered by a user and inputting them into the system.

[0781] "Means for converting input speech into text" refers to technology or devices that analyze speech signals and convert them into corresponding text data.

[0782] "Means for transmitting the converted text to a server via a network" refers to a technology for transmitting text data converted from speech to a server via a network such as the Internet.

[0783] "Means for generating text received by the server in an easy-to-understand format" refers to technology or devices that automatically process and reconstruct text data so that it is easy for users to understand.

[0784] The "means for converting reconstructed text into speech data" refers to a technique or device that converts text data into a speech signal.

[0785] "Means for receiving and playing audio data" refers to a technique for playing audio data received from a server on a device such as a speaker.

[0786] "Means for sending notifications to designated contacts based on voice input in an emergency" refers to technology or devices that automatically send notifications to pre-set contacts when a user inputs a voice command indicating an emergency.

[0787] The present invention provides a home security system that allows elderly people to obtain information using only their voice, check the safety of their home, and use emergency notification functions. The system includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, a voice synthesis engine, and a voice output device.

[0788] System configuration and usage

[0789] 1. Voice Input Device

[0790] A voice input device is a device that receives voice instructions from a user, such as a smart speaker. The device has the ability to perform noise reduction and signal normalization. The user speaks questions or commands into the device.

[0791] 2. Speech Recognition Engine

[0792] The speech captured by the voice input device is converted into text by a speech recognition engine, which can use the speech_recognition package. This engine analyzes the speech and converts it into text data.

[0793] 3. Network Communication Module

[0794] The text generated by the speech recognition engine is sent to the server via the network communication module, which may also include user meta-information such as age and information literacy.

[0795] 4. Generative AI Engine

[0796] The server passes the received text data to a generative AI engine, which generates information in easy-to-understand language according to the user's age group and information literacy. This engine is realized using a generative AI model such as ChatGPT.

[0797] 5. Speech synthesis engine

[0798] The generated response text is converted into voice data by a speech synthesis engine. The gTTS package can be used for this voice synthesis. The server then transmits the generated voice data back to the terminal via the network.

[0799] 6. Audio Output Device

[0800] The terminal plays the received voice data and communicates the response to the user, where information is provided using a speaker.

[0801] 7. Emergency notification function

[0802] In the event of an emergency, the user can enter a designated voice command (e.g., "Help me") and the system will automatically send a notification to pre-registered contacts, enabling a quick response.

[0803] Specific examples

[0804] For example, suppose a user speaks to a smart speaker saying, "Tell me the lock status of the windows." At this time, the voice input device captures the voice, and the voice recognition engine converts it into text, "Tell me the lock status of the windows." This text is then sent to the server via the network. The server passes the text to the generative AI engine, which generates the response, "All windows are locked." This response text is converted into voice data by the speech synthesis engine and sent back to the terminal, allowing the user to hear the response, "All windows are locked," on the speaker.

[0805] Additionally, if the user says "help me," the system will immediately send a notification to designated emergency contacts, allowing seniors to quickly seek assistance in an emergency.

[0806] Prompt Sentence Examples

[0807] "Tell me if the windows are locked."

[0808] "help me."

[0809] This allows the system of the present invention to allow elderly people to obtain information using only their voice, check the safety of their homes, and request assistance in emergencies.

[0810] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0811] Step 1:

[0812] Users speak questions and commands to the smart speaker.

[0813] Input: Example spoken question: "How are the windows locked?"

[0814] Data processing: The microphone built into the smart speaker captures the audio and performs noise reduction and signal normalization.

[0815] Output: Optimized audio signal

[0816] Step 2:

[0817] The device uses a speech recognition engine to convert the captured speech into text.

[0818] Input: Optimized audio signal

[0819] Data calculation: A speech recognition engine (e.g., speech_recognition) analyzes the audio signal and converts it into corresponding text data.

[0820] Output: Text data example: "Please tell me if the windows are locked."

[0821] Step 3:

[0822] The terminal transmits the converted text to the server via the network communication module.

[0823] Input: Text data, user meta information (e.g., age group, information literacy)

[0824] Data processing: Text data and meta information are packaged into packets and sent over the network.

[0825] Output: Receives text data and meta information to the server

[0826] Step 4:

[0827] The server analyzes the received text and meta information and passes it to the generative AI engine.

[0828] Input: Text data, meta information

[0829] Data calculation: A generative AI engine reconstructs text into easy-to-understand expressions according to the user's age group and information literacy. Example: Using a generative AI model

[0830] Output: Reconstructed text data Example: "All windows are locked."

[0831] Step 5:

[0832] The server passes the reconstructed text to a speech synthesis engine, which converts it into speech data.

[0833] Input: Reconstructed text data

[0834] Data processing: A speech synthesis engine (e.g., gTTS) converts text data into speech signals and generates an audio file.

[0835] Output: Example speech data: "All windows are locked."

[0836] Step 6:

[0837] The server transmits the voice data again to the terminal via the network.

[0838] Input: Audio data

[0839] Data processing: Packetizing the audio data and sending it over the network.

[0840] Output: Receiving audio data to the device

[0841] Step 7:

[0842] The terminal plays back the received voice data and conveys the response to the user.

[0843] Input: Audio data

[0844] Specific behavior: Plays audio data through the speaker, telling the user, "All windows are locked."

[0845] Output: Providing a voice response to the user

[0846] Step 8:

[0847] The user can enter further voice commands as needed, including emergency commands such as "help."

[0848] Input: Example user voice command: "Help me"

[0849] Data calculation: The smart speaker captures the voice again and converts it into text using the recognition engine. The server receives the text "Help" and sends an emergency notification to the specified contacts.

[0850] Output: Sending an emergency notification and acknowledging it

[0851] In this way, the system allows seniors to check the safety of their homes, obtain necessary information, and quickly call for help in emergencies using only their voice.

[0852] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0853] The system of the present invention includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, an emotion engine, a voice synthesis engine, and a voice output device, which allows elderly people to obtain information solely through voice and facilitate communication with family and relatives based on emotion analysis.

[0854] Voice input process

[0855] 1. A user speaks questions, commands, and everyday conversations into a smart speaker. For example, they can say, "I'm in a bad mood today, so I don't want to call my friends."

[0856] 2. The microphone on the device (smart speaker) captures the user's voice. At this time, the device performs noise reduction and signal normalization to optimize the voice.

[0857] 3. The device uses an emotion engine to analyze parameters such as tone, tempo, and emphasis from the captured voice to identify the user's emotion. For example, if the voice tone is low and the tempo is slow, it will be determined that the user is in a bad mood.

[0858] Voice recognition and transmission to the server

[0859] 4. The device uses a speech recognition engine to convert the captured speech into corresponding text data, for example, converting the speech "I'm in a bad mood today, so I don't want to call my friends" into text "I'm in a bad mood today, so I don't want to call my friends."

[0860] 5. The device sends the converted text data and user meta-information (such as age group, information literacy, and emotional information) to the server as an HTTP request. The request includes the text content as well as profile information for each user.

[0861] Information generation by generative AI

[0862] 6. The server analyzes the received HTTP request, extracts text data and emotional information, and passes this data to the generative AI engine.

[0863] 7. The generation AI analyzes the text data and generates a response in easy-to-understand language based on the user's profile information and emotional information. For example, if the received text reads, "I'm in a bad mood today, so I don't want to contact my friends," and the emotional information is bad, the generation AI will generate a response like, "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[0864] Audio generation and playback

[0865] 8. The server passes the generated response text to a speech synthesis engine to convert the text into speech data that can be read aloud to the user.

[0866] 9. The server sends the generated audio data to the device as an HTTP response.

[0867] Providing voice responses

[0868] 10. Use the device's function to play back the received voice data and provide a response to the user as voice. For example, tell the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself."

[0869] 11. The user listens to the voice response from the smart speaker and takes the necessary action, for example, taking action to relax.

[0870] Specific examples

[0871] For example, consider the case where a user says, "I'm in a bad mood today, so I don't want to contact my friends." At this time, the device captures the voice, converts it into text, and sends the content "I'm in a bad mood today, so I don't want to contact my friends." to the server. The emotion engine then analyzes the user's emotion as "bad mood." The server passes this text and emotion information to the generation AI engine, which generates a response saying, "You seem to be feeling unwell, but it's important to take it easy and get some rest." The generated response text is then converted into voice data and sent back to the device. Finally, the device plays back this voice data, telling the user, "You seem to be feeling unwell, but it's important to take it easy and get some rest."

[0872] In this way, the elderly can easily obtain the information they need through voice, and receive easy-to-understand responses according to their specific emotional state, which not only improves their quality of life and alleviates their sense of loneliness, but also provides them with appropriate support.

[0873] The processing flow will be explained below.

[0874] Step 1:

[0875] A user speaks a question or command containing emotion into a smart speaker, for example, "I'm in a bad mood today, so I don't want to contact my friends."

[0876] Step 2:

[0877] The microphone on the device (smart speaker) captures the user's voice, and performs noise reduction and signal normalization during capture to optimize the voice data.

[0878] Step 3:

[0879] The device uses a speech recognition engine to convert the captured speech into corresponding text, for example, converting the speech "I'm in a bad mood today, so I don't want to call my friends" into the text "I'm in a bad mood today, so I don't want to call my friends."

[0880] Step 4:

[0881] The device uses an emotion engine to analyze parameters such as tone, tempo, and emphasis from the voice data to identify the user's emotion. For example, if the voice tone is low and the tempo is slow, the user's emotion is identified as "unhappy."

[0882] Step 5:

[0883] The device sends the converted text data, analyzed emotional information, and user meta-information to the server as an HTTP request. The request includes the text content, the user's age group, information literacy, and emotional information.

[0884] Step 6:

[0885] The server analyzes the received HTTP request and extracts text data and emotional information.

[0886] Step 7:

[0887] The server passes the extracted text data and emotional information to the generative AI engine.

[0888] Step 8:

[0889] The generation AI analyzes the text data and generates a response in easy-to-understand and appropriate language based on the user's profile and emotional information. For example, if the received text is "I'm in a bad mood today, so I don't want to contact my friends," and the emotional information is "bad mood," the generation AI will generate a response like, "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[0890] Step 9:

[0891] The server passes the generated response text to a speech synthesis engine and converts it into voice data.

[0892] Step 10:

[0893] The server sends the generated audio data to the terminal as an HTTP response.

[0894] Step 11:

[0895] The device uses its function to play back the received voice data and provides a response to the user as voice, for example, telling the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself."

[0896] Step 12:

[0897] The user hears a voice response from the smart speaker and takes necessary action based on it, for example, taking action to relax.

[0898] Example 2

[0899] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0900] Currently, there are voice recognition systems that allow elderly people to obtain information and easily communicate, but they lack the ability to accurately analyze emotional states and provide optimal responses based on those states. Furthermore, they are unable to generate responses that correspond to the elderly person's own profile information (age group and information literacy), resulting in a poor user experience. Furthermore, voice recognition accuracy decreases in noisy environments, resulting in situations where satisfactory performance cannot be achieved.

[0901] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0902] In this invention, the server includes means for removing noise from the input speech and normalizing the signal, means for analyzing emotions from the input speech, and means for converting the input speech to text. This makes it possible to accurately analyze the emotional state of an elderly person from their input speech and generate an optimal response based on that state. Furthermore, noise removal and signal normalization enable highly accurate speech recognition even in a variety of environments, improving the user experience.

[0903] "Means for accepting input by voice" refers to a function for capturing voice from a user using a voice input device.

[0904] The "means for removing noise from input audio and normalizing the signal" refers to a function for removing unnecessary noise from captured audio data and performing processing to obtain a stable audio signal.

[0905] "Means for analyzing emotions from input voice" refers to a function that analyzes parameters such as tone, tempo, and emphasis contained in voice data to identify the user's emotional state.

[0906] "Means for converting input speech to text" refers to a function that uses a speech recognition engine to convert captured speech data into corresponding text data.

[0907] "Means for transmitting the converted text and user profile information to a server via a network" refers to a function for transmitting data including the converted text data and user meta information to a server via a network.

[0908] "Means for analyzing text and emotional information received by the server" refers to a function for analyzing text data and emotional information received by the server and extracting necessary information.

[0909] "Means for generating a response using generative AI based on analyzed information" refers to the function of using a generative AI engine to create an optimal response based on the generated text data and emotional information.

[0910] The "means for converting the generated text into voice data" refers to a function for converting the generated response text into voice data using a voice synthesis engine and conveying it to the user.

[0911] "Means for receiving and playing back audio data" refers to the function of receiving audio data sent from the server and playing back that audio data.

[0912] The system of the present invention includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, an emotion engine, a voice synthesis engine, and a voice output device. The following describes the details of each component and how they work together.

[0913] First, the user speaks into a smart speaker. A smart speaker is a piece of hardware used for voice capture, collecting the user's voice using a built-in microphone. The system acts as a voice input device to accept the user's voice. This voice input device includes a noise reduction and signal normalization function, which normalizes the signal while removing ambient noise.

[0914] Next, while the device is capturing the audio, it uses an emotion engine to analyze the audio for emotional information. The emotion engine determines the user's emotional state based on parameters such as voice tone, tempo, and emphasis. This analysis can determine whether the user is feeling sad, happy, angry, etc.

[0915] After the voice is captured and the emotion is analyzed, the device uses a speech recognition engine to convert the voice into text data. For example, "I'm in a bad mood today, so I don't want to contact my friends." The voice is converted into text "I'm in a bad mood today, so I don't want to contact my friends." This text data and the user's profile information (such as age group, information literacy, and emotional information) are sent to the server as an HTTP request.

[0916] The server analyzes the received text data and emotional information and passes the necessary information to the generative AI engine. The generative AI engine generates an appropriate response based on the user's profile information and emotional information. For example, based on the text "I'm in a bad mood today, so I don't want to contact my friends," and the emotional information "bad mood," the response generated is "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[0917] The generated response text is again passed to the speech synthesis engine by the server and converted into voice data, which is finally sent to the terminal as an HTTP response.

[0918] The device plays back the received voice data and provides a response to the user as voice. For example, by telling the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself," the user can take appropriate action.

[0919] Specific examples

[0920] For example, if a user says, "I'm in a bad mood today, so I don't want to contact my friends," the system works as follows:

[0921] 1. The device captures the audio and performs noise reduction and signal normalization.

[0922] 2. The device analyzes the emotional information of "bad mood" using an emotion engine.

[0923] 3. The device converts the speech into text data such as "I'm in a bad mood today, so I don't want to contact my friends," and sends this text data and profile information to the server.

[0924] 4. The server analyzes the received data, and the AI ​​engine generates a response saying, "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[0925] 5. The server converts the response text into voice data and sends it to the device.

[0926] 6. The device plays audio data and tells the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself."

[0927] This process allows seniors to easily obtain the information they need through voice and provides easy-to-understand responses tailored to their specific emotional states, thereby improving their quality of life, reducing loneliness, and providing appropriate support.

[0928] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0929] Step 1:

[0930] A user speaks to a smart speaker, for example, "I'm in a bad mood today, so I don't want to contact my friends." This speech is the input for the entire system, and it is the starting point for processing.

[0931] Step 2:

[0932] The smart speaker of the device captures the user's voice using the smart speaker's microphone. The captured voice data is accompanied by noise data, so noise removal and signal normalization are required in the next processing steps.

[0933] Step 3:

[0934] The device performs noise reduction and signal normalization on the audio data. At this stage, background noise is removed from the captured audio data and the amplitude of the audio signal is adjusted to obtain clear audio data. This clear audio data is the input for the next step.

[0935] Step 4:

[0936] The device uses an emotion engine to analyze emotional information from the normalized voice data. The emotion engine analyzes the tone, tempo, and emphasis of the voice to determine the emotion the user is feeling when speaking. In this case, the analyzed emotion is displeased. The output is the voice data before it is converted into text and the emotional information (displeased).

[0937] Step 5:

[0938] The device uses a speech recognition engine to convert the normalized voice data into text data. This conversion process generates the text, "I'm in a bad mood today, so I don't want to contact my friends." The output is this text data.

[0939] Step 6:

[0940] The device sends an HTTP request to the server, including the converted text data and the user's profile information (such as age group, information literacy, and emotional information). The server receives the text data and the user profile information.

[0941] Step 7:

[0942] The server analyzes the received HTTP request and extracts text data and emotional information. In this example, the text data "I'm in a bad mood today, so I don't want to contact my friends." and emotional information (bad mood) are extracted within the server. The output is the analyzed text data and emotional information.

[0943] Step 8:

[0944] The server passes the extracted text data and emotional information to the generative AI engine. The generative AI engine generates the optimal response based on the user's profile information and emotional information. In this case, the response generated is, "You seem to be feeling unwell, but it's important to take it easy and get some rest." The output is the generated response text.

[0945] Step 9:

[0946] The server passes the generated response text to the speech synthesis engine, which converts the text into speech data. The speech synthesis engine generates speech data saying, "You seem to be feeling unwell, but it's important to take a rest and not push yourself." The output is speech data.

[0947] Step 10:

[0948] The server sends the generated audio data to the terminal as an HTTP response, and the terminal receives this audio data.

[0949] Step 11:

[0950] The device plays the received voice data and provides a response to the user. For example, a smart speaker might say to the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself." The user can listen to this voice response and take appropriate action.

[0951] (Application example 2)

[0952] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0953] When elderly people order food delivery services by voice, there is a need for a system that can provide appropriate menu suggestions and allow them to proceed with the ordering process comfortably and without stress. In particular, it is a challenge to increase user satisfaction and improve the quality of life by making suggestions that correspond to the emotional state of the elderly.

[0954] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0955] In this invention, the server includes means for accepting input by voice, means for converting the input voice into text, means for transmitting the converted text to the server via a network, means for analyzing the text and emotional information received by the server and generating easy-to-understand expressions and suggestions according to the user's emotional state, means for converting the reconstructed text and suggestions into voice data, and means for receiving and playing the voice data. This enables elderly people to receive appropriate meal suggestions according to their emotional state when ordering food delivery by voice.

[0956] "Means for accepting voice input" refers to a device that detects voices emitted by the user and inputs them into the system, such as a microphone or the voice input function of a smartphone.

[0957] The "means for converting input voice into text" refers to a voice recognition engine that converts voice data into character data, and has the function of converting voice into text using voice analysis technology.

[0958] The "means for transmitting the converted text to a server via a network" is a communication module for transmitting the converted text data to a server via the Internet or another network.

[0959] "Generation means that analyzes the text and emotional information received by the server and generates easy-to-understand expressions and suggestions according to the user's emotional state" refers to a generative AI engine or emotion analysis engine that generates appropriate responses and suggestions based on the received text and the user's emotional information.

[0960] The "means for converting the reconstructed text and suggestions into speech data" refers to a speech synthesis engine that converts the generated text and suggestions into speech data, and has the function of converting the generated information into easy-to-listen speech.

[0961] "Means for receiving and playing audio data" refers to a device that receives the converted audio data and plays it in a form that the user can hear, such as a speaker or the audio output function of a smartphone.

[0962] This invention realizes a system that allows elderly people to easily order food delivery by voice by using a voice input device, a voice recognition engine, a network communication module, a generative AI engine, an emotion engine, a voice synthesis engine, and a voice output device.

[0963] When using a smartphone, the user expresses their order preference using the smartphone's voice input function. For example, they might say, "I'm tired, please give me something to eat that will energize me." This voice is captured by the smartphone's microphone. The captured voice data is converted into text data by a speech recognition engine. Specifically, the voice is converted into text using the Google Cloud Speech-to-Text API.

[0964] The converted text is then sent to the server via the network. This network communication module uses HTTP. The server receives the received text and the user's emotional information, which is analyzed by the emotion engine. This emotion analysis uses IBM Watson Tone Analyzer.

[0965] The server then uses a generative AI engine (such as the OpenAI GPT-4 API) to generate easy-to-understand expressions and suggestions based on the emotional state of the user based on the received text and emotional information. For example, if the user is tired, a suggestion such as "Would you like some nutritious chicken soup and fresh fruit?" is generated. This generated text is then converted into speech data using a speech synthesis engine. Amazon Polly is used for this speech synthesis engine.

[0966] The converted voice data is then sent back to the smartphone via the network and played back through the smartphone's speaker, allowing the user to listen to the menu of voice suggestions and make a selection from them.

[0967] As a specific example, if a user says, "I'm tired, please give me something to eat that will give me energy," the following prompt sentence will be sent to the generation AI.

[0968] Example prompt sentence:

[0969] User said: I'm tired, please give me something to eat to cheer me up. Tone: Sad. Suggest a meal.

[0970] Here's an example of a response suggested by the generative AI:

[0971] How about a nutritious chicken soup with vegetables and a side of fresh fruit to boost your energy?

[0972] In this way, a system has been realized that allows elderly people to order food delivery by voice without stress, and suggestions based on emotion analysis improve satisfaction. This also improves the quality of life for elderly people and makes their daily lives more convenient.

[0973] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0974] Step 1:

[0975] The user uses the smartphone's voice input function to verbally state their order. At this time, the smartphone's microphone captures the user's voice and stores it as voice data. The input is the user's voice, and the output is the captured voice data.

[0976] Step 2:

[0977] The device converts the captured voice data into text data using the Google Cloud Speech-to-Text API. The input is the voice data, and the output is the converted text data. The speech recognition engine analyzes the voice and generates the corresponding text.

[0978] Step 3:

[0979] The terminal sends the converted text data to the server via the network communication module. The input is text data, and the output is text data as an HTTP request. The network communication module is responsible for sending.

[0980] Step 4:

[0981] The server uses IBM Watson Tone Analyzer to analyze the received text data and the user's emotional information. The input is the received text data and the emotional parameters extracted from the user's voice, and the output is the analyzed emotional information. The emotional analysis engine identifies the emotional state.

[0982] Step 5:

[0983] The server passes the analyzed emotional information and text data to the OpenAI GPT-4 API, and the generative AI engine generates appropriate suggestions based on this information. The input is text data and emotional information, and the output is the generated suggested text. The generative AI engine generates the suggestions.

[0984] Step 6:

[0985] The server converts the generated suggested text into speech data using Amazon Polly. The input is the suggested text and the output is the generated speech data. The speech synthesis engine converts the text into speech.

[0986] Step 7:

[0987] The server sends the generated voice data to the terminal via the network. The input is the voice data, and the output is the voice data as an HTTP response. The network communication module is responsible for sending the voice data.

[0988] Step 8:

[0989] The device plays the received voice data and conveys the suggestion to the user. The input is the received voice data, and the output is a voice response to the user. The smartphone speaker performs the specific operation of playing the voice data.

[0990] Step 9:

[0991] The user listens to the voice response played from the terminal, selects a suggested menu from the list, and proceeds with the ordering process. For example, the user confirms the suggested menu and then issues another voice command such as "I would like to order this menu item," thereby completing the order. The input is the voice response from the terminal, and the output is the execution of the order based on the user's selection.

[0992] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0993] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0994] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0995] [Fourth embodiment]

[0996] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0997] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0998] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0999] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1000] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1001] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1002] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1003] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1004] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1005] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1006] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1007] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1008] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1009] The system of the present invention includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, a voice synthesis engine, and a voice output device, which allows elderly people to obtain information using only their voice and facilitate communication with family and relatives in remote locations.

[1010] Voice input process

[1011] 1. A user speaks a question or command to a smart speaker, such as "What's the weather like today?" or "Tell me how to apply for a pension."

[1012] 2. The microphone on the device (smart speaker) captures the user's voice. At this time, the device performs noise reduction and signal normalization to optimize the voice.

[1013] Voice recognition and transmission to the server

[1014] 3. The device uses a speech recognition engine to convert the captured speech into corresponding text, for example, converting the speech "What's the weather today?" into the text "What's the weather today?"

[1015] 4. The device sends this text data to the server via the network, along with meta-information such as the user's age group and information literacy.

[1016] Information generation by generative AI

[1017] 5. The server analyzes the received text data and passes it to the generative AI engine, which generates information in easy-to-understand language according to the user's age and literacy level.

[1018] 6. For example, if the received text is "Please tell me how to apply for a pension," the generation AI will generate the response "To apply for a pension, first get the necessary documents from city hall."

[1019] Audio generation and playback

[1020] 7. The server passes the generated response text to the speech synthesis engine and converts it into voice data. The speech synthesis engine converts the text "To apply for a pension, first get the necessary documents from city hall" into voice data.

[1021] 8. The server sends the audio data back to the terminal via the network.

[1022] Providing voice responses

[1023] 9. The device plays back the received voice data and communicates the response to the user. For example, a voice may be heard from the speaker saying, "To apply for a pension, first obtain the necessary documents from city hall."

[1024] 10. The user listens to this and takes the necessary steps or actions.

[1025] Specific examples

[1026] For example, consider the case where a user speaks to a smart speaker saying, "Tell me what the weather is today." At this time, the device captures the voice, converts it into text, and sends the message "What's the weather today?" to the server. The server passes this text to a generative AI engine, which generates a response saying, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." The generated response text is then converted into voice data and sent back to the device. Finally, the device plays back this voice data, telling the user, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out."

[1027] In this way, the elderly can easily obtain the necessary information by voice, and can also receive easy-to-understand responses according to specific conditions, thereby improving the quality of life for the elderly.

[1028] The processing flow will be explained below.

[1029] Step 1:

[1030] The user speaks a question or command to the smart speaker, for example, "What's the weather like today?" or "How do I apply for a pension?"

[1031] Step 2:

[1032] The microphone on the device (smart speaker) captures the user's voice, which includes collecting and pre-processing the voice signal (noise removal and signal normalization).

[1033] Step 3:

[1034] The device uses a speech recognition engine to convert the captured speech into text data, for example converting the speech "What's the weather today?" into text "What's the weather today?"

[1035] Step 4:

[1036] The device sends the converted text data and user meta-information (such as age group and information literacy) to the server as an HTTP request. The request includes the text content as well as profile information for each user.

[1037] Step 5:

[1038] The server analyzes the received HTTP request and extracts text data, which it then passes to the AI ​​generation engine.

[1039] Step 6:

[1040] The generative AI analyzes the text data and generates a response in easy-to-understand language based on the user's profile information. For example, in response to the text "Please tell me how to apply for a pension," it creates the response "To apply for a pension, first obtain the necessary documents from city hall."

[1041] Step 7:

[1042] The server passes the generated response text to a speech synthesis engine, which converts the text into speech data that can be read aloud to the user.

[1043] Step 8:

[1044] The server sends the generated audio data to the terminal as an HTTP response.

[1045] Step 9:

[1046] The device uses its function to play back the received voice data and provides a response to the user as voice. For example, it tells the user, "To apply for a pension, first get the necessary documents from your city hall."

[1047] Step 10:

[1048] The user hears a voice response from the smart speaker and takes the necessary action, for example, preparing to go to city hall.

[1049] The above processing flow allows users, especially elderly people, to easily obtain information using voice and to smoothly communicate with family and relatives.

[1050] Example 1

[1051] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1052] Elderly people and users with low information literacy need a system that makes it easier to obtain information and allows smooth communication with family and relatives in remote locations using only voice. However, previous voice recognition systems have had difficulty providing appropriate information according to the user's age and information literacy, and have also had problems with noise removal and signal normalization.

[1053] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1054] In this invention, the server includes means for accepting voice input, means for converting the input voice into text, means for transmitting the converted voice text to the server via a network, means for the server to analyze the received text and input it into the generative AI model, means for converting the response generated by the generative AI model into voice data, and means for receiving and playing the voice data. This enables the provision of easy-to-understand information according to the user's age group and information literacy, as well as noise removal and signal normalization during voice input.

[1055] "Means for receiving input by voice" refers to a device or component that has the function of capturing voice uttered by a user and inputting it into the system.

[1056] "Means for converting input voice into text" refers to a device or software that has the function of appropriately analyzing the captured voice signal and converting its content into text data.

[1057] "Means for transmitting speech-converted text to a server via a network" refers to devices or software that have the function of transmitting text data converted from speech to a server via a network such as the Internet.

[1058] "Means for analyzing text received by the server and inputting it into the generative AI model" refers to devices or software that have the function of analyzing text data received via a network and inputting appropriate prompt sentences into the generative AI model based on the analysis results.

[1059] "Means for converting responses generated by a generative AI model into voice data" refers to devices or software that have the function of converting text-format responses generated by a generative AI model into voice data.

[1060] "Means for receiving and reproducing audio data" refers to a device or component that has the function of reproducing audio data received via a network and providing information to the user by audio.

[1061] "Noise reduction" refers to technology that removes unnecessary background sounds and noises that are included in voice input, and clearly captures only the voice spoken by the user.

[1062] "Signal normalization" refers to a technique for adjusting the amplitude and volume of an audio signal to a certain range in order to improve the accuracy of voice recognition.

[1063] "Automatically adjusting responses according to the user's age group and information literacy" refers to the ability of the generative AI model to generate responses with optimal expressions and content that are tailored to the user's characteristics.

[1064] MODE FOR CARRYING OUT THE INVENTION

[1065] The system of the present invention combines a voice input device, a voice recognition engine, a network communication module, a generative AI engine, a voice synthesis engine, and a voice output device, allowing elderly people to obtain information using only their voice and facilitating communication with family and relatives in remote locations.

[1066] Hardware and Software Configuration

[1067] The server has a generative AI engine (e.g., OpenAI GPT-4) and a speech synthesis engine (e.g., Google Text-to-Speech or Amazon Polly). The terminal has a voice input device (e.g., a smart speaker), a speech recognition engine (e.g., Google Cloud Speech-to-Text or Amazon Transcribe), and a network communication module (e.g., Wi-Fi or LTE). The user uses these devices to obtain information by voice and receive voice responses.

[1068] Audio Input and Preprocessing

[1069] The user speaks questions or commands to the smart speaker, such as "What's the weather like today?" or "How do I apply for a pension?" The device uses a voice input device to capture the user's voice, then performs noise reduction and signal normalization to obtain an optimal voice signal.

[1070] Voice recognition and data transmission

[1071] The device uses a speech recognition engine to convert the captured voice into text. For example, it converts the voice "What's the weather today?" into text "What's the weather today?" The device then sends the converted text data to the server via the network communication module.

[1072] Text analysis and response generation

[1073] The server analyzes the received text data and inputs it as a prompt to the generative AI engine. Based on this prompt, the generative AI engine generates an appropriate response according to the user's age group and information literacy. For example, in response to the text "Please tell me how to apply for a pension," it generates the response "To apply for a pension, first obtain the necessary documents from city hall."

[1074] Speech synthesis and data transmission

[1075] The server passes the generated response text to a speech synthesis engine, which converts it into voice data. For example, the text "To apply for a pension, first get the necessary documents at city hall" is converted into voice data. The converted voice data is then sent back to the terminal via the network.

[1076] Playing a voice response

[1077] The device plays the received voice data and communicates the response to the user. For example, a voice may come from the speaker saying, "To apply for a pension, first get the necessary documents from your city hall." The user listens to this and takes the necessary steps or actions.

[1078] Specific examples

[1079] Specifically, imagine a case where a user asks a smart speaker, "What's the weather today?" At this time, the device captures the voice, converts it into text, "What's the weather today?", and sends it to the server. The server inputs the prompt text into a generation AI engine, which generates a response, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." This response is converted into voice data and sent back to the device. Finally, the device plays back this voice data, telling the user, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out."

[1080] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1081] Step 1:

[1082] A user speaks a question or command into a smart speaker, and the input audio signal is captured by the microphone. For example, "What's the weather like today?"

[1083] Step 2:

[1084] The device's audio input device receives the captured audio signal and performs noise reduction and signal normalization. Noise reduction removes background noise from the audio signal, and signal normalization adjusts the amplitude of the audio signal to an appropriate range, thereby making the input audio clearer.

[1085] Step 3:

[1086] The device uses a speech recognition engine to convert the preprocessed voice signal into text data. Specifically, the speech recognition engine analyzes the voice signal and generates text data such as "What's the weather like today?". This text data is output.

[1087] Step 4:

[1088] The terminal sends the converted text data to the server via the network communication module. When sending, meta-information such as the user's age group and information literacy level is also sent along with the text data. The text data and meta-information are sent to the server as output.

[1089] Step 5:

[1090] The server analyzes the received text data and meta information and inputs it as a prompt to the generative AI model. Specifically, the received text "What's the weather like today?" is processed as a prompt for the generative AI model, and an easy-to-understand response is generated that takes into account the user's age group and literacy level. "What's the weather like today?"

[1091] Step 6:

[1092] The generative AI model responds by generating text such as, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." This text data is output on the server.

[1093] Step 7:

[1094] The server passes the generated response text to the speech synthesis engine and converts it into voice data. The speech synthesis engine converts the text "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." into voice data. This voice data is generated.

[1095] Step 8:

[1096] The server then transmits the converted audio data to the terminal via the network, using a network protocol for transmission.

[1097] Step 9:

[1098] The device plays the received voice data and conveys the response to the user. The smart speaker's speaker will say, "Today's weather is sunny. The temperature is 25 degrees. It's a good day to go out." The user listens to this voice response and takes the necessary action.

[1099] Through the above steps, the user can easily obtain information and receive appropriate responses using only voice.

[1100] (Application example 1)

[1101] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1102] The challenge is to enable elderly people to easily obtain necessary information using only their voice, and to be able to quickly request assistance in an emergency. There is also a need to provide an environment where elderly people can easily check the safety of their homes and live with peace of mind.

[1103] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1104] In this invention, the server includes means for accepting voice input, means for converting the input voice into text, means for transmitting the converted text to the server via a network, means for reconstructing the text received by the server in an easy-to-understand manner, means for converting the reconstructed text into voice data, means for receiving and playing the voice data, and means for sending a notification to a contact specified based on the voice input in an emergency. This enables elderly people to obtain information, confirm their safety, and quickly request assistance in an emergency using only their voice.

[1105] "Means for accepting input via voice" refers to devices or technologies that have the function of capturing voices uttered by a user and inputting them into the system.

[1106] "Means for converting input speech into text" refers to technology or devices that analyze speech signals and convert them into corresponding text data.

[1107] "Means for transmitting the converted text to a server via a network" refers to a technology for transmitting text data converted from speech to a server via a network such as the Internet.

[1108] "Means for generating text received by the server in an easy-to-understand format" refers to technology or devices that automatically process and reconstruct text data so that it is easy for users to understand.

[1109] The "means for converting reconstructed text into speech data" refers to a technique or device that converts text data into a speech signal.

[1110] "Means for receiving and playing audio data" refers to a technique for playing audio data received from a server on a device such as a speaker.

[1111] "Means for sending notifications to designated contacts based on voice input in an emergency" refers to technology or devices that automatically send notifications to pre-set contacts when a user inputs a voice command indicating an emergency.

[1112] The present invention provides a home security system that allows elderly people to obtain information using only their voice, check the safety of their home, and use emergency notification functions. The system includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, a voice synthesis engine, and a voice output device.

[1113] System configuration and usage

[1114] 1. Voice Input Device

[1115] A voice input device is a device that receives voice instructions from a user, such as a smart speaker. The device has the ability to perform noise reduction and signal normalization. The user speaks questions or commands into the device.

[1116] 2. Speech Recognition Engine

[1117] The speech captured by the voice input device is converted into text by a speech recognition engine, which can use the speech_recognition package. This engine analyzes the speech and converts it into text data.

[1118] 3. Network Communication Module

[1119] The text generated by the speech recognition engine is sent to the server via the network communication module, which may also include user meta-information such as age and information literacy.

[1120] 4. Generative AI Engine

[1121] The server passes the received text data to a generative AI engine, which generates information in easy-to-understand language according to the user's age group and information literacy. This engine is realized using a generative AI model such as ChatGPT.

[1122] 5. Speech synthesis engine

[1123] The generated response text is converted into voice data by a speech synthesis engine. The gTTS package can be used for this voice synthesis. The server then transmits the generated voice data back to the terminal via the network.

[1124] 6. Audio Output Device

[1125] The terminal plays the received voice data and communicates the response to the user, where information is provided using a speaker.

[1126] 7. Emergency notification function

[1127] In the event of an emergency, the user can enter a designated voice command (e.g., "Help me") and the system will automatically send a notification to pre-registered contacts, enabling a quick response.

[1128] Specific examples

[1129] For example, suppose a user speaks to a smart speaker saying, "Tell me the lock status of the windows." At this time, the voice input device captures the voice, and the voice recognition engine converts it into text, "Tell me the lock status of the windows." This text is then sent to the server via the network. The server passes the text to the generative AI engine, which generates the response, "All windows are locked." This response text is converted into voice data by the speech synthesis engine and sent back to the terminal, allowing the user to hear the response, "All windows are locked," on the speaker.

[1130] Additionally, if the user says "help me," the system will immediately send a notification to designated emergency contacts, allowing seniors to quickly seek assistance in an emergency.

[1131] Prompt Sentence Examples

[1132] "Tell me if the windows are locked."

[1133] "help me."

[1134] This allows the system of the present invention to allow elderly people to obtain information using only their voice, check the safety of their homes, and request assistance in emergencies.

[1135] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1136] Step 1:

[1137] Users speak questions and commands to the smart speaker.

[1138] Input: Example spoken question: "How are the windows locked?"

[1139] Data processing: The microphone built into the smart speaker captures the audio and performs noise reduction and signal normalization.

[1140] Output: Optimized audio signal

[1141] Step 2:

[1142] The device uses a speech recognition engine to convert the captured speech into text.

[1143] Input: Optimized audio signal

[1144] Data calculation: A speech recognition engine (e.g., speech_recognition) analyzes the audio signal and converts it into corresponding text data.

[1145] Output: Text data example: "Please tell me if the windows are locked."

[1146] Step 3:

[1147] The terminal transmits the converted text to the server via the network communication module.

[1148] Input: Text data, user meta information (e.g., age group, information literacy)

[1149] Data processing: Text data and meta information are packaged into packets and sent over the network.

[1150] Output: Receives text data and meta information to the server

[1151] Step 4:

[1152] The server analyzes the received text and meta information and passes it to the generative AI engine.

[1153] Input: Text data, meta information

[1154] Data calculation: A generative AI engine reconstructs text into easy-to-understand expressions according to the user's age group and information literacy. Example: Using a generative AI model

[1155] Output: Reconstructed text data Example: "All windows are locked."

[1156] Step 5:

[1157] The server passes the reconstructed text to a speech synthesis engine, which converts it into speech data.

[1158] Input: Reconstructed text data

[1159] Data processing: A speech synthesis engine (e.g., gTTS) converts text data into speech signals and generates an audio file.

[1160] Output: Example speech data: "All windows are locked."

[1161] Step 6:

[1162] The server transmits the voice data again to the terminal via the network.

[1163] Input: Audio data

[1164] Data processing: Packetizing the audio data and sending it over the network.

[1165] Output: Receiving audio data to the device

[1166] Step 7:

[1167] The terminal plays back the received voice data and conveys the response to the user.

[1168] Input: Audio data

[1169] Specific behavior: Plays audio data through the speaker, telling the user, "All windows are locked."

[1170] Output: Providing a voice response to the user

[1171] Step 8:

[1172] The user can enter further voice commands as needed, including emergency commands such as "help."

[1173] Input: Example user voice command: "Help me"

[1174] Data calculation: The smart speaker captures the voice again and converts it into text using the recognition engine. The server receives the text "Help" and sends an emergency notification to the specified contacts.

[1175] Output: Sending an emergency notification and acknowledging it

[1176] In this way, the system allows seniors to check the safety of their homes, obtain necessary information, and quickly call for help in emergencies using only their voice.

[1177] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1178] The system of the present invention includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, an emotion engine, a voice synthesis engine, and a voice output device, which allows elderly people to obtain information solely through voice and facilitate communication with family and relatives based on emotion analysis.

[1179] Voice input process

[1180] 1. A user speaks questions, commands, and everyday conversations into a smart speaker. For example, they can say, "I'm in a bad mood today, so I don't want to call my friends."

[1181] 2. The microphone on the device (smart speaker) captures the user's voice. At this time, the device performs noise reduction and signal normalization to optimize the voice.

[1182] 3. The device uses an emotion engine to analyze parameters such as tone, tempo, and emphasis from the captured voice to identify the user's emotion. For example, if the voice tone is low and the tempo is slow, it will be determined that the user is in a bad mood.

[1183] Voice recognition and transmission to the server

[1184] 4. The device uses a speech recognition engine to convert the captured speech into corresponding text data, for example, converting the speech "I'm in a bad mood today, so I don't want to call my friends" into text "I'm in a bad mood today, so I don't want to call my friends."

[1185] 5. The device sends the converted text data and user meta-information (such as age group, information literacy, and emotional information) to the server as an HTTP request. The request includes the text content as well as profile information for each user.

[1186] Information generation by generative AI

[1187] 6. The server analyzes the received HTTP request, extracts text data and emotional information, and passes this data to the generative AI engine.

[1188] 7. The generation AI analyzes the text data and generates a response in easy-to-understand language based on the user's profile information and emotional information. For example, if the received text reads, "I'm in a bad mood today, so I don't want to contact my friends," and the emotional information is bad, the generation AI will generate a response like, "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[1189] Audio generation and playback

[1190] 8. The server passes the generated response text to a speech synthesis engine to convert the text into speech data that can be read aloud to the user.

[1191] 9. The server sends the generated audio data to the device as an HTTP response.

[1192] Providing voice responses

[1193] 10. Use the device's function to play back the received voice data and provide a response to the user as voice. For example, tell the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself."

[1194] 11. The user listens to the voice response from the smart speaker and takes the necessary action, for example, taking action to relax.

[1195] Specific examples

[1196] For example, consider the case where a user says, "I'm in a bad mood today, so I don't want to contact my friends." At this time, the device captures the voice, converts it into text, and sends the content "I'm in a bad mood today, so I don't want to contact my friends." to the server. The emotion engine then analyzes the user's emotion as "bad mood." The server passes this text and emotion information to the generation AI engine, which generates a response saying, "You seem to be feeling unwell, but it's important to take it easy and get some rest." The generated response text is then converted into voice data and sent back to the device. Finally, the device plays back this voice data, telling the user, "You seem to be feeling unwell, but it's important to take it easy and get some rest."

[1197] In this way, the elderly can easily obtain the information they need through voice, and receive easy-to-understand responses according to their specific emotional state, which not only improves their quality of life and alleviates their sense of loneliness, but also provides them with appropriate support.

[1198] The processing flow will be explained below.

[1199] Step 1:

[1200] A user speaks a question or command containing emotion into a smart speaker, for example, "I'm in a bad mood today, so I don't want to contact my friends."

[1201] Step 2:

[1202] The microphone on the device (smart speaker) captures the user's voice, and performs noise reduction and signal normalization during capture to optimize the voice data.

[1203] Step 3:

[1204] The device uses a speech recognition engine to convert the captured speech into corresponding text, for example, converting the speech "I'm in a bad mood today, so I don't want to call my friends" into the text "I'm in a bad mood today, so I don't want to call my friends."

[1205] Step 4:

[1206] The device uses an emotion engine to analyze parameters such as tone, tempo, and emphasis from the voice data to identify the user's emotion. For example, if the voice tone is low and the tempo is slow, the user's emotion is identified as "unhappy."

[1207] Step 5:

[1208] The device sends the converted text data, analyzed emotional information, and user meta-information to the server as an HTTP request. The request includes the text content, the user's age group, information literacy, and emotional information.

[1209] Step 6:

[1210] The server analyzes the received HTTP request and extracts text data and emotional information.

[1211] Step 7:

[1212] The server passes the extracted text data and emotional information to the generative AI engine.

[1213] Step 8:

[1214] The generation AI analyzes the text data and generates a response in easy-to-understand and appropriate language based on the user's profile and emotional information. For example, if the received text is "I'm in a bad mood today, so I don't want to contact my friends," and the emotional information is "bad mood," the generation AI will generate a response like, "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[1215] Step 9:

[1216] The server passes the generated response text to a speech synthesis engine and converts it into voice data.

[1217] Step 10:

[1218] The server sends the generated audio data to the terminal as an HTTP response.

[1219] Step 11:

[1220] The device uses its function to play back the received voice data and provides a response to the user as voice, for example, telling the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself."

[1221] Step 12:

[1222] The user hears a voice response from the smart speaker and takes necessary action based on it, for example, taking action to relax.

[1223] Example 2

[1224] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1225] Currently, there are voice recognition systems that allow elderly people to obtain information and easily communicate, but they lack the ability to accurately analyze emotional states and provide optimal responses based on those states. Furthermore, they are unable to generate responses that correspond to the elderly person's own profile information (age group and information literacy), resulting in a poor user experience. Furthermore, voice recognition accuracy decreases in noisy environments, resulting in situations where satisfactory performance cannot be achieved.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1227] In this invention, the server includes means for removing noise from the input speech and normalizing the signal, means for analyzing emotions from the input speech, and means for converting the input speech to text. This makes it possible to accurately analyze the emotional state of an elderly person from their input speech and generate an optimal response based on that state. Furthermore, noise removal and signal normalization enable highly accurate speech recognition even in a variety of environments, improving the user experience.

[1228] "Means for accepting input by voice" refers to a function for capturing voice from a user using a voice input device.

[1229] The "means for removing noise from input audio and normalizing the signal" refers to a function for removing unnecessary noise from captured audio data and performing processing to obtain a stable audio signal.

[1230] "Means for analyzing emotions from input voice" refers to a function that analyzes parameters such as tone, tempo, and emphasis contained in voice data to identify the user's emotional state.

[1231] "Means for converting input speech to text" refers to a function that uses a speech recognition engine to convert captured speech data into corresponding text data.

[1232] "Means for transmitting the converted text and user profile information to a server via a network" refers to a function for transmitting data including the converted text data and user meta information to a server via a network.

[1233] "Means for analyzing text and emotional information received by the server" refers to a function for analyzing text data and emotional information received by the server and extracting necessary information.

[1234] "Means for generating a response using generative AI based on analyzed information" refers to the function of using a generative AI engine to create an optimal response based on the generated text data and emotional information.

[1235] The "means for converting the generated text into voice data" refers to a function for converting the generated response text into voice data using a voice synthesis engine and conveying it to the user.

[1236] "Means for receiving and playing back audio data" refers to the function of receiving audio data sent from the server and playing back that audio data.

[1237] The system of the present invention includes a voice input device, a voice recognition engine, a network communication module, a generative AI engine, an emotion engine, a voice synthesis engine, and a voice output device. The following describes the details of each component and how they work together.

[1238] First, the user speaks into a smart speaker. A smart speaker is a piece of hardware used for voice capture, collecting the user's voice using a built-in microphone. The system acts as a voice input device to accept the user's voice. This voice input device includes a noise reduction and signal normalization function, which normalizes the signal while removing ambient noise.

[1239] Next, while the device is capturing the audio, it uses an emotion engine to analyze the audio for emotional information. The emotion engine determines the user's emotional state based on parameters such as voice tone, tempo, and emphasis. This analysis can determine whether the user is feeling sad, happy, angry, etc.

[1240] After the voice is captured and the emotion is analyzed, the device uses a speech recognition engine to convert the voice into text data. For example, "I'm in a bad mood today, so I don't want to contact my friends." The voice is converted into text "I'm in a bad mood today, so I don't want to contact my friends." This text data and the user's profile information (such as age group, information literacy, and emotional information) are sent to the server as an HTTP request.

[1241] The server analyzes the received text data and emotional information and passes the necessary information to the generative AI engine. The generative AI engine generates an appropriate response based on the user's profile information and emotional information. For example, based on the text "I'm in a bad mood today, so I don't want to contact my friends," and the emotional information "bad mood," the response generated is "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[1242] The generated response text is again passed to the speech synthesis engine by the server and converted into voice data, which is finally sent to the terminal as an HTTP response.

[1243] The device plays back the received voice data and provides a response to the user as voice. For example, by telling the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself," the user can take appropriate action.

[1244] Specific examples

[1245] For example, if a user says, "I'm in a bad mood today, so I don't want to contact my friends," the system works as follows:

[1246] 1. The device captures the audio and performs noise reduction and signal normalization.

[1247] 2. The device analyzes the emotional information of "bad mood" using an emotion engine.

[1248] 3. The device converts the speech into text data such as "I'm in a bad mood today, so I don't want to contact my friends," and sends this text data and profile information to the server.

[1249] 4. The server analyzes the received data, and the AI ​​engine generates a response saying, "You seem to be feeling unwell, but it's important to take a rest and not push yourself too hard."

[1250] 5. The server converts the response text into voice data and sends it to the device.

[1251] 6. The device plays audio data and tells the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself."

[1252] This process allows seniors to easily obtain the information they need through voice and provides easy-to-understand responses tailored to their specific emotional states, thereby improving their quality of life, reducing loneliness, and providing appropriate support.

[1253] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1254] Step 1:

[1255] A user speaks to a smart speaker, for example, "I'm in a bad mood today, so I don't want to contact my friends." This speech is the input for the entire system, and it is the starting point for processing.

[1256] Step 2:

[1257] The smart speaker of the device captures the user's voice using the smart speaker's microphone. The captured voice data is accompanied by noise data, so noise removal and signal normalization are required in the next processing steps.

[1258] Step 3:

[1259] The device performs noise reduction and signal normalization on the audio data. At this stage, background noise is removed from the captured audio data and the amplitude of the audio signal is adjusted to obtain clear audio data. This clear audio data is the input for the next step.

[1260] Step 4:

[1261] The device uses an emotion engine to analyze emotional information from the normalized voice data. The emotion engine analyzes the tone, tempo, and emphasis of the voice to determine the emotion the user is feeling when speaking. In this case, the analyzed emotion is displeased. The output is the voice data before it is converted into text and the emotional information (displeased).

[1262] Step 5:

[1263] The device uses a speech recognition engine to convert the normalized voice data into text data. This conversion process generates the text, "I'm in a bad mood today, so I don't want to contact my friends." The output is this text data.

[1264] Step 6:

[1265] The device sends an HTTP request to the server, including the converted text data and the user's profile information (such as age group, information literacy, and emotional information). The server receives the text data and the user profile information.

[1266] Step 7:

[1267] The server analyzes the received HTTP request and extracts text data and emotional information. In this example, the text data "I'm in a bad mood today, so I don't want to contact my friends." and emotional information (bad mood) are extracted within the server. The output is the analyzed text data and emotional information.

[1268] Step 8:

[1269] The server passes the extracted text data and emotional information to the generative AI engine. The generative AI engine generates the optimal response based on the user's profile information and emotional information. In this case, the response generated is, "You seem to be feeling unwell, but it's important to take it easy and get some rest." The output is the generated response text.

[1270] Step 9:

[1271] The server passes the generated response text to the speech synthesis engine, which converts the text into speech data. The speech synthesis engine generates speech data saying, "You seem to be feeling unwell, but it's important to take a rest and not push yourself." The output is speech data.

[1272] Step 10:

[1273] The server sends the generated audio data to the terminal as an HTTP response, and the terminal receives this audio data.

[1274] Step 11:

[1275] The device plays the received voice data and provides a response to the user. For example, a smart speaker might say to the user, "You seem to be feeling unwell, but it's important to take a rest and not push yourself." The user can listen to this voice response and take appropriate action.

[1276] (Application example 2)

[1277] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1278] When elderly people order food delivery services by voice, there is a need for a system that can provide appropriate menu suggestions and allow them to proceed with the ordering process comfortably and without stress. In particular, it is a challenge to increase user satisfaction and improve the quality of life by making suggestions that correspond to the emotional state of the elderly.

[1279] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1280] In this invention, the server includes means for accepting input by voice, means for converting the input voice into text, means for transmitting the converted text to the server via a network, means for analyzing the text and emotional information received by the server and generating easy-to-understand expressions and suggestions according to the user's emotional state, means for converting the reconstructed text and suggestions into voice data, and means for receiving and playing the voice data. This enables elderly people to receive appropriate meal suggestions according to their emotional state when ordering food delivery by voice.

[1281] "Means for accepting voice input" refers to a device that detects voices emitted by the user and inputs them into the system, such as a microphone or the voice input function of a smartphone.

[1282] The "means for converting input voice into text" refers to a voice recognition engine that converts voice data into character data, and has the function of converting voice into text using voice analysis technology.

[1283] The "means for transmitting the converted text to a server via a network" is a communication module for transmitting the converted text data to a server via the Internet or another network.

[1284] "Generation means that analyzes the text and emotional information received by the server and generates easy-to-understand expressions and suggestions according to the user's emotional state" refers to a generative AI engine or emotion analysis engine that generates appropriate responses and suggestions based on the received text and the user's emotional information.

[1285] The "means for converting the reconstructed text and suggestions into speech data" refers to a speech synthesis engine that converts the generated text and suggestions into speech data, and has the function of converting the generated information into easy-to-listen speech.

[1286] "Means for receiving and playing audio data" refers to a device that receives the converted audio data and plays it in a form that the user can hear, such as a speaker or the audio output function of a smartphone.

[1287] This invention realizes a system that allows elderly people to easily order food delivery by voice by using a voice input device, a voice recognition engine, a network communication module, a generative AI engine, an emotion engine, a voice synthesis engine, and a voice output device.

[1288] When using a smartphone, the user expresses their order preference using the smartphone's voice input function. For example, they might say, "I'm tired, please give me something to eat that will energize me." This voice is captured by the smartphone's microphone. The captured voice data is converted into text data by a speech recognition engine. Specifically, the voice is converted into text using the Google Cloud Speech-to-Text API.

[1289] The converted text is then sent to the server via the network. This network communication module uses HTTP. The server receives the received text and the user's emotional information, which is analyzed by the emotion engine. This emotion analysis uses IBM Watson Tone Analyzer.

[1290] The server then uses a generative AI engine (such as the OpenAI GPT-4 API) to generate easy-to-understand expressions and suggestions based on the emotional state of the user based on the received text and emotional information. For example, if the user is tired, a suggestion such as "Would you like some nutritious chicken soup and fresh fruit?" is generated. This generated text is then converted into speech data using a speech synthesis engine. Amazon Polly is used for this speech synthesis engine.

[1291] The converted voice data is then sent back to the smartphone via the network and played back through the smartphone's speaker, allowing the user to listen to the menu of voice suggestions and make a selection from them.

[1292] As a specific example, if a user says, "I'm tired, please give me something to eat that will give me energy," the following prompt sentence will be sent to the generation AI.

[1293] Example prompt sentence:

[1294] User said: I'm tired, please give me something to eat to cheer me up. Tone: Sad. Suggest a meal.

[1295] Here's an example of a response suggested by the generative AI:

[1296] How about a nutritious chicken soup with vegetables and a side of fresh fruit to boost your energy?

[1297] In this way, a system has been realized that allows elderly people to order food delivery by voice without stress, and suggestions based on emotion analysis improve satisfaction. This also improves the quality of life for elderly people and makes their daily lives more convenient.

[1298] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1299] Step 1:

[1300] The user uses the smartphone's voice input function to verbally state their order. At this time, the smartphone's microphone captures the user's voice and stores it as voice data. The input is the user's voice, and the output is the captured voice data.

[1301] Step 2:

[1302] The device converts the captured voice data into text data using the Google Cloud Speech-to-Text API. The input is the voice data, and the output is the converted text data. The speech recognition engine analyzes the voice and generates the corresponding text.

[1303] Step 3:

[1304] The terminal sends the converted text data to the server via the network communication module. The input is text data, and the output is text data as an HTTP request. The network communication module is responsible for sending.

[1305] Step 4:

[1306] The server uses IBM Watson Tone Analyzer to analyze the received text data and the user's emotional information. The input is the received text data and the emotional parameters extracted from the user's voice, and the output is the analyzed emotional information. The emotional analysis engine identifies the emotional state.

[1307] Step 5:

[1308] The server passes the analyzed emotional information and text data to the OpenAI GPT-4 API, and the generative AI engine generates appropriate suggestions based on this information. The input is text data and emotional information, and the output is the generated suggested text. The generative AI engine generates the suggestions.

[1309] Step 6:

[1310] The server converts the generated suggested text into speech data using Amazon Polly. The input is the suggested text and the output is the generated speech data. The speech synthesis engine converts the text into speech.

[1311] Step 7:

[1312] The server sends the generated voice data to the terminal via the network. The input is the voice data, and the output is the voice data as an HTTP response. The network communication module is responsible for sending the voice data.

[1313] Step 8:

[1314] The device plays the received voice data and conveys the suggestion to the user. The input is the received voice data, and the output is a voice response to the user. The smartphone speaker performs the specific operation of playing the voice data.

[1315] Step 9:

[1316] The user listens to the voice response played from the terminal, selects a suggested menu from the list, and proceeds with the ordering process. For example, the user confirms the suggested menu and then issues another voice command such as "I would like to order this menu item," thereby completing the order. The input is the voice response from the terminal, and the output is the execution of the order based on the user's selection.

[1317] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1318] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1319] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1320] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1321] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1322] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1323] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1324] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1325] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1326] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1327] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1328] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1329] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1330] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1331] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1332] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1333] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1334] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1335] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1336] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1337] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1338] The following is further disclosed regarding the above embodiment.

[1339] (Claim 1)

[1340] means for accepting input by voice;

[1341] a means for converting input speech into text;

[1342] means for transmitting the converted text to a server via a network;

[1343] a generating means for reconstructing the text received by the server in an easy-to-understand representation;

[1344] means for converting the reconstructed text into audio data;

[1345] means for receiving and playing audio data;

[1346] A system including:

[1347] (Claim 2)

[1348] 2. The system of claim 1, wherein the generating means automatically adjusts the text depending on the user's age group and information literacy level.

[1349] (Claim 3)

[1350] 10. The system of claim 1, wherein the means for accepting voice input comprises noise reduction and signal normalization.

[1351] "Example 1"

[1352] (Claim 1)

[1353] means for accepting input by voice;

[1354] a means for converting input speech into text;

[1355] means for transmitting the converted text to a server via a network;

[1356] A means for the server to analyze the received text and input it into the generative AI model;

[1357] A means for converting the response generated by the generative AI model into audio data;

[1358] means for receiving and playing audio data;

[1359] A system including:

[1360] (Claim 2)

[1361] 10. The system of claim 1, wherein the generative AI model automatically adjusts responses based on a user's age group and information literacy.

[1362] (Claim 3)

[1363] 10. The system of claim 1, wherein the means for accepting voice input comprises noise reduction and signal normalization.

[1364] "Application Example 1"

[1365] (Claim 1)

[1366] means for accepting input by voice;

[1367] a means for converting input speech into text;

[1368] means for transmitting the converted text to a server over a network;

[1369] a generating means for reconstructing the text received by the server in an easy-to-understand representation;

[1370] means for converting the reconstructed text into audio data;

[1371] means for receiving and playing audio data;

[1372] means for sending a notification to designated contacts based on voice input in the event of an emergency;

[1373] A system including:

[1374] (Claim 2)

[1375] 2. The system of claim 1, wherein the generating means automatically adjusts the text depending on the user's age group and information literacy level.

[1376] (Claim 3)

[1377] 10. The system of claim 1, wherein the means for accepting voice input comprises noise reduction and signal normalization.

[1378] "Example 2: Combining Emotion Engines"

[1379] (Claim 1)

[1380] means for accepting input by voice;

[1381] means for denoising and normalising the input speech signal;

[1382] A means for analyzing emotions from input speech;

[1383] a means for converting input speech into text;

[1384] means for transmitting the converted text and the user's profile information to a server via a network;

[1385] means for analyzing the received text and emotion information by the server;

[1386] A means for generating a response using a generative AI based on the analyzed information;

[1387] means for converting the generated text into audio data;

[1388] means for receiving and playing audio data;

[1389] A system including:

[1390] (Claim 2)

[1391] The system of claim 1 , wherein the generating means automatically adjusts the text according to the user's age group, information literacy, and emotional information.

[1392] (Claim 3)

[1393] 10. The system of claim 1, wherein the means for accepting voice input comprises noise reduction and signal normalization.

[1394] "Application example 2 when combining emotion engines"

[1395] (Claim 1)

[1396] means for accepting input by voice;

[1397] a means for converting input speech into text;

[1398] means for transmitting the converted text to a server via a network;

[1399] a generating means for analyzing the text and emotion information received by the server and generating easy-to-understand expressions and suggestions according to the user's emotional state;

[1400] means for converting the reconstructed text and suggestions into audio data;

[1401] means for receiving and playing audio data;

[1402] A system including:

[1403] (Claim 2)

[1404] The system of claim 1 , wherein the generating means automatically adjusts the text and suggestions according to the user's age group, information literacy, and emotional information.

[1405] (Claim 3)

[1406] 10. The system of claim 1, wherein the means for accepting voice input comprises noise reduction and signal normalization. [Explanation of symbols]

[1407] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for accepting input by voice; a means for converting input speech into text; means for transmitting the converted text to a server via a network; a generating means for reconstructing the text received by the server in an easy-to-understand representation; means for converting the reconstructed text into audio data; means for receiving and playing audio data; A system including:

2. The system of claim 1 , wherein the generating means automatically adjusts the text depending on the user's age group and information literacy level.

3. 10. The system of claim 1, wherein the means for accepting voice input includes noise reduction and signal normalization.

Images