System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring voice of a user; means for transmitting the voice to a server; means for converting the voice into text data in the server; means for analyzing the text data and generating an appropriate answer; means for converting the generated answer into voice data; means for transmitting the voice data to a terminal of the user; and means for reproducing the voice data in the terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] One problem facing seniors when it comes to obtaining information in their daily lives is the high barriers to using technology. Text input and complex operations are particularly difficult for seniors, hindering their ability to obtain information and resolve everyday concerns. Furthermore, conventional information acquisition systems have complex user interfaces that make it difficult for seniors to operate them intuitively. To address these issues, the present invention provides a system that uses voice to provide appropriate information simply and quickly. [Means for solving the problem]

[0005] The present invention provides a system including a means for acquiring a user's voice, a means for transmitting the voice to a server, a means for converting the voice into text data on the server, a means for analyzing the text data and generating an appropriate answer, a means for converting the generated answer into voice data, a means for transmitting the voice data to a user's terminal, and a means for playing the voice data on the terminal. This allows elderly people to easily ask questions by voice and quickly receive appropriate answers to those questions by voice. Furthermore, by using a natural language processing engine, the accuracy of analysis of the question content can be improved, and appropriate answers can be provided. Furthermore, by transmitting and receiving voice data using a communication protocol, real-time, highly accurate information provision is achieved.

[0006] "Voice" refers to input data that is words or vocalizations that a user speaks into a terminal.

[0007] "Users" are people who use this system to obtain information and support for everyday needs.

[0008] "Terminal" refers to an electronic device used by a user that is capable of voice input and voice output.

[0009] The "server" is a central processing unit that receives voice data from a terminal, converts the voice into text data, analyzes it, generates a response, converts it into voice data, and sends it to the terminal.

[0010] A "voice recognition engine" is software or hardware that runs on a terminal or server and converts voice data into text data.

[0011] "Text data" is character information of voice data generated by a voice recognition engine.

[0012] A "natural language processing engine" is software or hardware that analyzes text data, understands the user's intent, and generates appropriate answers.

[0013] An "answer" is text data generated by a natural language processing engine and containing appropriate information and advice in response to the user's question.

[0014] A "speech synthesis engine" is software or hardware for converting text data into voice data.

[0015] "Voice data" refers to digital data of voice generated by a voice recognition engine or a voice synthesis engine.

[0016] A "communication protocol" is a rule for sending and receiving voice data and text data between a terminal and a server.

[0017] "Analysis" is the process of using a natural language processing engine to understand text data and generate appropriate answers. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a system that realizes a "anything-to-anything consultation app" that responds to the everyday worries and questions of elderly users, and features no codes, no prompts, and no text input. This system allows users to obtain information easily, quickly, and accurately using only voice input.

[0040] An embodiment of the system operates as follows.

[0041] User voice input

[0042] The user speaks a question into the device. For example, "My legs have been hurting lately. What should I do?"

[0043] Recording audio on your device

[0044] The terminal has a function for recording the user's voice, records the user's question, and generates the voice data.

[0045] Sending voice data to the server

[0046] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket.

[0047] Server-based speech recognition

[0048] The server inputs the received voice data into a voice recognition engine and converts it into text data. For example, the resulting text data is "My leg has been hurting lately. What should I do?"

[0049] Server-based natural language analysis

[0050] The server sends the obtained text data to a natural language processing engine, which analyzes the user's question, understands the intent of the question, and generates an appropriate answer.

[0051] Speech conversion and transmission of text data

[0052] The generated text data of the answer is converted into voice data by a speech synthesis engine. For example, voice data with the content "If your leg hurts, it is important to first rest and cool it. If the pain persists, consult a doctor" is generated. The server sends this voice data to the device.

[0053] Playing audio on your device

[0054] The device receives the audio data sent from the server and plays it back, telling the user, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor."

[0055] As a concrete example, if a user asks the terminal, "My legs have been hurting lately. What should I do?", the system will operate as follows.

[0056] 1. The user provides voice input.

[0057] 2. The device records the audio and sends the data to the server.

[0058] 3. The server converts the audio data into text.

[0059] 4. The server analyzes the text data and generates an appropriate answer.

[0060] 5. The server converts the answer into audio data and sends it to the device.

[0061] 6. The device plays the audio data and tells the user the answer.

[0062] In this way, users can easily get answers to their concerns and questions through voice input. Also, since it is targeted at elderly users, the series of operations is designed to be intuitive and easy to perform.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] The user speaks a question into the device, which then records the user's voice. For example, the user might ask, "My legs have been hurting lately. What should I do?"

[0066] Step 2:

[0067] The device records the audio and prepares the recording, which is saved as an audio file.

[0068] Step 3:

[0069] The device sends the recorded audio data to the server using the HTTP or WebSocket communication protocol.

[0070] Step 4:

[0071] The server inputs the received voice data into a voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. For example, the generated text data is "My leg has been hurting lately. What should I do?"

[0072] Step 5:

[0073] The server sends the obtained text data to a natural language processing engine, which analyzes the text data and understands the meaning of the user's question.

[0074] Step 6:

[0075] The server's natural language processing engine generates an appropriate answer based on the analysis results. For example, a text data answer such as "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor."

[0076] Step 7:

[0077] The server generates a text response, which is then input to a speech synthesis engine and converted into voice data. The speech synthesis engine then converts the text response into voice data.

[0078] Step 8:

[0079] The server transmits the generated voice data to the terminal, and the voice data is transmitted using a communication protocol.

[0080] Step 9:

[0081] The terminal receives the voice data sent from the server, and then acquires and prepares the voice data.

[0082] Step 10:

[0083] The device plays back the audio data and gives the user a spoken response, specifically saying, "If your leg hurts, it's important to rest and cool it down. If the pain persists, consult a doctor."

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] For users who have difficulty using keyboards or touch input, such as the elderly, there is a need to provide a simple and intuitive interface that can quickly and accurately respond to their daily concerns and questions. However, conventional systems are complicated to operate, and the variety of input methods results in a high learning cost. To solve this problem, it is necessary to build a system that can obtain information using only voice input.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes a means for acquiring the user's voice, a means for converting the recorded voice data into text data, and a means for analyzing the text data and generating appropriate answers, thereby enabling users to quickly obtain answers to their questions and concerns by simply inputting their voice.

[0089] "Means for acquiring user's voice" refers to a device or function that physically detects the voice spoken by the user and acquires it as digital data.

[0090] The "recording terminal means" refers to a terminal or device that electronically records acquired voice and saves it as voice data.

[0091] The "means for transmitting recorded voice data to a server" refers to a function or protocol for transferring voice data recorded on a terminal to a server via a communication network.

[0092] "Means for converting voice data into text data on a server" refers to a process or system that uses voice recognition technology to convert voice data into text information.

[0093] The "means for analyzing text data and generating appropriate answers" is a function that uses natural language processing technology to understand the meaning of text data and generate appropriate answers to user questions.

[0094] The "means for converting the generated answers into voice data" refers to a process or device that converts the text-format answers into voice data using voice synthesis technology.

[0095] The "means for transmitting voice data to a user's terminal" refers to a function or protocol for transferring the generated voice data to a user's terminal via a communication network.

[0096] "Means for reproducing audio data at a terminal" refers to a function or device for reproducing audio data received by a user's terminal and audibly conveying the data to the user.

[0097] A "natural language processing model" is a trained algorithm or framework for understanding the meaning of text data and generating appropriate answers based on that understanding.

[0098] A "communications protocol" is a standardized procedure or rule for sending and receiving data over a network.

[0099] This invention is a system that realizes a "consultation app" that responds to the everyday worries and questions of elderly users. This system features no codes, no prompts, and no text input, allowing users to obtain information easily, quickly, and accurately using only voice input.

[0100] User voice input

[0101] The user speaks a question into the device. For example, they can ask a specific question such as, "My legs have been hurting lately. What should I do?" The user does not need to enter any special text, and can ask questions using only voice input.

[0102] Recording audio on your device

[0103] The device records the user's voice using the built-in microphone. The device uses a voice recorder application built into the device's operating system. When the user finishes answering a question, the recording stops automatically and the voice data is generated.

[0104] Sending voice data to the server

[0105] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket, where it is converted into an appropriate format (e.g., WAV or MP3) and uploaded to a specified endpoint on the server.

[0106] Server-based speech recognition

[0107] The server inputs the received voice data into a voice recognition engine. The voice recognition engine used can be a general-purpose voice recognition API (for example, Google Speech-to-Text API or IBM Watson Speech to Text). This converts the voice data into text data, and specific text data such as "My leg has been hurting lately. What should I do?" is obtained.

[0108] Server-based natural language analysis

[0109] The server sends the text data to a natural language processing engine, such as OpenAI's GPT-3 or Google Natural Language API, to analyze the question, allowing the server to generate an appropriate answer to the user's question.

[0110] Speech conversion and transmission of text data

[0111] The generated text data of the answer is sent by the server to a speech synthesis engine. Examples of speech synthesis engines used include Google Text-to-Speech API and Amazon Polly. For example, specific speech data such as "If your leg hurts, it's important to first rest and ice it. If the pain persists, consult a doctor" is generated. The server then sends this speech data to the device.

[0112] Playing audio on your device

[0113] The device receives the audio data sent from the server and plays it back using a built-in speaker, providing the user with specific advice such as, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor."

[0114] Specific examples

[0115] For example, if a user asks a question by voice, such as "My legs have been hurting lately, what should I do?", the system operates as follows: First, the device records the voice and sends the data to the server. Next, the server converts the voice data into text, analyzes the text data, and generates an appropriate answer. Finally, the generated answer is converted into voice data, sent to the device, and played back to the user.

[0116] In this way, users can easily get answers to their concerns and questions through voice input. Also, since it is targeted at elderly users, the series of operations is designed to be intuitive and easy to perform.

[0117] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0118] Step 1:

[0119] The user speaks a question into the terminal. For example, "My legs have been hurting lately. What should I do?" The input is the user's voice, and the output is the voice data recorded by the terminal.

[0120] Step 2:

[0121] The device uses a built-in microphone to record the user's voice and save it as digital audio data. Specifically, it uses a voice recorder application built into the device's operating system and saves it as an audio file in wav or mp3 format. The input is the user's voice and the output is an audio file.

[0122] Step 3:

[0123] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket. Specifically, the device converts the audio data into an appropriate format and uploads it to a specified endpoint on the server. The input is the audio file, and the output is the audio data sent to the server.

[0124] Step 4:

[0125] The server inputs the received voice data into a voice recognition engine. The voice recognition engine used can be a general voice recognition API (for example, Google Speech-to-Text API). The input is voice data, and the output is converted text data. Specifically, the voice data is sent to the voice recognition API, and text data such as "My leg has been hurting lately, what should I do?" is obtained.

[0126] Step 5:

[0127] The server sends the obtained text data to a natural language processing engine, which analyzes the question. The natural language processing engine uses OpenAI's GPT-3 or equivalent natural language processing technology. The input is text data, and the output is the analyzed answer text. Specifically, the text data is sent to the natural language processing engine, and the answer obtained is, "If your leg hurts, it is important to first rest and ice it. If the pain persists, consult a doctor."

[0128] Step 6:

[0129] The server sends the generated answer text to a speech synthesis engine and converts it into audio data. Examples of speech synthesis engines that can be used include Google Text-to-Speech API and Amazon Polly. The input is the answer text and the output is audio data. Specifically, the answer text is sent to the speech synthesis engine, which generates an audio file (e.g., mp3 format).

[0130] Step 7:

[0131] The server sends the generated audio data to the device. The transmission method uses a communication protocol (HTTP or WebSocket). The input is the audio data, and the output is the audio data sent to the device.

[0132] Step 8:

[0133] The device plays the received voice data. Specifically, it plays the voice data using the device's speaker and provides the answer in an audible format to the user. The input is the voice data, and the output is the voice answer presented to the user.

[0134] (Application example 1)

[0135] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] There is a demand for systems that can easily resolve questions and problems that elderly people and users unfamiliar with technology have in their daily lives and while shopping. In particular, physical stores such as supermarkets lack easy ways to obtain information about products and sales areas. Conventional systems that require text input or complex operations are difficult for elderly people to use, so there is a need to develop systems that can be used intuitively with only voice input.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0138] In this invention, the server includes a means for converting speech into text data, a means for generating appropriate responses using a generative AI model, and a means for inputting prompt sentences, which enables elderly people and users who are unfamiliar with technology to easily obtain product locations and information in a store using only voice input.

[0139] "User" means a person who uses the system.

[0140] "Means for acquiring audio" refers to devices or software that record the audio emitted by the user and store it as digital data.

[0141] "Server" refers to a computer system for receiving and processing audio data.

[0142] "Means for converting into text data" refers to the technology and software that converts audio data into text information.

[0143] "Means of analysis and generating appropriate answers" refers to the technologies and algorithms used to analyze text data and create optimal answers to users' questions.

[0144] "Means for converting generated answers into audio data" refers to technology or software that converts text-based answers into data that can be played as audio.

[0145] "Terminal" refers to a device (smartphone, smart glasses, etc.) that is directly operated by the user and used to input and output audio.

[0146] "Means for playback" refers to the function for outputting audio data received by the terminal as audio through a speaker or the like.

[0147] "A means of support for users to ask about product locations and information by voice in a store" refers to an auxiliary function that allows users to ask questions about specific products or information in a physical store.

[0148] "Generative AI model" refers to artificial intelligence technology used to generate appropriate answers to user questions.

[0149] "Means for inputting prompt sentences" refers to the technology or method for inputting sentences or data that give operational instructions to the generative AI model.

[0150] "Communication protocol" refers to the rules and procedures for transmitting digital information.

[0151] "Natural language processing engine" refers to technologies and algorithms for understanding, processing, and generating natural language text.

[0152] The system of this invention aims to enable elderly people and users unfamiliar with technology to easily obtain information about products and sales areas in brick-and-mortar stores such as supermarkets using only their voice. The specific system configuration and operating procedure are shown below.

[0153] System configuration

[0154] Device: A smartphone or smart glasses held by a user, used to input voice and play audio output.

[0155] Server: Responsible for processing voice data. Has functions such as speech recognition, natural language processing, and answer generation using generative AI models.

[0156] Hardware and software used

[0157] Speech recognition engine: Used to convert the user's voice into text data.

[0158] Natural language processing engine: An algorithm that analyzes text data and generates appropriate answers using generative AI models.

[0159] Generative AI model: An artificial intelligence technique for generating optimal answers to user questions.

[0160] Speech synthesis engine: Used to convert the generated text response into audio data.

[0161] Communication protocol: A communication rule such as HTTP or WebSocket for sending and receiving data between a terminal and a server.

[0162] Program processing explanation

[0163] 1. User voice input: The user voices a question into the device, for example, "Where are the tomatoes?"

[0164] 2. Voice recording on the device: The device records the user's voice and generates the voice data.

[0165] 3. Sending audio data to the server: The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket.

[0166] 4. Speech recognition on the server: The server inputs the received voice data into a voice recognition engine and converts the voice into text data. For example, the resulting text data is "Where are the tomatoes?"

[0167] 5. Natural language analysis on the server: The server sends the obtained text data to a natural language processing engine, which analyzes the question, understands the intent of the question, and generates an appropriate answer using a generative AI model.

[0168] 6. Text-to-speech conversion and transmission: The generated text answer is converted into voice data by a speech synthesis engine. For example, voice data such as "The tomatoes are in aisle 3" is generated, and the server transmits this voice data to the device.

[0169] 7. Playback on the device: The device receives the audio data sent from the server and plays it back, telling the user the answer: "The tomatoes are in aisle 3."

[0170] Specific examples

[0171] If a user asks a terminal, "Where are the tomatoes?", the system will do the following:

[0172] Speak: "Where are the tomatoes?"

[0173] Request to server: "Where are the tomatoes?"

[0174] Server responds: "Tomatoes are in aisle 3."

[0175] Speech synthesis and playback: The audio response "The tomatoes are in aisle 3" is played.

[0176] Prompt Sentence Examples

[0177] "If a user asks, 'Where are the tomatoes?' generate the right answer."

[0178] This system will enable even elderly people and those unfamiliar with technology to easily obtain information using voice within the supermarket, greatly improving convenience.

[0179] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0180] Step 1:

[0181] The user speaks a question into the device. For example, "Where are the tomatoes?" The input for this step is the user's voice, and the output is the voice data recorded by the device. Specifically, the device's microphone captures the voice and records it in digital format.

[0182] Step 2:

[0183] The device records the user's voice and generates voice data. The generated voice data is saved as a digital file. The input for this step is the user's voice, and the output is the recorded voice data. Specifically, the data is saved in a format required for input to the voice recognition engine.

[0184] Step 3:

[0185] The device sends the recorded audio data to the server. The input of this step is the audio data, and the output is the audio data sent to the server. Specifically, the device sends the audio data to the server using a communication protocol such as HTTP or WebSocket.

[0186] Step 4:

[0187] The server inputs the received voice data into a voice recognition engine and converts the voice into text data. The input for this step is the voice data sent to the server, and the output is the converted text data. Specifically, the voice recognition engine analyzes the sound wave data and generates the text "Where are the tomatoes?"

[0188] Step 5:

[0189] The server sends the obtained text data to a natural language processing engine, which analyzes the user's question. It understands the intent of the question and generates an appropriate answer using a generative AI model. The input to this step is text data, and the output is the text data of the generated answer. Specifically, the natural language processing engine analyzes the text and provides a prompt sentence to the generative AI model to generate an answer.

[0190] Step 6:

[0191] The server converts the generated text data of the answer into voice data using a voice synthesis engine. The input of this step is the generated text data of the answer, and the output is the converted voice data. Specifically, the voice synthesis engine generates the voice saying, "The tomatoes are in aisle 3."

[0192] Step 7:

[0193] The server sends the generated voice data to the user's terminal. The input of this step is the voice data, and the output is the voice data sent to the terminal. In specific operations, the server sends the voice data to the terminal using a communication protocol.

[0194] Step 8:

[0195] The device receives the voice data sent from the server and plays it back. The input of this step is the voice data sent to the device, and the output is the voice response played back to the user. Specifically, the device's speaker plays back the voice, "The tomatoes are in aisle 3."

[0196] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0197] This invention is a system that realizes a "consultation app" that responds to the daily worries and questions of elderly users, and features no codes, no prompts, and no text input. This system allows users to easily, quickly, and accurately obtain information using only voice input. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide answers that correspond to the user's emotional state.

[0198] An embodiment of the system operates as follows.

[0199] User voice input

[0200] The user speaks a question into the device, which then records the user's voice. For example, the user might ask, "My legs have been hurting lately. What should I do?"

[0201] Recording audio on your device

[0202] The terminal has a function for recording the user's voice, records the user's question, and generates the voice data.

[0203] Sending voice data to the server

[0204] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket.

[0205] Server-based speech recognition

[0206] The server inputs the received voice data into a voice recognition engine and converts it into text data. For example, the resulting text data is "My leg has been hurting lately. What should I do?"

[0207] Emotion recognition on the server

[0208] The server sends the acquired voice and text data to the emotion engine to recognize the user's emotion. The emotion engine analyzes the tone, speed, and intensity of the voice to recognize the user's emotional state.

[0209] Server-based natural language analysis

[0210] The server sends the emotion data and text data obtained from the emotion engine to the natural language processing engine, which analyzes the text data, understands the meaning of the user's question, and generates an appropriate answer taking the emotion data into account.

[0211] Speech conversion and transmission of text data

[0212] The generated text data of the answer is converted into voice data by a speech synthesis engine. For example, voice data with the content "If your leg hurts, it is important to first rest and cool it. If the pain persists, consult a doctor" is generated. The server sends this voice data to the device.

[0213] Playing audio on your device

[0214] The device receives the audio data sent from the server and plays it back, telling the user, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor."

[0215] As a concrete example, if a user asks the terminal, "My legs have been hurting lately. What should I do?", the system will operate as follows.

[0216] 1. The user provides voice input.

[0217] 2. The device records the audio and sends the data to the server.

[0218] 3. The server converts the audio data into text.

[0219] 4. The server analyzes the text and voice data using an emotion engine to recognize the user's emotions.

[0220] 5. The server analyzes the text data and generates an appropriate answer.

[0221] 6. The server converts the answer into audio data and sends it to the device.

[0222] 7. The device plays the audio data and tells the user the answer.

[0223] In this way, users can easily get answers to their concerns and questions through voice input. Since the app is targeted at elderly users, the entire process is designed to be intuitive and easy to follow. Furthermore, the introduction of an emotion engine makes it possible to respond in accordance with the user's emotions, providing more personalized support.

[0224] The processing flow will be explained below.

[0225] Step 1:

[0226] The user speaks a question into the device, which then records the user's voice. For example, the user might ask, "My legs have been hurting lately. What should I do?"

[0227] Step 2:

[0228] The device records the user's voice and prepares the recording, which is then saved as an audio file.

[0229] Step 3:

[0230] The device sends the recorded audio data to the server using the HTTP or WebSocket communication protocol.

[0231] Step 4:

[0232] The server inputs the received voice data into a voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. For example, the generated text data is "My leg has been hurting lately. What should I do?"

[0233] Step 5:

[0234] The server inputs the acquired voice data into the emotion engine, which analyzes the tone, speed, and intensity of the voice to recognize the user's emotional state. For example, if the voice tone is low and the speed is slow, it will recognize that the user is feeling anxious.

[0235] Step 6:

[0236] The server sends the emotion data and text data obtained from the emotion engine to the natural language processing engine, which analyzes the text data and understands the meaning of the user's question.

[0237] Step 7:

[0238] Based on the analysis results, the server's natural language processing engine takes emotional data into consideration and generates an appropriate answer. For example, a text answer such as "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor" is generated.

[0239] Step 8:

[0240] The server generates a text response, which is then input to a speech synthesis engine and converted into voice data. The speech synthesis engine then converts the text response into voice data.

[0241] Step 9:

[0242] The server transmits the generated voice data to the terminal, and the voice data is transmitted using a communication protocol.

[0243] Step 10:

[0244] The terminal receives the voice data sent from the server, and then acquires and prepares the voice data.

[0245] Step 11:

[0246] The device plays back the audio data and gives the user a response in voice. Specifically, it says, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor." Furthermore, if it recognizes that the user is feeling anxious, it plays back the audio in a calmer tone.

[0247] Example 2

[0248] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0249] In modern society, there is a need for prompt and accurate responses to the worries and questions that elderly people have in their daily lives. However, text input and complex operations are a significant burden for elderly people, which creates a barrier to obtaining information. Furthermore, a lack of responses that respond to emotional changes is one of the factors that amplify the anxiety and stress of elderly people. To solve these problems, a system that allows easy information acquisition through voice input alone and that responds with empathy is needed.

[0250] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0251] In this invention, the server includes means for acquiring a user's voice, means for transmitting the voice to the server, means for converting the voice to text data in the server, means for analyzing the text data and voice data to recognize the user's emotion, means for analyzing the text data and generating an appropriate answer, means for converting the generated answer to voice data, means for transmitting the voice data to the user's terminal, and means for playing back the voice data in the terminal. This enables elderly people to easily obtain answers to their worries and questions through voice input, and realizes responses that are sensitive to their emotions.

[0252] A "user" is a person who wishes to use the system to provide voice input and obtain information.

[0253] "Means for acquiring voice" refers to a device for collecting voice information from a user and recording it as digital data.

[0254] A "server" is a computer system that processes and analyzes voice data over a network and generates appropriate responses.

[0255] The "means for transmitting audio to a server" refers to a device or program that has the function of transferring audio data acquired by a terminal to a server using a communication protocol.

[0256] A "means for converting voice into text data" is a program or engine that uses voice recognition technology to convert the content of voice into text.

[0257] "Means for recognizing a user's emotions by analyzing text data and voice data" refers to software or an engine for determining a user's emotional state from the tone of voice and the content of text.

[0258] "Means for analyzing text data and generating appropriate answers" refers to a program or engine that uses natural language processing technology to understand the meaning of input text and generate answers based on that content.

[0259] The "means for converting the generated answer into voice data" is software or an engine with speech synthesis technology that converts text data into a format that can be played back as voice.

[0260] The "means for transmitting voice data to a user's terminal" refers to a device or program that has the function of transmitting voice data generated by a server to a user's terminal using a communication protocol.

[0261] "Means for reproducing the audio data at the terminal" refers to a device that has the function of reproducing the audio data received at the terminal in an audible format for the user.

[0262] This invention is a system that realizes a "consultation app" that responds to the daily worries and questions of elderly users, and features no codes, no prompts, and no text input. Specifically, users can easily, quickly, and accurately obtain information using only voice input. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide answers that correspond to the user's emotional state.

[0263] User voice input

[0264] The user speaks their question into the device. The device is equipped with a built-in microphone, and the user can start voice input by pressing the voice input button. For example, they might ask, "My leg has been hurting lately. What should I do?"

[0265] Recording audio on your device

[0266] The terminal has a function to record the user's voice, and uses a built-in microphone to record the user's questions and generate the voice data. The recording module converts the input voice into digital data.

[0267] Sending voice data to the server

[0268] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket. For example, the audio data can be sent to the server using an HTTP POST request.

[0269] Server-based speech recognition

[0270] The server inputs the received voice data into a voice recognition engine and converts it into text data. The voice recognition engine uses, for example, the Google Speech-to-Text API. The resulting text data is, "My leg has been hurting lately. What should I do?"

[0271] Emotion recognition on the server

[0272] The server sends the acquired voice and text data to an emotion engine, such as IBM Watson Tone Analyzer, which analyzes the tone, speed, and intensity of the voice to determine whether the user is "confused."

[0273] Server-based natural language analysis

[0274] The server sends the emotion data obtained from the emotion engine and the recognized text data to a natural language processing engine. For example, OpenAI GPT-3 is used for the natural language processing engine. This analyzes the text data, understands the meaning of the user's question, and generates an appropriate answer taking into account the emotion data. The answer it generates is, "If your leg hurts, it's important to first rest and apply ice. If the pain persists, consult a doctor."

[0275] Speech conversion and transmission of text data

[0276] The generated text data of the answer is converted into voice data by a speech synthesis engine. For example, Amazon Polly is used as the speech synthesis engine. This generates voice data that says, "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor." The server then sends this voice data to the device via an HTTP POST request.

[0277] Playing audio on your device

[0278] The device receives the voice data sent from the server and plays it back. Specifically, it tells the user, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor." The device with a speaker also plays back the voice.

[0279] Specific examples

[0280] When a user asks the terminal, "My legs have been hurting lately. What should I do?", the system operates as follows.

[0281] 1. The user provides voice input.

[0282] 2. The device records the audio and sends the data to the server.

[0283] 3. The server converts the audio data into text.

[0284] 4. The server analyzes the text and voice data using an emotion engine to recognize the user's emotions.

[0285] 5. The server analyzes the text data and generates an appropriate answer.

[0286] 6. The server converts the answer into audio data and sends it to the device.

[0287] 7. The device plays the audio data and tells the user the answer.

[0288] This system allows users to easily obtain answers to their concerns and questions through voice input. As it is targeted at elderly users, the system is designed to be intuitive and easy to operate. Furthermore, the introduction of an emotion engine makes it possible to respond in accordance with the user's emotions, providing more personalized support. This system is designed to be particularly easy to use for the elderly, and is capable of responding appropriately through emotion recognition, which is expected to increase user satisfaction.

[0289] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0290] Step 1: User Speech Input

[0291] The user speaks a question into the device. For example, if the question is, "My legs have been hurting lately, what should I do?", this voice becomes the input for the system. The user starts voice input by pressing the voice input button on the device and speaking.

[0292] Step 2: Record audio on your device

[0293] The device records the user's voice. It uses the built-in microphone to capture the user's question and generates the voice data. This generated voice data becomes the input for the next process. Specifically, the recording module generates the voice data, "My legs have been hurting lately. What should I do?"

[0294] Step 3: Sending audio data to the server

[0295] The device sends the recorded audio data to the server. A communication protocol such as HTTP or WebSocket is used for transmission. This input audio data is transferred to the server via an HTTP POST request. The audio data sent to the server becomes the input for the next process.

[0296] Step 4: Speech recognition on the server

[0297] The server inputs the received voice data into a voice recognition engine and converts it into text data. The voice recognition engine uses the Google Speech-to-Text API or similar. The generated text data, "My leg has been hurting lately, what should I do?", becomes the input for the next process.

[0298] Step 5: Emotion Recognition on the Server

[0299] The server sends the obtained voice and text data to an emotion engine to recognize the user's emotions. The emotion engine uses a system such as IBM Watson Tone Analyzer. It analyzes the tone, speed, and intensity of the voice and determines that the user is "confused." This emotion data and text data become the input for the next process.

[0300] Step 6: Natural Language Analysis on the Server

[0301] The server sends the emotion data obtained from the emotion engine and the recognized text data to a natural language processing engine. The natural language processing engine uses OpenAI GPT-3 and other technologies. This analyzes the text data and generates a response text such as, "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor." The generated response text becomes the input for the next process.

[0302] Step 7: Convert text to speech and send it

[0303] The generated text response is converted into voice data by a speech synthesis engine. Amazon Polly or another similar engine is used as the voice synthesis engine. Voice data is generated that says, "If your leg hurts, it's important to rest and cool it. If the pain persists, consult a doctor." This voice data becomes the input for the next process and is sent to the device via an HTTP POST request.

[0304] Step 8: Play audio on your device

[0305] The device receives the voice data sent from the server and plays it back. Specifically, the device's speaker plays back the following voice: "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor." This voice playback is the system's final output, allowing the user to obtain an appropriate answer.

[0306] This system allows users to easily obtain answers to their concerns and questions through voice input. Since it is targeted at elderly users, the operation is intuitive and easy. Furthermore, the introduction of an emotion engine makes it possible to respond in accordance with the user's emotions, providing more personalized support.

[0307] (Application example 2)

[0308] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0309] In recent years, when elderly users use food delivery services in their daily lives, they are faced with a lack of diverse options and suggestions that take into account their health status. Furthermore, elderly users are often unfamiliar with the operation of these services, and certain emotional states can make it difficult for them to make appropriate choices. Furthermore, while intuitive operation based on voice input is required, existing systems do not take emotion recognition into account, which can result in low user satisfaction.

[0310] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a user's voice, means for transmitting the voice to the server, means for converting the voice to text data in the server, means for analyzing the text data and generating an appropriate response, means for converting the generated response to voice data, means for transmitting the voice data to the user's terminal, means for playing the voice data in the terminal, means for recognizing the user's emotion, and means for generating a response based on the emotion. This enables elderly users to intuitively and easily use food delivery services and receive appropriate suggestions while taking their emotions into consideration.

[0311] The "means for acquiring the user's voice" refers to a combination of hardware and software for recognizing and acquiring the voice uttered by the user via a device.

[0312] The "means for transmitting the voice to the server" refers to a communication protocol and software function for transmitting the acquired voice data to the server via a network.

[0313] The "means for converting the voice into text data in the server" is a function for converting voice data into text data using a voice recognition engine in the server.

[0314] The "means for analyzing the text data and generating an appropriate response" refers to the algorithm and software functions that analyze the text data using a natural language processing engine and generate a response appropriate to the user's intention or question.

[0315] The "means for converting the generated answer into voice data" is a function for converting the generated text-format answer into voice data using a voice synthesis engine.

[0316] The "means for transmitting the voice data to the user's terminal" refers to a communication protocol and software function for transmitting the generated voice data to the user's terminal via a network.

[0317] The "means for reproducing the audio data at the terminal" refers to the hardware and software functions for reproducing the audio data received at the terminal and allowing the user to listen to it.

[0318] The "means for recognizing the user's emotions" is a function that analyzes acquired voice and text data and identifies the user's emotional state using an emotion engine.

[0319] The "means for generating an answer based on emotion" is a function of the natural language processing engine that takes into account the recognized emotional state of the user to generate the most appropriate answer.

[0320] This invention is a "smart food assistant" system that allows elderly people to use food delivery services easily and intuitively. The system uses emotion recognition based on the user's voice input to provide optimal meal suggestions.

[0321] System configuration

[0322] Hardware

[0323] 1. User device: Use a device that allows voice input, such as a smartphone or smart speaker.

[0324] 2. Server: A server equipped with a high-performance processor performs speech recognition, natural language processing, and emotion recognition.

[0325] software

[0326] 1. Speech recognition engine: Used to convert the user's voice into text data. Common examples include Google's speech recognition API.

[0327] 2. Natural Language Processing Engine (NLP Engine): Used to analyze text data using generative AI models and generate appropriate suggestions.

[0328] 3. Emotion Recognition Engine: Identifies the user's emotional state from voice and text data. Transformers, a Python library, is available.

[0329] 4. Speech synthesis engine: Used to convert the generated suggestions into audio data. gTTS is an example of this.

[0330] Processing flow

[0331] Generating a Program

[0332] The program consists of the following steps:

[0333] 1. Voice input acceptance: When a user voice-over asks questions or asks questions about food delivery, the device captures and records the voice.

[0334] 2. Audio data transmission: The acquired audio data is sent to the server via the Internet using HTTP or WebSocket as the communication protocol.

[0335] 3. Speech recognition and text conversion: The server uses a speech recognition engine to convert the voice data into text data.

[0336] 4. Emotion recognition: Using the converted text data and the original audio data, the emotion recognition engine analyzes the user's emotions.

[0337] 5. Natural Language Processing and Suggestion Generation: The NLP engine analyzes the user's intent and sentiment to generate optimal meal suggestions.

[0338] 6. Speech synthesis: Converting the generated suggestions into audio data.

[0339] 7. Sending and playing back audio data: The server sends the generated audio data to the terminal, and the terminal plays back the audio data to convey the suggestion to the user.

[0340] Specific examples

[0341] Example 1: Elderly users have trouble deciding what to eat

[0342] User says: "I'm not sure what to have for dinner tonight. Can you recommend something?"

[0343] The system responds: "The weather is nice today, how about a fresh vegetable salad?" (If the user seems a little depressed: "I recommend a meat dish to energize you today.")

[0344] Example prompt

[0345] "I'm not sure what to have for dinner tonight. What do you recommend?" -> An older user says they're having trouble deciding what to eat. They seem a little depressed. Take their emotions into consideration and offer healthy, uplifting meal suggestions.

[0346] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0347] Step 1:

[0348] Voice input reception

[0349] The user speaks, "I'm not sure what to have for dinner tonight. What do you recommend?" The device records the speech. The recorded speech data undergoes pre-processing before being sent to the speech recognition engine. Specifically, digital signal processing is performed to change the sampling rate of the speech data to an appropriate value and remove noise.

[0350] Input: User's voice

[0351] Output: Preprocessed audio data

[0352] Step 2:

[0353] Audio data transmission

[0354] After preprocessing, the audio data is sent from the device to the server using HTTP or WebSocket communication protocols. The audio data is split into network packets and securely transmitted using a secure transport layer.

[0355] Input: Preprocessed audio data

[0356] Output: Audio data sent to the server

[0357] Step 3:

[0358] Speech recognition and text conversion

[0359] The server inputs the received voice data into a voice recognition engine, which converts the voice data into text data. Here, features are extracted from the voice data and a deep learning model is used to convert the voice into text. As a specific example, the generated text is, "I'm not sure what to have for dinner tonight. What do you recommend?"

[0360] Input: Audio data received by the server

[0361] Output: Text data (e.g., "I'm not sure what to have for dinner tonight. What do you recommend?")

[0362] Step 4:

[0363] emotion recognition

[0364] The server inputs the converted text data and the original voice data into an emotion recognition engine to analyze the user's emotions. The emotion engine analyzes the tone, speed, and intensity of the voice and the content of the text to identify the user's emotional state. For example, it may recognize that the user is depressed.

[0365] Input: Text and audio data

[0366] Output: Emotion data (e.g., depressed)

[0367] Step 5:

[0368] Natural Language Processing and Suggestion Generation

[0369] The server inputs the emotion data and text data into a natural language processing engine (NLP engine) and generates suggestions based on the user's intentions and emotions. The NLP engine uses a generative AI model to generate suggestions such as "I'm not sure what to have for dinner tonight" and "I'm feeling down," based on the information that the user is unsure about what to have for dinner tonight.

[0370] Input: Text data and emotion data

[0371] Output: Suggestion text (e.g. "Today, I recommend a meat dish to energize you.")

[0372] Step 6:

[0373] Speech synthesis

[0374] The server inputs the generated suggested text into a speech synthesis engine and converts it into voice data. The speech synthesis engine converts the text into natural language and outputs it as an audio file. Specifically, software such as "gTTS" is used to convert the suggested text into an audio file.

[0375] Input: Suggested text

[0376] Output: Speech data (e.g., "I recommend a meat dish to energize you today.")

[0377] Step 7:

[0378] Sending and playing audio data

[0379] The server sends the generated audio data to the user's device using HTTP or WebSocket communication protocols. The device plays the received audio data and provides suggestions to the user by decoding the audio data and playing it on the speaker.

[0380] Input: Audio data sent from the server

[0381] Output: Speech played to the user (e.g., "Today, I recommend a meat dish to energize you.")

[0382] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0383] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0384] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0385] [Second embodiment]

[0386] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0387] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0388] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0389] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0390] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0391] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0392] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0393] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0394] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0395] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0396] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0397] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0398] This invention is a system that realizes a "anything-to-anything consultation app" that responds to the everyday worries and questions of elderly users, and features no codes, no prompts, and no text input. This system allows users to obtain information easily, quickly, and accurately using only voice input.

[0399] An embodiment of the system operates as follows.

[0400] User voice input

[0401] The user speaks a question into the device. For example, "My legs have been hurting lately. What should I do?"

[0402] Recording audio on your device

[0403] The terminal has a function for recording the user's voice, records the user's question, and generates the voice data.

[0404] Sending voice data to the server

[0405] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket.

[0406] Server-based speech recognition

[0407] The server inputs the received voice data into a voice recognition engine and converts it into text data. For example, the resulting text data is "My leg has been hurting lately. What should I do?"

[0408] Server-based natural language analysis

[0409] The server sends the obtained text data to a natural language processing engine, which analyzes the user's question, understands the intent of the question, and generates an appropriate answer.

[0410] Speech conversion and transmission of text data

[0411] The generated text data of the answer is converted into voice data by a speech synthesis engine. For example, voice data with the content "If your leg hurts, it is important to first rest and cool it. If the pain persists, consult a doctor" is generated. The server sends this voice data to the device.

[0412] Playing audio on your device

[0413] The device receives the audio data sent from the server and plays it back, telling the user, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor."

[0414] As a concrete example, if a user asks the terminal, "My legs have been hurting lately. What should I do?", the system will operate as follows.

[0415] 1. The user provides voice input.

[0416] 2. The device records the audio and sends the data to the server.

[0417] 3. The server converts the audio data into text.

[0418] 4. The server analyzes the text data and generates an appropriate answer.

[0419] 5. The server converts the answer into audio data and sends it to the device.

[0420] 6. The device plays the audio data and tells the user the answer.

[0421] In this way, users can easily get answers to their concerns and questions through voice input. Also, since it is targeted at elderly users, the series of operations is designed to be intuitive and easy to perform.

[0422] The processing flow will be explained below.

[0423] Step 1:

[0424] The user speaks a question into the device, which then records the user's voice. For example, the user might ask, "My legs have been hurting lately. What should I do?"

[0425] Step 2:

[0426] The device records the audio and prepares the recording, which is saved as an audio file.

[0427] Step 3:

[0428] The device sends the recorded audio data to the server using the HTTP or WebSocket communication protocol.

[0429] Step 4:

[0430] The server inputs the received voice data into a voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. For example, the generated text data is "My leg has been hurting lately. What should I do?"

[0431] Step 5:

[0432] The server sends the obtained text data to a natural language processing engine, which analyzes the text data and understands the meaning of the user's question.

[0433] Step 6:

[0434] The server's natural language processing engine generates an appropriate answer based on the analysis results. For example, a text data answer such as "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor."

[0435] Step 7:

[0436] The server generates a text response, which is then input to a speech synthesis engine and converted into voice data. The speech synthesis engine then converts the text response into voice data.

[0437] Step 8:

[0438] The server transmits the generated voice data to the terminal, and the voice data is transmitted using a communication protocol.

[0439] Step 9:

[0440] The terminal receives the voice data sent from the server, and then acquires and prepares the voice data.

[0441] Step 10:

[0442] The device plays back the audio data and gives the user a spoken response, specifically saying, "If your leg hurts, it's important to rest and cool it down. If the pain persists, consult a doctor."

[0443] Example 1

[0444] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0445] For users who have difficulty using keyboards or touch input, such as the elderly, there is a need to provide a simple and intuitive interface that can quickly and accurately respond to their daily concerns and questions. However, conventional systems are complicated to operate, and the variety of input methods results in a high learning cost. To solve this problem, it is necessary to build a system that can obtain information using only voice input.

[0446] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0447] In this invention, the server includes a means for acquiring the user's voice, a means for converting the recorded voice data into text data, and a means for analyzing the text data and generating appropriate answers, thereby enabling users to quickly obtain answers to their questions and concerns by simply inputting their voice.

[0448] "Means for acquiring user's voice" refers to a device or function that physically detects the voice spoken by the user and acquires it as digital data.

[0449] The "recording terminal means" refers to a terminal or device that electronically records acquired voice and saves it as voice data.

[0450] The "means for transmitting recorded voice data to a server" refers to a function or protocol for transferring voice data recorded on a terminal to a server via a communication network.

[0451] "Means for converting voice data into text data on a server" refers to a process or system that uses voice recognition technology to convert voice data into text information.

[0452] The "means for analyzing text data and generating appropriate answers" is a function that uses natural language processing technology to understand the meaning of text data and generate appropriate answers to user questions.

[0453] The "means for converting the generated answers into voice data" refers to a process or device that converts the text-format answers into voice data using voice synthesis technology.

[0454] The "means for transmitting voice data to a user's terminal" refers to a function or protocol for transferring the generated voice data to a user's terminal via a communication network.

[0455] "Means for reproducing audio data at a terminal" refers to a function or device for reproducing audio data received by a user's terminal and audibly conveying the data to the user.

[0456] A "natural language processing model" is a trained algorithm or framework for understanding the meaning of text data and generating appropriate answers based on that understanding.

[0457] A "communications protocol" is a standardized procedure or rule for sending and receiving data over a network.

[0458] This invention is a system that realizes a "consultation app" that responds to the everyday worries and questions of elderly users. This system features no codes, no prompts, and no text input, allowing users to obtain information easily, quickly, and accurately using only voice input.

[0459] User voice input

[0460] The user speaks a question into the device. For example, they can ask a specific question such as, "My legs have been hurting lately. What should I do?" The user does not need to enter any special text, and can ask questions using only voice input.

[0461] Recording audio on your device

[0462] The device records the user's voice using the built-in microphone. The device uses a voice recorder application built into the device's operating system. When the user finishes answering a question, the recording stops automatically and the voice data is generated.

[0463] Sending voice data to the server

[0464] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket, where it is converted into an appropriate format (e.g., WAV or MP3) and uploaded to a specified endpoint on the server.

[0465] Server-based speech recognition

[0466] The server inputs the received voice data into a voice recognition engine. The voice recognition engine used can be a general-purpose voice recognition API (for example, Google Speech-to-Text API or IBM Watson Speech to Text). This converts the voice data into text data, and specific text data such as "My leg has been hurting lately. What should I do?" is obtained.

[0467] Server-based natural language analysis

[0468] The server sends the text data to a natural language processing engine, such as OpenAI's GPT-3 or Google Natural Language API, to analyze the question, allowing the server to generate an appropriate answer to the user's question.

[0469] Speech conversion and transmission of text data

[0470] The generated text data of the answer is sent by the server to a speech synthesis engine. Examples of speech synthesis engines used include Google Text-to-Speech API and Amazon Polly. For example, specific speech data such as "If your leg hurts, it's important to first rest and ice it. If the pain persists, consult a doctor" is generated. The server then sends this speech data to the device.

[0471] Playing audio on your device

[0472] The device receives the audio data sent from the server and plays it back using a built-in speaker, providing the user with specific advice such as, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor."

[0473] Specific examples

[0474] For example, if a user asks a question by voice, such as "My legs have been hurting lately, what should I do?", the system operates as follows: First, the device records the voice and sends the data to the server. Next, the server converts the voice data into text, analyzes the text data, and generates an appropriate answer. Finally, the generated answer is converted into voice data, sent to the device, and played back to the user.

[0475] In this way, users can easily get answers to their concerns and questions through voice input. Also, since it is targeted at elderly users, the series of operations is designed to be intuitive and easy to perform.

[0476] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0477] Step 1:

[0478] The user speaks a question into the terminal. For example, "My legs have been hurting lately. What should I do?" The input is the user's voice, and the output is the voice data recorded by the terminal.

[0479] Step 2:

[0480] The device uses a built-in microphone to record the user's voice and save it as digital audio data. Specifically, it uses a voice recorder application built into the device's operating system and saves it as an audio file in wav or mp3 format. The input is the user's voice and the output is an audio file.

[0481] Step 3:

[0482] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket. Specifically, the device converts the audio data into an appropriate format and uploads it to a specified endpoint on the server. The input is the audio file, and the output is the audio data sent to the server.

[0483] Step 4:

[0484] The server inputs the received voice data into a voice recognition engine. The voice recognition engine used can be a general voice recognition API (for example, Google Speech-to-Text API). The input is voice data, and the output is converted text data. Specifically, the voice data is sent to the voice recognition API, and text data such as "My leg has been hurting lately, what should I do?" is obtained.

[0485] Step 5:

[0486] The server sends the obtained text data to a natural language processing engine, which analyzes the question. The natural language processing engine uses OpenAI's GPT-3 or equivalent natural language processing technology. The input is text data, and the output is the analyzed answer text. Specifically, the text data is sent to the natural language processing engine, and the answer obtained is, "If your leg hurts, it is important to first rest and ice it. If the pain persists, consult a doctor."

[0487] Step 6:

[0488] The server sends the generated answer text to a speech synthesis engine and converts it into audio data. Examples of speech synthesis engines that can be used include Google Text-to-Speech API and Amazon Polly. The input is the answer text and the output is audio data. Specifically, the answer text is sent to the speech synthesis engine, which generates an audio file (e.g., mp3 format).

[0489] Step 7:

[0490] The server sends the generated audio data to the device. The transmission method uses a communication protocol (HTTP or WebSocket). The input is the audio data, and the output is the audio data sent to the device.

[0491] Step 8:

[0492] The device plays the received voice data. Specifically, it plays the voice data using the device's speaker and provides the answer in an audible format to the user. The input is the voice data, and the output is the voice answer presented to the user.

[0493] (Application example 1)

[0494] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0495] There is a demand for systems that can easily resolve questions and problems that elderly people and users unfamiliar with technology have in their daily lives and while shopping. In particular, physical stores such as supermarkets lack easy ways to obtain information about products and sales areas. Conventional systems that require text input or complex operations are difficult for elderly people to use, so there is a need to develop systems that can be used intuitively with only voice input.

[0496] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0497] In this invention, the server includes a means for converting speech into text data, a means for generating appropriate responses using a generative AI model, and a means for inputting prompt sentences, which enables elderly people and users who are unfamiliar with technology to easily obtain product locations and information in a store using only voice input.

[0498] "User" means a person who uses the system.

[0499] "Means for acquiring audio" refers to devices or software that record the audio emitted by the user and store it as digital data.

[0500] "Server" refers to a computer system for receiving and processing audio data.

[0501] "Means for converting into text data" refers to the technology and software that converts audio data into text information.

[0502] "Means of analysis and generating appropriate answers" refers to the technologies and algorithms used to analyze text data and create optimal answers to users' questions.

[0503] "Means for converting generated answers into audio data" refers to technology or software that converts text-based answers into data that can be played as audio.

[0504] "Terminal" refers to a device (smartphone, smart glasses, etc.) that is directly operated by the user and used to input and output audio.

[0505] "Means for playback" refers to the function for outputting audio data received by the terminal as audio through a speaker or the like.

[0506] "A means of support for users to ask about product locations and information by voice in a store" refers to an auxiliary function that allows users to ask questions about specific products or information in a physical store.

[0507] "Generative AI model" refers to artificial intelligence technology used to generate appropriate answers to user questions.

[0508] "Means for inputting prompt sentences" refers to the technology or method for inputting sentences or data that give operational instructions to the generative AI model.

[0509] "Communication protocol" refers to the rules and procedures for transmitting digital information.

[0510] "Natural language processing engine" refers to technologies and algorithms for understanding, processing, and generating natural language text.

[0511] The system of this invention aims to enable elderly people and users unfamiliar with technology to easily obtain information about products and sales areas in brick-and-mortar stores such as supermarkets using only their voice. The specific system configuration and operating procedure are shown below.

[0512] System configuration

[0513] Device: A smartphone or smart glasses held by a user, used to input voice and play audio output.

[0514] Server: Responsible for processing voice data. Has functions such as speech recognition, natural language processing, and answer generation using generative AI models.

[0515] Hardware and software used

[0516] Speech recognition engine: Used to convert the user's voice into text data.

[0517] Natural language processing engine: An algorithm that analyzes text data and generates appropriate answers using generative AI models.

[0518] Generative AI model: An artificial intelligence technique for generating optimal answers to user questions.

[0519] Speech synthesis engine: Used to convert the generated text response into audio data.

[0520] Communication protocol: A communication rule such as HTTP or WebSocket for sending and receiving data between a terminal and a server.

[0521] Program processing explanation

[0522] 1. User voice input: The user voices a question into the device, for example, "Where are the tomatoes?"

[0523] 2. Voice recording on the device: The device records the user's voice and generates the voice data.

[0524] 3. Sending audio data to the server: The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket.

[0525] 4. Speech recognition on the server: The server inputs the received voice data into a voice recognition engine and converts the voice into text data. For example, the resulting text data is "Where are the tomatoes?"

[0526] 5. Natural language analysis on the server: The server sends the obtained text data to a natural language processing engine, which analyzes the question, understands the intent of the question, and generates an appropriate answer using a generative AI model.

[0527] 6. Text-to-speech conversion and transmission: The generated text answer is converted into voice data by a speech synthesis engine. For example, voice data such as "The tomatoes are in aisle 3" is generated, and the server transmits this voice data to the device.

[0528] 7. Playback on the device: The device receives the audio data sent from the server and plays it back, telling the user the answer: "The tomatoes are in aisle 3."

[0529] Specific examples

[0530] If a user asks a terminal, "Where are the tomatoes?", the system will do the following:

[0531] Speak: "Where are the tomatoes?"

[0532] Request to server: "Where are the tomatoes?"

[0533] Server responds: "Tomatoes are in aisle 3."

[0534] Speech synthesis and playback: The audio response "The tomatoes are in aisle 3" is played.

[0535] Prompt Sentence Examples

[0536] "If a user asks, 'Where are the tomatoes?' generate the right answer."

[0537] This system will enable even elderly people and those unfamiliar with technology to easily obtain information using voice within the supermarket, greatly improving convenience.

[0538] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0539] Step 1:

[0540] The user speaks a question into the device. For example, "Where are the tomatoes?" The input for this step is the user's voice, and the output is the voice data recorded by the device. Specifically, the device's microphone captures the voice and records it in digital format.

[0541] Step 2:

[0542] The device records the user's voice and generates voice data. The generated voice data is saved as a digital file. The input for this step is the user's voice, and the output is the recorded voice data. Specifically, the data is saved in a format required for input to the voice recognition engine.

[0543] Step 3:

[0544] The device sends the recorded audio data to the server. The input of this step is the audio data, and the output is the audio data sent to the server. Specifically, the device sends the audio data to the server using a communication protocol such as HTTP or WebSocket.

[0545] Step 4:

[0546] The server inputs the received voice data into a voice recognition engine and converts the voice into text data. The input for this step is the voice data sent to the server, and the output is the converted text data. Specifically, the voice recognition engine analyzes the sound wave data and generates the text "Where are the tomatoes?"

[0547] Step 5:

[0548] The server sends the obtained text data to a natural language processing engine, which analyzes the user's question. It understands the intent of the question and generates an appropriate answer using a generative AI model. The input to this step is text data, and the output is the text data of the generated answer. Specifically, the natural language processing engine analyzes the text and provides a prompt sentence to the generative AI model to generate an answer.

[0549] Step 6:

[0550] The server converts the generated text data of the answer into voice data using a voice synthesis engine. The input of this step is the generated text data of the answer, and the output is the converted voice data. Specifically, the voice synthesis engine generates the voice saying, "The tomatoes are in aisle 3."

[0551] Step 7:

[0552] The server sends the generated voice data to the user's terminal. The input of this step is the voice data, and the output is the voice data sent to the terminal. In specific operations, the server sends the voice data to the terminal using a communication protocol.

[0553] Step 8:

[0554] The device receives the voice data sent from the server and plays it back. The input of this step is the voice data sent to the device, and the output is the voice response played back to the user. Specifically, the device's speaker plays back the voice, "The tomatoes are in aisle 3."

[0555] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0556] This invention is a system that realizes a "consultation app" that responds to the daily worries and questions of elderly users, and features no codes, no prompts, and no text input. This system allows users to easily, quickly, and accurately obtain information using only voice input. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide answers that correspond to the user's emotional state.

[0557] An embodiment of the system operates as follows.

[0558] User voice input

[0559] The user speaks a question into the device, which then records the user's voice. For example, the user might ask, "My legs have been hurting lately. What should I do?"

[0560] Recording audio on your device

[0561] The terminal has a function for recording the user's voice, records the user's question, and generates the voice data.

[0562] Sending voice data to the server

[0563] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket.

[0564] Server-based speech recognition

[0565] The server inputs the received voice data into a voice recognition engine and converts it into text data. For example, the resulting text data is "My leg has been hurting lately. What should I do?"

[0566] Emotion recognition on the server

[0567] The server sends the acquired voice and text data to the emotion engine to recognize the user's emotion. The emotion engine analyzes the tone, speed, and intensity of the voice to recognize the user's emotional state.

[0568] Server-based natural language analysis

[0569] The server sends the emotion data and text data obtained from the emotion engine to the natural language processing engine, which analyzes the text data, understands the meaning of the user's question, and generates an appropriate answer taking the emotion data into account.

[0570] Speech conversion and transmission of text data

[0571] The generated text data of the answer is converted into voice data by a speech synthesis engine. For example, voice data with the content "If your leg hurts, it is important to first rest and cool it. If the pain persists, consult a doctor" is generated. The server sends this voice data to the device.

[0572] Playing audio on your device

[0573] The device receives the audio data sent from the server and plays it back, telling the user, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor."

[0574] As a concrete example, if a user asks the terminal, "My legs have been hurting lately. What should I do?", the system will operate as follows.

[0575] 1. The user provides voice input.

[0576] 2. The device records the audio and sends the data to the server.

[0577] 3. The server converts the audio data into text.

[0578] 4. The server analyzes the text and voice data using an emotion engine to recognize the user's emotions.

[0579] 5. The server analyzes the text data and generates an appropriate answer.

[0580] 6. The server converts the answer into audio data and sends it to the device.

[0581] 7. The device plays the audio data and tells the user the answer.

[0582] In this way, users can easily get answers to their concerns and questions through voice input. Since the app is targeted at elderly users, the entire process is designed to be intuitive and easy to follow. Furthermore, the introduction of an emotion engine makes it possible to respond in accordance with the user's emotions, providing more personalized support.

[0583] The processing flow will be explained below.

[0584] Step 1:

[0585] The user speaks a question into the device, which then records the user's voice. For example, the user might ask, "My legs have been hurting lately. What should I do?"

[0586] Step 2:

[0587] The device records the user's voice and prepares the recording, which is then saved as an audio file.

[0588] Step 3:

[0589] The device sends the recorded audio data to the server using the HTTP or WebSocket communication protocol.

[0590] Step 4:

[0591] The server inputs the received voice data into a voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. For example, the generated text data is "My leg has been hurting lately. What should I do?"

[0592] Step 5:

[0593] The server inputs the acquired voice data into the emotion engine, which analyzes the tone, speed, and intensity of the voice to recognize the user's emotional state. For example, if the voice tone is low and the speed is slow, it will recognize that the user is feeling anxious.

[0594] Step 6:

[0595] The server sends the emotion data and text data obtained from the emotion engine to the natural language processing engine, which analyzes the text data and understands the meaning of the user's question.

[0596] Step 7:

[0597] Based on the analysis results, the server's natural language processing engine takes emotional data into consideration and generates an appropriate answer. For example, a text answer such as "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor" is generated.

[0598] Step 8:

[0599] The server generates a text response, which is then input to a speech synthesis engine and converted into voice data. The speech synthesis engine then converts the text response into voice data.

[0600] Step 9:

[0601] The server transmits the generated voice data to the terminal, and the voice data is transmitted using a communication protocol.

[0602] Step 10:

[0603] The terminal receives the voice data sent from the server, and then acquires and prepares the voice data.

[0604] Step 11:

[0605] The device plays back the audio data and gives the user a response in voice. Specifically, it says, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor." Furthermore, if it recognizes that the user is feeling anxious, it plays back the audio in a calmer tone.

[0606] Example 2

[0607] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0608] In modern society, there is a need for prompt and accurate responses to the worries and questions that elderly people have in their daily lives. However, text input and complex operations are a significant burden for elderly people, which creates a barrier to obtaining information. Furthermore, a lack of responses that respond to emotional changes is one of the factors that amplify the anxiety and stress of elderly people. To solve these problems, a system that allows easy information acquisition through voice input alone and that responds with empathy is needed.

[0609] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0610] In this invention, the server includes means for acquiring a user's voice, means for transmitting the voice to the server, means for converting the voice to text data in the server, means for analyzing the text data and voice data to recognize the user's emotion, means for analyzing the text data and generating an appropriate answer, means for converting the generated answer to voice data, means for transmitting the voice data to the user's terminal, and means for playing back the voice data in the terminal. This enables elderly people to easily obtain answers to their worries and questions through voice input, and realizes responses that are sensitive to their emotions.

[0611] A "user" is a person who wishes to use the system to provide voice input and obtain information.

[0612] "Means for acquiring voice" refers to a device for collecting voice information from a user and recording it as digital data.

[0613] A "server" is a computer system that processes and analyzes voice data over a network and generates appropriate responses.

[0614] The "means for transmitting audio to a server" refers to a device or program that has the function of transferring audio data acquired by a terminal to a server using a communication protocol.

[0615] A "means for converting voice into text data" is a program or engine that uses voice recognition technology to convert the content of voice into text.

[0616] "Means for recognizing a user's emotions by analyzing text data and voice data" refers to software or an engine for determining a user's emotional state from the tone of voice and the content of text.

[0617] "Means for analyzing text data and generating appropriate answers" refers to a program or engine that uses natural language processing technology to understand the meaning of input text and generate answers based on that content.

[0618] The "means for converting the generated answer into voice data" is software or an engine with speech synthesis technology that converts text data into a format that can be played back as voice.

[0619] The "means for transmitting voice data to a user's terminal" refers to a device or program that has the function of transmitting voice data generated by a server to a user's terminal using a communication protocol.

[0620] "Means for reproducing the audio data at the terminal" refers to a device that has the function of reproducing the audio data received at the terminal in an audible format for the user.

[0621] This invention is a system that realizes a "consultation app" that responds to the daily worries and questions of elderly users, and features no codes, no prompts, and no text input. Specifically, users can easily, quickly, and accurately obtain information using only voice input. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide answers that correspond to the user's emotional state.

[0622] User voice input

[0623] The user speaks their question into the device. The device is equipped with a built-in microphone, and the user can start voice input by pressing the voice input button. For example, they might ask, "My leg has been hurting lately. What should I do?"

[0624] Recording audio on your device

[0625] The terminal has a function to record the user's voice, and uses a built-in microphone to record the user's questions and generate the voice data. The recording module converts the input voice into digital data.

[0626] Sending voice data to the server

[0627] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket. For example, the audio data can be sent to the server using an HTTP POST request.

[0628] Server-based speech recognition

[0629] The server inputs the received voice data into a voice recognition engine and converts it into text data. The voice recognition engine uses, for example, the Google Speech-to-Text API. The resulting text data is, "My leg has been hurting lately. What should I do?"

[0630] Emotion recognition on the server

[0631] The server sends the acquired voice and text data to an emotion engine, such as IBM Watson Tone Analyzer, which analyzes the tone, speed, and intensity of the voice to determine whether the user is "confused."

[0632] Server-based natural language analysis

[0633] The server sends the emotion data obtained from the emotion engine and the recognized text data to a natural language processing engine. For example, OpenAI GPT-3 is used for the natural language processing engine. This analyzes the text data, understands the meaning of the user's question, and generates an appropriate answer taking into account the emotion data. The answer it generates is, "If your leg hurts, it's important to first rest and apply ice. If the pain persists, consult a doctor."

[0634] Speech conversion and transmission of text data

[0635] The generated text data of the answer is converted into voice data by a speech synthesis engine. For example, Amazon Polly is used as the speech synthesis engine. This generates voice data that says, "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor." The server then sends this voice data to the device via an HTTP POST request.

[0636] Playing audio on your device

[0637] The device receives the voice data sent from the server and plays it back. Specifically, it tells the user, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor." The device with a speaker also plays back the voice.

[0638] Specific examples

[0639] When a user asks the terminal, "My legs have been hurting lately. What should I do?", the system operates as follows.

[0640] 1. The user provides voice input.

[0641] 2. The device records the audio and sends the data to the server.

[0642] 3. The server converts the audio data into text.

[0643] 4. The server analyzes the text and voice data using an emotion engine to recognize the user's emotions.

[0644] 5. The server analyzes the text data and generates an appropriate answer.

[0645] 6. The server converts the answer into audio data and sends it to the device.

[0646] 7. The device plays the audio data and tells the user the answer.

[0647] This system allows users to easily obtain answers to their concerns and questions through voice input. As it is targeted at elderly users, the system is designed to be intuitive and easy to operate. Furthermore, the introduction of an emotion engine makes it possible to respond in accordance with the user's emotions, providing more personalized support. This system is designed to be particularly easy to use for the elderly, and is capable of responding appropriately through emotion recognition, which is expected to increase user satisfaction.

[0648] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0649] Step 1: User Speech Input

[0650] The user speaks a question into the device. For example, if the question is, "My legs have been hurting lately, what should I do?", this voice becomes the input for the system. The user starts voice input by pressing the voice input button on the device and speaking.

[0651] Step 2: Record audio on your device

[0652] The device records the user's voice. It uses the built-in microphone to capture the user's question and generates the voice data. This generated voice data becomes the input for the next process. Specifically, the recording module generates the voice data, "My legs have been hurting lately. What should I do?"

[0653] Step 3: Sending audio data to the server

[0654] The device sends the recorded audio data to the server. A communication protocol such as HTTP or WebSocket is used for transmission. This input audio data is transferred to the server via an HTTP POST request. The audio data sent to the server becomes the input for the next process.

[0655] Step 4: Speech recognition on the server

[0656] The server inputs the received voice data into a voice recognition engine and converts it into text data. The voice recognition engine uses the Google Speech-to-Text API or similar. The generated text data, "My leg has been hurting lately, what should I do?", becomes the input for the next process.

[0657] Step 5: Emotion Recognition on the Server

[0658] The server sends the obtained voice and text data to an emotion engine to recognize the user's emotions. The emotion engine uses a system such as IBM Watson Tone Analyzer. It analyzes the tone, speed, and intensity of the voice and determines that the user is "confused." This emotion data and text data become the input for the next process.

[0659] Step 6: Natural Language Analysis on the Server

[0660] The server sends the emotion data obtained from the emotion engine and the recognized text data to a natural language processing engine. The natural language processing engine uses OpenAI GPT-3 and other technologies. This analyzes the text data and generates a response text such as, "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor." The generated response text becomes the input for the next process.

[0661] Step 7: Convert text to speech and send it

[0662] The generated text response is converted into voice data by a speech synthesis engine. Amazon Polly or another similar engine is used as the voice synthesis engine. Voice data is generated that says, "If your leg hurts, it's important to rest and cool it. If the pain persists, consult a doctor." This voice data becomes the input for the next process and is sent to the device via an HTTP POST request.

[0663] Step 8: Play audio on your device

[0664] The device receives the voice data sent from the server and plays it back. Specifically, the device's speaker plays back the following voice: "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor." This voice playback is the system's final output, allowing the user to obtain an appropriate answer.

[0665] This system allows users to easily obtain answers to their concerns and questions through voice input. Since it is targeted at elderly users, the operation is intuitive and easy. Furthermore, the introduction of an emotion engine makes it possible to respond in accordance with the user's emotions, providing more personalized support.

[0666] (Application example 2)

[0667] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0668] In recent years, when elderly users use food delivery services in their daily lives, they are faced with a lack of diverse options and suggestions that take into account their health status. Furthermore, elderly users are often unfamiliar with the operation of these services, and certain emotional states can make it difficult for them to make appropriate choices. Furthermore, while intuitive operation based on voice input is required, existing systems do not take emotion recognition into account, which can result in low user satisfaction.

[0669] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a user's voice, means for transmitting the voice to the server, means for converting the voice to text data in the server, means for analyzing the text data and generating an appropriate response, means for converting the generated response to voice data, means for transmitting the voice data to the user's terminal, means for playing the voice data in the terminal, means for recognizing the user's emotion, and means for generating a response based on the emotion. This enables elderly users to intuitively and easily use food delivery services and receive appropriate suggestions while taking their emotions into consideration.

[0670] The "means for acquiring the user's voice" refers to a combination of hardware and software for recognizing and acquiring the voice uttered by the user via a device.

[0671] The "means for transmitting the voice to the server" refers to a communication protocol and software function for transmitting the acquired voice data to the server via a network.

[0672] The "means for converting the voice into text data in the server" is a function for converting voice data into text data using a voice recognition engine in the server.

[0673] The "means for analyzing the text data and generating an appropriate response" refers to the algorithm and software functions that analyze the text data using a natural language processing engine and generate a response appropriate to the user's intention or question.

[0674] The "means for converting the generated answer into voice data" is a function for converting the generated text-format answer into voice data using a voice synthesis engine.

[0675] The "means for transmitting the voice data to the user's terminal" refers to a communication protocol and software function for transmitting the generated voice data to the user's terminal via a network.

[0676] The "means for reproducing the audio data at the terminal" refers to the hardware and software functions for reproducing the audio data received at the terminal and allowing the user to listen to it.

[0677] The "means for recognizing the user's emotions" is a function that analyzes acquired voice and text data and identifies the user's emotional state using an emotion engine.

[0678] The "means for generating an answer based on emotion" is a function of the natural language processing engine that takes into account the recognized emotional state of the user to generate the most appropriate answer.

[0679] This invention is a "smart food assistant" system that allows elderly people to use food delivery services easily and intuitively. The system uses emotion recognition based on the user's voice input to provide optimal meal suggestions.

[0680] System configuration

[0681] Hardware

[0682] 1. User device: Use a device that allows voice input, such as a smartphone or smart speaker.

[0683] 2. Server: A server equipped with a high-performance processor performs speech recognition, natural language processing, and emotion recognition.

[0684] software

[0685] 1. Speech recognition engine: Used to convert the user's voice into text data. Common examples include Google's speech recognition API.

[0686] 2. Natural Language Processing Engine (NLP Engine): Used to analyze text data using generative AI models and generate appropriate suggestions.

[0687] 3. Emotion Recognition Engine: Identifies the user's emotional state from voice and text data. Transformers, a Python library, is available.

[0688] 4. Speech synthesis engine: Used to convert the generated suggestions into audio data. gTTS is an example of this.

[0689] Processing flow

[0690] Generating a Program

[0691] The program consists of the following steps:

[0692] 1. Voice input acceptance: When a user voice-over asks questions or asks questions about food delivery, the device captures and records the voice.

[0693] 2. Audio data transmission: The acquired audio data is sent to the server via the Internet using HTTP or WebSocket as the communication protocol.

[0694] 3. Speech recognition and text conversion: The server uses a speech recognition engine to convert the voice data into text data.

[0695] 4. Emotion recognition: Using the converted text data and the original audio data, the emotion recognition engine analyzes the user's emotions.

[0696] 5. Natural Language Processing and Suggestion Generation: The NLP engine analyzes the user's intent and sentiment to generate optimal meal suggestions.

[0697] 6. Speech synthesis: Converting the generated suggestions into audio data.

[0698] 7. Sending and playing back audio data: The server sends the generated audio data to the terminal, and the terminal plays back the audio data to convey the suggestion to the user.

[0699] Specific examples

[0700] Example 1: Elderly users have trouble deciding what to eat

[0701] User says: "I'm not sure what to have for dinner tonight. Can you recommend something?"

[0702] The system responds: "The weather is nice today, how about a fresh vegetable salad?" (If the user seems a little depressed: "I recommend a meat dish to energize you today.")

[0703] Example prompt

[0704] "I'm not sure what to have for dinner tonight. What do you recommend?" -> An older user says they're having trouble deciding what to eat. They seem a little depressed. Take their emotions into consideration and offer healthy, uplifting meal suggestions.

[0705] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0706] Step 1:

[0707] Voice input reception

[0708] The user speaks, "I'm not sure what to have for dinner tonight. What do you recommend?" The device records the speech. The recorded speech data undergoes pre-processing before being sent to the speech recognition engine. Specifically, digital signal processing is performed to change the sampling rate of the speech data to an appropriate value and remove noise.

[0709] Input: User's voice

[0710] Output: Preprocessed audio data

[0711] Step 2:

[0712] Audio data transmission

[0713] After preprocessing, the audio data is sent from the device to the server using HTTP or WebSocket communication protocols. The audio data is split into network packets and securely transmitted using a secure transport layer.

[0714] Input: Preprocessed audio data

[0715] Output: Audio data sent to the server

[0716] Step 3:

[0717] Speech recognition and text conversion

[0718] The server inputs the received voice data into a voice recognition engine, which converts the voice data into text data. Here, features are extracted from the voice data and a deep learning model is used to convert the voice into text. As a specific example, the generated text is, "I'm not sure what to have for dinner tonight. What do you recommend?"

[0719] Input: Audio data received by the server

[0720] Output: Text data (e.g., "I'm not sure what to have for dinner tonight. What do you recommend?")

[0721] Step 4:

[0722] emotion recognition

[0723] The server inputs the converted text data and the original voice data into an emotion recognition engine to analyze the user's emotions. The emotion engine analyzes the tone, speed, and intensity of the voice and the content of the text to identify the user's emotional state. For example, it may recognize that the user is depressed.

[0724] Input: Text and audio data

[0725] Output: Emotion data (e.g., depressed)

[0726] Step 5:

[0727] Natural Language Processing and Suggestion Generation

[0728] The server inputs the emotion data and text data into a natural language processing engine (NLP engine) and generates suggestions based on the user's intentions and emotions. The NLP engine uses a generative AI model to generate suggestions such as "I'm not sure what to have for dinner tonight" and "I'm feeling down," based on the information that the user is unsure about what to have for dinner tonight.

[0729] Input: Text data and emotion data

[0730] Output: Suggestion text (e.g. "Today, I recommend a meat dish to energize you.")

[0731] Step 6:

[0732] Speech synthesis

[0733] The server inputs the generated suggested text into a speech synthesis engine and converts it into voice data. The speech synthesis engine converts the text into natural language and outputs it as an audio file. Specifically, software such as "gTTS" is used to convert the suggested text into an audio file.

[0734] Input: Suggested text

[0735] Output: Speech data (e.g., "I recommend a meat dish to energize you today.")

[0736] Step 7:

[0737] Sending and playing audio data

[0738] The server sends the generated audio data to the user's device using HTTP or WebSocket communication protocols. The device plays the received audio data and provides suggestions to the user by decoding the audio data and playing it on the speaker.

[0739] Input: Audio data sent from the server

[0740] Output: Speech played to the user (e.g., "Today, I recommend a meat dish to energize you.")

[0741] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0742] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0743] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0744] [Third embodiment]

[0745] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0746] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0747] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0748] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0749] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0750] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0751] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0752] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0753] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0754] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0755] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0756] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0757] This invention is a system that realizes a "anything-to-anything consultation app" that responds to the everyday worries and questions of elderly users, and features no codes, no prompts, and no text input. This system allows users to obtain information easily, quickly, and accurately using only voice input.

[0758] An embodiment of the system operates as follows.

[0759] User voice input

[0760] The user speaks a question into the device. For example, "My legs have been hurting lately. What should I do?"

[0761] Recording audio on your device

[0762] The terminal has a function for recording the user's voice, records the user's question, and generates the voice data.

[0763] Sending voice data to the server

[0764] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket.

[0765] Server-based speech recognition

[0766] The server inputs the received voice data into a voice recognition engine and converts it into text data. For example, the resulting text data is "My leg has been hurting lately. What should I do?"

[0767] Server-based natural language analysis

[0768] The server sends the obtained text data to a natural language processing engine, which analyzes the user's question, understands the intent of the question, and generates an appropriate answer.

[0769] Speech conversion and transmission of text data

[0770] The generated text data of the answer is converted into voice data by a speech synthesis engine. For example, voice data with the content "If your leg hurts, it is important to first rest and cool it. If the pain persists, consult a doctor" is generated. The server sends this voice data to the device.

[0771] Playing audio on your device

[0772] The device receives the audio data sent from the server and plays it back, telling the user, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor."

[0773] As a concrete example, if a user asks the terminal, "My legs have been hurting lately. What should I do?", the system will operate as follows.

[0774] 1. The user provides voice input.

[0775] 2. The device records the audio and sends the data to the server.

[0776] 3. The server converts the audio data into text.

[0777] 4. The server analyzes the text data and generates an appropriate answer.

[0778] 5. The server converts the answer into audio data and sends it to the device.

[0779] 6. The device plays the audio data and tells the user the answer.

[0780] In this way, users can easily get answers to their concerns and questions through voice input. Also, since it is targeted at elderly users, the series of operations is designed to be intuitive and easy to perform.

[0781] The processing flow will be explained below.

[0782] Step 1:

[0783] The user speaks a question into the device, which then records the user's voice. For example, the user might ask, "My legs have been hurting lately. What should I do?"

[0784] Step 2:

[0785] The device records the audio and prepares the recording, which is saved as an audio file.

[0786] Step 3:

[0787] The device sends the recorded audio data to the server using the HTTP or WebSocket communication protocol.

[0788] Step 4:

[0789] The server inputs the received voice data into a voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. For example, the generated text data is "My leg has been hurting lately. What should I do?"

[0790] Step 5:

[0791] The server sends the obtained text data to a natural language processing engine, which analyzes the text data and understands the meaning of the user's question.

[0792] Step 6:

[0793] The server's natural language processing engine generates an appropriate answer based on the analysis results. For example, a text data answer such as "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor."

[0794] Step 7:

[0795] The server generates a text response, which is then input to a speech synthesis engine and converted into voice data. The speech synthesis engine then converts the text response into voice data.

[0796] Step 8:

[0797] The server transmits the generated voice data to the terminal, and the voice data is transmitted using a communication protocol.

[0798] Step 9:

[0799] The terminal receives the voice data sent from the server, and then acquires and prepares the voice data.

[0800] Step 10:

[0801] The device plays back the audio data and gives the user a spoken response, specifically saying, "If your leg hurts, it's important to rest and cool it down. If the pain persists, consult a doctor."

[0802] Example 1

[0803] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0804] For users who have difficulty using keyboards or touch input, such as the elderly, there is a need to provide a simple and intuitive interface that can quickly and accurately respond to their daily concerns and questions. However, conventional systems are complicated to operate, and the variety of input methods results in a high learning cost. To solve this problem, it is necessary to build a system that can obtain information using only voice input.

[0805] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0806] In this invention, the server includes a means for acquiring the user's voice, a means for converting the recorded voice data into text data, and a means for analyzing the text data and generating appropriate answers, thereby enabling users to quickly obtain answers to their questions and concerns by simply inputting their voice.

[0807] "Means for acquiring user's voice" refers to a device or function that physically detects the voice spoken by the user and acquires it as digital data.

[0808] The "recording terminal means" refers to a terminal or device that electronically records acquired voice and saves it as voice data.

[0809] The "means for transmitting recorded voice data to a server" refers to a function or protocol for transferring voice data recorded on a terminal to a server via a communication network.

[0810] "Means for converting voice data into text data on a server" refers to a process or system that uses voice recognition technology to convert voice data into text information.

[0811] The "means for analyzing text data and generating appropriate answers" is a function that uses natural language processing technology to understand the meaning of text data and generate appropriate answers to user questions.

[0812] The "means for converting the generated answers into voice data" refers to a process or device that converts the text-format answers into voice data using voice synthesis technology.

[0813] The "means for transmitting voice data to a user's terminal" refers to a function or protocol for transferring the generated voice data to a user's terminal via a communication network.

[0814] "Means for reproducing audio data at a terminal" refers to a function or device for reproducing audio data received by a user's terminal and audibly conveying the data to the user.

[0815] A "natural language processing model" is a trained algorithm or framework for understanding the meaning of text data and generating appropriate answers based on that understanding.

[0816] A "communications protocol" is a standardized procedure or rule for sending and receiving data over a network.

[0817] This invention is a system that realizes a "consultation app" that responds to the everyday worries and questions of elderly users. This system features no codes, no prompts, and no text input, allowing users to obtain information easily, quickly, and accurately using only voice input.

[0818] User voice input

[0819] The user speaks a question into the device. For example, they can ask a specific question such as, "My legs have been hurting lately. What should I do?" The user does not need to enter any special text, and can ask questions using only voice input.

[0820] Recording audio on your device

[0821] The device records the user's voice using the built-in microphone. The device uses a voice recorder application built into the device's operating system. When the user finishes answering a question, the recording stops automatically and the voice data is generated.

[0822] Sending voice data to the server

[0823] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket, where it is converted into an appropriate format (e.g., WAV or MP3) and uploaded to a specified endpoint on the server.

[0824] Server-based speech recognition

[0825] The server inputs the received voice data into a voice recognition engine. The voice recognition engine used can be a general-purpose voice recognition API (for example, Google Speech-to-Text API or IBM Watson Speech to Text). This converts the voice data into text data, and specific text data such as "My leg has been hurting lately. What should I do?" is obtained.

[0826] Server-based natural language analysis

[0827] The server sends the text data to a natural language processing engine, such as OpenAI's GPT-3 or Google Natural Language API, to analyze the question, allowing the server to generate an appropriate answer to the user's question.

[0828] Speech conversion and transmission of text data

[0829] The generated text data of the answer is sent by the server to a speech synthesis engine. Examples of speech synthesis engines used include Google Text-to-Speech API and Amazon Polly. For example, specific speech data such as "If your leg hurts, it's important to first rest and ice it. If the pain persists, consult a doctor" is generated. The server then sends this speech data to the device.

[0830] Playing audio on your device

[0831] The device receives the audio data sent from the server and plays it back using a built-in speaker, providing the user with specific advice such as, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor."

[0832] Specific examples

[0833] For example, if a user asks a question by voice, such as "My legs have been hurting lately, what should I do?", the system operates as follows: First, the device records the voice and sends the data to the server. Next, the server converts the voice data into text, analyzes the text data, and generates an appropriate answer. Finally, the generated answer is converted into voice data, sent to the device, and played back to the user.

[0834] In this way, users can easily get answers to their concerns and questions through voice input. Also, since it is targeted at elderly users, the series of operations is designed to be intuitive and easy to perform.

[0835] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0836] Step 1:

[0837] The user speaks a question into the terminal. For example, "My legs have been hurting lately. What should I do?" The input is the user's voice, and the output is the voice data recorded by the terminal.

[0838] Step 2:

[0839] The device uses a built-in microphone to record the user's voice and save it as digital audio data. Specifically, it uses a voice recorder application built into the device's operating system and saves it as an audio file in wav or mp3 format. The input is the user's voice and the output is an audio file.

[0840] Step 3:

[0841] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket. Specifically, the device converts the audio data into an appropriate format and uploads it to a specified endpoint on the server. The input is the audio file, and the output is the audio data sent to the server.

[0842] Step 4:

[0843] The server inputs the received voice data into a voice recognition engine. The voice recognition engine used can be a general voice recognition API (for example, Google Speech-to-Text API). The input is voice data, and the output is converted text data. Specifically, the voice data is sent to the voice recognition API, and text data such as "My leg has been hurting lately, what should I do?" is obtained.

[0844] Step 5:

[0845] The server sends the obtained text data to a natural language processing engine, which analyzes the question. The natural language processing engine uses OpenAI's GPT-3 or equivalent natural language processing technology. The input is text data, and the output is the analyzed answer text. Specifically, the text data is sent to the natural language processing engine, and the answer obtained is, "If your leg hurts, it is important to first rest and ice it. If the pain persists, consult a doctor."

[0846] Step 6:

[0847] The server sends the generated answer text to a speech synthesis engine and converts it into audio data. Examples of speech synthesis engines that can be used include Google Text-to-Speech API and Amazon Polly. The input is the answer text and the output is audio data. Specifically, the answer text is sent to the speech synthesis engine, which generates an audio file (e.g., mp3 format).

[0848] Step 7:

[0849] The server sends the generated audio data to the device. The transmission method uses a communication protocol (HTTP or WebSocket). The input is the audio data, and the output is the audio data sent to the device.

[0850] Step 8:

[0851] The device plays the received voice data. Specifically, it plays the voice data using the device's speaker and provides the answer in an audible format to the user. The input is the voice data, and the output is the voice answer presented to the user.

[0852] (Application example 1)

[0853] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0854] There is a demand for systems that can easily resolve questions and problems that elderly people and users unfamiliar with technology have in their daily lives and while shopping. In particular, physical stores such as supermarkets lack easy ways to obtain information about products and sales areas. Conventional systems that require text input or complex operations are difficult for elderly people to use, so there is a need to develop systems that can be used intuitively with only voice input.

[0855] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0856] In this invention, the server includes a means for converting speech into text data, a means for generating appropriate responses using a generative AI model, and a means for inputting prompt sentences, which enables elderly people and users who are unfamiliar with technology to easily obtain product locations and information in a store using only voice input.

[0857] "User" means a person who uses the system.

[0858] "Means for acquiring audio" refers to devices or software that record the audio emitted by the user and store it as digital data.

[0859] "Server" refers to a computer system for receiving and processing audio data.

[0860] "Means for converting into text data" refers to the technology and software that converts audio data into text information.

[0861] "Means of analysis and generating appropriate answers" refers to the technologies and algorithms used to analyze text data and create optimal answers to users' questions.

[0862] "Means for converting generated answers into audio data" refers to technology or software that converts text-based answers into data that can be played as audio.

[0863] "Terminal" refers to a device (smartphone, smart glasses, etc.) that is directly operated by the user and used to input and output audio.

[0864] "Means for playback" refers to the function for outputting audio data received by the terminal as audio through a speaker or the like.

[0865] "A means of support for users to ask about product locations and information by voice in a store" refers to an auxiliary function that allows users to ask questions about specific products or information in a physical store.

[0866] "Generative AI model" refers to artificial intelligence technology used to generate appropriate answers to user questions.

[0867] "Means for inputting prompt sentences" refers to the technology or method for inputting sentences or data that give operational instructions to the generative AI model.

[0868] "Communication protocol" refers to the rules and procedures for transmitting digital information.

[0869] "Natural language processing engine" refers to technologies and algorithms for understanding, processing, and generating natural language text.

[0870] The system of this invention aims to enable elderly people and users unfamiliar with technology to easily obtain information about products and sales areas in brick-and-mortar stores such as supermarkets using only their voice. The specific system configuration and operating procedure are shown below.

[0871] System configuration

[0872] Device: A smartphone or smart glasses held by a user, used to input voice and play audio output.

[0873] Server: Responsible for processing voice data. Has functions such as speech recognition, natural language processing, and answer generation using generative AI models.

[0874] Hardware and software used

[0875] Speech recognition engine: Used to convert the user's voice into text data.

[0876] Natural language processing engine: An algorithm that analyzes text data and generates appropriate answers using generative AI models.

[0877] Generative AI model: An artificial intelligence technique for generating optimal answers to user questions.

[0878] Speech synthesis engine: Used to convert the generated text response into audio data.

[0879] Communication protocol: A communication rule such as HTTP or WebSocket for sending and receiving data between a terminal and a server.

[0880] Program processing explanation

[0881] 1. User voice input: The user voices a question into the device, for example, "Where are the tomatoes?"

[0882] 2. Voice recording on the device: The device records the user's voice and generates the voice data.

[0883] 3. Sending audio data to the server: The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket.

[0884] 4. Speech recognition on the server: The server inputs the received voice data into a voice recognition engine and converts the voice into text data. For example, the resulting text data is "Where are the tomatoes?"

[0885] 5. Natural language analysis on the server: The server sends the obtained text data to a natural language processing engine, which analyzes the question, understands the intent of the question, and generates an appropriate answer using a generative AI model.

[0886] 6. Text-to-speech conversion and transmission: The generated text answer is converted into voice data by a speech synthesis engine. For example, voice data such as "The tomatoes are in aisle 3" is generated, and the server transmits this voice data to the device.

[0887] 7. Playback on the device: The device receives the audio data sent from the server and plays it back, telling the user the answer: "The tomatoes are in aisle 3."

[0888] Specific examples

[0889] If a user asks a terminal, "Where are the tomatoes?", the system will do the following:

[0890] Speak: "Where are the tomatoes?"

[0891] Request to server: "Where are the tomatoes?"

[0892] Server responds: "Tomatoes are in aisle 3."

[0893] Speech synthesis and playback: The audio response "The tomatoes are in aisle 3" is played.

[0894] Prompt Sentence Examples

[0895] "If a user asks, 'Where are the tomatoes?' generate the right answer."

[0896] This system will enable even elderly people and those unfamiliar with technology to easily obtain information using voice within the supermarket, greatly improving convenience.

[0897] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0898] Step 1:

[0899] The user speaks a question into the device. For example, "Where are the tomatoes?" The input for this step is the user's voice, and the output is the voice data recorded by the device. Specifically, the device's microphone captures the voice and records it in digital format.

[0900] Step 2:

[0901] The device records the user's voice and generates voice data. The generated voice data is saved as a digital file. The input for this step is the user's voice, and the output is the recorded voice data. Specifically, the data is saved in a format required for input to the voice recognition engine.

[0902] Step 3:

[0903] The device sends the recorded audio data to the server. The input of this step is the audio data, and the output is the audio data sent to the server. Specifically, the device sends the audio data to the server using a communication protocol such as HTTP or WebSocket.

[0904] Step 4:

[0905] The server inputs the received voice data into a voice recognition engine and converts the voice into text data. The input for this step is the voice data sent to the server, and the output is the converted text data. Specifically, the voice recognition engine analyzes the sound wave data and generates the text "Where are the tomatoes?"

[0906] Step 5:

[0907] The server sends the obtained text data to a natural language processing engine, which analyzes the user's question. It understands the intent of the question and generates an appropriate answer using a generative AI model. The input to this step is text data, and the output is the text data of the generated answer. Specifically, the natural language processing engine analyzes the text and provides a prompt sentence to the generative AI model to generate an answer.

[0908] Step 6:

[0909] The server converts the generated text data of the answer into voice data using a voice synthesis engine. The input of this step is the generated text data of the answer, and the output is the converted voice data. Specifically, the voice synthesis engine generates the voice saying, "The tomatoes are in aisle 3."

[0910] Step 7:

[0911] The server sends the generated voice data to the user's terminal. The input of this step is the voice data, and the output is the voice data sent to the terminal. In specific operations, the server sends the voice data to the terminal using a communication protocol.

[0912] Step 8:

[0913] The device receives the voice data sent from the server and plays it back. The input of this step is the voice data sent to the device, and the output is the voice response played back to the user. Specifically, the device's speaker plays back the voice, "The tomatoes are in aisle 3."

[0914] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0915] This invention is a system that realizes a "consultation app" that responds to the daily worries and questions of elderly users, and features no codes, no prompts, and no text input. This system allows users to easily, quickly, and accurately obtain information using only voice input. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide answers that correspond to the user's emotional state.

[0916] An embodiment of the system operates as follows.

[0917] User voice input

[0918] The user speaks a question into the device, which then records the user's voice. For example, the user might ask, "My legs have been hurting lately. What should I do?"

[0919] Recording audio on your device

[0920] The terminal has a function for recording the user's voice, records the user's question, and generates the voice data.

[0921] Sending voice data to the server

[0922] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket.

[0923] Server-based speech recognition

[0924] The server inputs the received voice data into a voice recognition engine and converts it into text data. For example, the resulting text data is "My leg has been hurting lately. What should I do?"

[0925] Emotion recognition on the server

[0926] The server sends the acquired voice and text data to the emotion engine to recognize the user's emotion. The emotion engine analyzes the tone, speed, and intensity of the voice to recognize the user's emotional state.

[0927] Server-based natural language analysis

[0928] The server sends the emotion data and text data obtained from the emotion engine to the natural language processing engine, which analyzes the text data, understands the meaning of the user's question, and generates an appropriate answer taking the emotion data into account.

[0929] Speech conversion and transmission of text data

[0930] The generated text data of the answer is converted into voice data by a speech synthesis engine. For example, voice data with the content "If your leg hurts, it is important to first rest and cool it. If the pain persists, consult a doctor" is generated. The server sends this voice data to the device.

[0931] Playing audio on your device

[0932] The device receives the audio data sent from the server and plays it back, telling the user, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor."

[0933] As a concrete example, if a user asks the terminal, "My legs have been hurting lately. What should I do?", the system will operate as follows.

[0934] 1. The user provides voice input.

[0935] 2. The device records the audio and sends the data to the server.

[0936] 3. The server converts the audio data into text.

[0937] 4. The server analyzes the text and voice data using an emotion engine to recognize the user's emotions.

[0938] 5. The server analyzes the text data and generates an appropriate answer.

[0939] 6. The server converts the answer into audio data and sends it to the device.

[0940] 7. The device plays the audio data and tells the user the answer.

[0941] In this way, users can easily get answers to their concerns and questions through voice input. Since the app is targeted at elderly users, the entire process is designed to be intuitive and easy to follow. Furthermore, the introduction of an emotion engine makes it possible to respond in accordance with the user's emotions, providing more personalized support.

[0942] The processing flow will be explained below.

[0943] Step 1:

[0944] The user speaks a question into the device, which then records the user's voice. For example, the user might ask, "My legs have been hurting lately. What should I do?"

[0945] Step 2:

[0946] The device records the user's voice and prepares the recording, which is then saved as an audio file.

[0947] Step 3:

[0948] The device sends the recorded audio data to the server using the HTTP or WebSocket communication protocol.

[0949] Step 4:

[0950] The server inputs the received voice data into a voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. For example, the generated text data is "My leg has been hurting lately. What should I do?"

[0951] Step 5:

[0952] The server inputs the acquired voice data into the emotion engine, which analyzes the tone, speed, and intensity of the voice to recognize the user's emotional state. For example, if the voice tone is low and the speed is slow, it will recognize that the user is feeling anxious.

[0953] Step 6:

[0954] The server sends the emotion data and text data obtained from the emotion engine to the natural language processing engine, which analyzes the text data and understands the meaning of the user's question.

[0955] Step 7:

[0956] Based on the analysis results, the server's natural language processing engine takes emotional data into consideration and generates an appropriate answer. For example, a text answer such as "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor" is generated.

[0957] Step 8:

[0958] The server generates a text response, which is then input to a speech synthesis engine and converted into voice data. The speech synthesis engine then converts the text response into voice data.

[0959] Step 9:

[0960] The server transmits the generated voice data to the terminal, and the voice data is transmitted using a communication protocol.

[0961] Step 10:

[0962] The terminal receives the voice data sent from the server, and then acquires and prepares the voice data.

[0963] Step 11:

[0964] The device plays back the audio data and gives the user a response in voice. Specifically, it says, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor." Furthermore, if it recognizes that the user is feeling anxious, it plays back the audio in a calmer tone.

[0965] Example 2

[0966] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0967] In modern society, there is a need for prompt and accurate responses to the worries and questions that elderly people have in their daily lives. However, text input and complex operations are a significant burden for elderly people, which creates a barrier to obtaining information. Furthermore, a lack of responses that respond to emotional changes is one of the factors that amplify the anxiety and stress of elderly people. To solve these problems, a system that allows easy information acquisition through voice input alone and that responds with empathy is needed.

[0968] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0969] In this invention, the server includes means for acquiring a user's voice, means for transmitting the voice to the server, means for converting the voice to text data in the server, means for analyzing the text data and voice data to recognize the user's emotion, means for analyzing the text data and generating an appropriate answer, means for converting the generated answer to voice data, means for transmitting the voice data to the user's terminal, and means for playing back the voice data in the terminal. This enables elderly people to easily obtain answers to their worries and questions through voice input, and realizes responses that are sensitive to their emotions.

[0970] A "user" is a person who wishes to use the system to provide voice input and obtain information.

[0971] "Means for acquiring voice" refers to a device for collecting voice information from a user and recording it as digital data.

[0972] A "server" is a computer system that processes and analyzes voice data over a network and generates appropriate responses.

[0973] The "means for transmitting audio to a server" refers to a device or program that has the function of transferring audio data acquired by a terminal to a server using a communication protocol.

[0974] A "means for converting voice into text data" is a program or engine that uses voice recognition technology to convert the content of voice into text.

[0975] "Means for recognizing a user's emotions by analyzing text data and voice data" refers to software or an engine for determining a user's emotional state from the tone of voice and the content of text.

[0976] "Means for analyzing text data and generating appropriate answers" refers to a program or engine that uses natural language processing technology to understand the meaning of input text and generate answers based on that content.

[0977] The "means for converting the generated answer into voice data" is software or an engine with speech synthesis technology that converts text data into a format that can be played back as voice.

[0978] The "means for transmitting voice data to a user's terminal" refers to a device or program that has the function of transmitting voice data generated by a server to a user's terminal using a communication protocol.

[0979] "Means for reproducing the audio data at the terminal" refers to a device that has the function of reproducing the audio data received at the terminal in an audible format for the user.

[0980] This invention is a system that realizes a "consultation app" that responds to the daily worries and questions of elderly users, and features no codes, no prompts, and no text input. Specifically, users can easily, quickly, and accurately obtain information using only voice input. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide answers that correspond to the user's emotional state.

[0981] User voice input

[0982] The user speaks their question into the device. The device is equipped with a built-in microphone, and the user can start voice input by pressing the voice input button. For example, they might ask, "My leg has been hurting lately. What should I do?"

[0983] Recording audio on your device

[0984] The terminal has a function to record the user's voice, and uses a built-in microphone to record the user's questions and generate the voice data. The recording module converts the input voice into digital data.

[0985] Sending voice data to the server

[0986] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket. For example, the audio data can be sent to the server using an HTTP POST request.

[0987] Server-based speech recognition

[0988] The server inputs the received voice data into a voice recognition engine and converts it into text data. The voice recognition engine uses, for example, the Google Speech-to-Text API. The resulting text data is, "My leg has been hurting lately. What should I do?"

[0989] Emotion recognition on the server

[0990] The server sends the acquired voice and text data to an emotion engine, such as IBM Watson Tone Analyzer, which analyzes the tone, speed, and intensity of the voice to determine whether the user is "confused."

[0991] Server-based natural language analysis

[0992] The server sends the emotion data obtained from the emotion engine and the recognized text data to a natural language processing engine. For example, OpenAI GPT-3 is used for the natural language processing engine. This analyzes the text data, understands the meaning of the user's question, and generates an appropriate answer taking into account the emotion data. The answer it generates is, "If your leg hurts, it's important to first rest and apply ice. If the pain persists, consult a doctor."

[0993] Speech conversion and transmission of text data

[0994] The generated text data of the answer is converted into voice data by a speech synthesis engine. For example, Amazon Polly is used as the speech synthesis engine. This generates voice data that says, "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor." The server then sends this voice data to the device via an HTTP POST request.

[0995] Playing audio on your device

[0996] The device receives the voice data sent from the server and plays it back. Specifically, it tells the user, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor." The device with a speaker also plays back the voice.

[0997] Specific examples

[0998] When a user asks the terminal, "My legs have been hurting lately. What should I do?", the system operates as follows.

[0999] 1. The user provides voice input.

[1000] 2. The device records the audio and sends the data to the server.

[1001] 3. The server converts the audio data into text.

[1002] 4. The server analyzes the text and voice data using an emotion engine to recognize the user's emotions.

[1003] 5. The server analyzes the text data and generates an appropriate answer.

[1004] 6. The server converts the answer into audio data and sends it to the device.

[1005] 7. The device plays the audio data and tells the user the answer.

[1006] This system allows users to easily obtain answers to their concerns and questions through voice input. As it is targeted at elderly users, the system is designed to be intuitive and easy to operate. Furthermore, the introduction of an emotion engine makes it possible to respond in accordance with the user's emotions, providing more personalized support. This system is designed to be particularly easy to use for the elderly, and is capable of responding appropriately through emotion recognition, which is expected to increase user satisfaction.

[1007] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1008] Step 1: User Speech Input

[1009] The user speaks a question into the device. For example, if the question is, "My legs have been hurting lately, what should I do?", this voice becomes the input for the system. The user starts voice input by pressing the voice input button on the device and speaking.

[1010] Step 2: Record audio on your device

[1011] The device records the user's voice. It uses the built-in microphone to capture the user's question and generates the voice data. This generated voice data becomes the input for the next process. Specifically, the recording module generates the voice data, "My legs have been hurting lately. What should I do?"

[1012] Step 3: Sending audio data to the server

[1013] The device sends the recorded audio data to the server. A communication protocol such as HTTP or WebSocket is used for transmission. This input audio data is transferred to the server via an HTTP POST request. The audio data sent to the server becomes the input for the next process.

[1014] Step 4: Speech recognition on the server

[1015] The server inputs the received voice data into a voice recognition engine and converts it into text data. The voice recognition engine uses the Google Speech-to-Text API or similar. The generated text data, "My leg has been hurting lately, what should I do?", becomes the input for the next process.

[1016] Step 5: Emotion Recognition on the Server

[1017] The server sends the obtained voice and text data to an emotion engine to recognize the user's emotions. The emotion engine uses a system such as IBM Watson Tone Analyzer. It analyzes the tone, speed, and intensity of the voice and determines that the user is "confused." This emotion data and text data become the input for the next process.

[1018] Step 6: Natural Language Analysis on the Server

[1019] The server sends the emotion data obtained from the emotion engine and the recognized text data to a natural language processing engine. The natural language processing engine uses OpenAI GPT-3 and other technologies. This analyzes the text data and generates a response text such as, "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor." The generated response text becomes the input for the next process.

[1020] Step 7: Convert text to speech and send it

[1021] The generated text response is converted into voice data by a speech synthesis engine. Amazon Polly or another similar engine is used as the voice synthesis engine. Voice data is generated that says, "If your leg hurts, it's important to rest and cool it. If the pain persists, consult a doctor." This voice data becomes the input for the next process and is sent to the device via an HTTP POST request.

[1022] Step 8: Play audio on your device

[1023] The device receives the voice data sent from the server and plays it back. Specifically, the device's speaker plays back the following voice: "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor." This voice playback is the system's final output, allowing the user to obtain an appropriate answer.

[1024] This system allows users to easily obtain answers to their concerns and questions through voice input. Since it is targeted at elderly users, the operation is intuitive and easy. Furthermore, the introduction of an emotion engine makes it possible to respond in accordance with the user's emotions, providing more personalized support.

[1025] (Application example 2)

[1026] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1027] In recent years, when elderly users use food delivery services in their daily lives, they are faced with a lack of diverse options and suggestions that take into account their health status. Furthermore, elderly users are often unfamiliar with the operation of these services, and certain emotional states can make it difficult for them to make appropriate choices. Furthermore, while intuitive operation based on voice input is required, existing systems do not take emotion recognition into account, which can result in low user satisfaction.

[1028] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a user's voice, means for transmitting the voice to the server, means for converting the voice to text data in the server, means for analyzing the text data and generating an appropriate response, means for converting the generated response to voice data, means for transmitting the voice data to the user's terminal, means for playing the voice data in the terminal, means for recognizing the user's emotion, and means for generating a response based on the emotion. This enables elderly users to intuitively and easily use food delivery services and receive appropriate suggestions while taking their emotions into consideration.

[1029] The "means for acquiring the user's voice" refers to a combination of hardware and software for recognizing and acquiring the voice uttered by the user via a device.

[1030] The "means for transmitting the voice to the server" refers to a communication protocol and software function for transmitting the acquired voice data to the server via a network.

[1031] The "means for converting the voice into text data in the server" is a function for converting voice data into text data using a voice recognition engine in the server.

[1032] The "means for analyzing the text data and generating an appropriate response" refers to the algorithm and software functions that analyze the text data using a natural language processing engine and generate a response appropriate to the user's intention or question.

[1033] The "means for converting the generated answer into voice data" is a function for converting the generated text-format answer into voice data using a voice synthesis engine.

[1034] The "means for transmitting the voice data to the user's terminal" refers to a communication protocol and software function for transmitting the generated voice data to the user's terminal via a network.

[1035] The "means for reproducing the audio data at the terminal" refers to the hardware and software functions for reproducing the audio data received at the terminal and allowing the user to listen to it.

[1036] The "means for recognizing the user's emotions" is a function that analyzes acquired voice and text data and identifies the user's emotional state using an emotion engine.

[1037] The "means for generating an answer based on emotion" is a function of the natural language processing engine that takes into account the recognized emotional state of the user to generate the most appropriate answer.

[1038] This invention is a "smart food assistant" system that allows elderly people to use food delivery services easily and intuitively. The system uses emotion recognition based on the user's voice input to provide optimal meal suggestions.

[1039] System configuration

[1040] Hardware

[1041] 1. User device: Use a device that allows voice input, such as a smartphone or smart speaker.

[1042] 2. Server: A server equipped with a high-performance processor performs speech recognition, natural language processing, and emotion recognition.

[1043] software

[1044] 1. Speech recognition engine: Used to convert the user's voice into text data. Common examples include Google's speech recognition API.

[1045] 2. Natural Language Processing Engine (NLP Engine): Used to analyze text data using generative AI models and generate appropriate suggestions.

[1046] 3. Emotion Recognition Engine: Identifies the user's emotional state from voice and text data. Transformers, a Python library, is available.

[1047] 4. Speech synthesis engine: Used to convert the generated suggestions into audio data. gTTS is an example of this.

[1048] Processing flow

[1049] Generating a Program

[1050] The program consists of the following steps:

[1051] 1. Voice input acceptance: When a user voice-over asks questions or asks questions about food delivery, the device captures and records the voice.

[1052] 2. Audio data transmission: The acquired audio data is sent to the server via the Internet using HTTP or WebSocket as the communication protocol.

[1053] 3. Speech recognition and text conversion: The server uses a speech recognition engine to convert the voice data into text data.

[1054] 4. Emotion recognition: Using the converted text data and the original audio data, the emotion recognition engine analyzes the user's emotions.

[1055] 5. Natural Language Processing and Suggestion Generation: The NLP engine analyzes the user's intent and sentiment to generate optimal meal suggestions.

[1056] 6. Speech synthesis: Converting the generated suggestions into audio data.

[1057] 7. Sending and playing back audio data: The server sends the generated audio data to the terminal, and the terminal plays back the audio data to convey the suggestion to the user.

[1058] Specific examples

[1059] Example 1: Elderly users have trouble deciding what to eat

[1060] User says: "I'm not sure what to have for dinner tonight. Can you recommend something?"

[1061] The system responds: "The weather is nice today, how about a fresh vegetable salad?" (If the user seems a little depressed: "I recommend a meat dish to energize you today.")

[1062] Example prompt

[1063] "I'm not sure what to have for dinner tonight. What do you recommend?" -> An older user says they're having trouble deciding what to eat. They seem a little depressed. Take their emotions into consideration and offer healthy, uplifting meal suggestions.

[1064] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1065] Step 1:

[1066] Voice input reception

[1067] The user speaks, "I'm not sure what to have for dinner tonight. What do you recommend?" The device records the speech. The recorded speech data undergoes pre-processing before being sent to the speech recognition engine. Specifically, digital signal processing is performed to change the sampling rate of the speech data to an appropriate value and remove noise.

[1068] Input: User's voice

[1069] Output: Preprocessed audio data

[1070] Step 2:

[1071] Audio data transmission

[1072] After preprocessing, the audio data is sent from the device to the server using HTTP or WebSocket communication protocols. The audio data is split into network packets and securely transmitted using a secure transport layer.

[1073] Input: Preprocessed audio data

[1074] Output: Audio data sent to the server

[1075] Step 3:

[1076] Speech recognition and text conversion

[1077] The server inputs the received voice data into a voice recognition engine, which converts the voice data into text data. Here, features are extracted from the voice data and a deep learning model is used to convert the voice into text. As a specific example, the generated text is, "I'm not sure what to have for dinner tonight. What do you recommend?"

[1078] Input: Audio data received by the server

[1079] Output: Text data (e.g., "I'm not sure what to have for dinner tonight. What do you recommend?")

[1080] Step 4:

[1081] emotion recognition

[1082] The server inputs the converted text data and the original voice data into an emotion recognition engine to analyze the user's emotions. The emotion engine analyzes the tone, speed, and intensity of the voice and the content of the text to identify the user's emotional state. For example, it may recognize that the user is depressed.

[1083] Input: Text and audio data

[1084] Output: Emotion data (e.g., depressed)

[1085] Step 5:

[1086] Natural Language Processing and Suggestion Generation

[1087] The server inputs the emotion data and text data into a natural language processing engine (NLP engine) and generates suggestions based on the user's intentions and emotions. The NLP engine uses a generative AI model to generate suggestions such as "I'm not sure what to have for dinner tonight" and "I'm feeling down," based on the information that the user is unsure about what to have for dinner tonight.

[1088] Input: Text data and emotion data

[1089] Output: Suggestion text (e.g. "Today, I recommend a meat dish to energize you.")

[1090] Step 6:

[1091] Speech synthesis

[1092] The server inputs the generated suggested text into a speech synthesis engine and converts it into voice data. The speech synthesis engine converts the text into natural language and outputs it as an audio file. Specifically, software such as "gTTS" is used to convert the suggested text into an audio file.

[1093] Input: Suggested text

[1094] Output: Speech data (e.g., "I recommend a meat dish to energize you today.")

[1095] Step 7:

[1096] Sending and playing audio data

[1097] The server sends the generated audio data to the user's device using HTTP or WebSocket communication protocols. The device plays the received audio data and provides suggestions to the user by decoding the audio data and playing it on the speaker.

[1098] Input: Audio data sent from the server

[1099] Output: Speech played to the user (e.g., "Today, I recommend a meat dish to energize you.")

[1100] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1101] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1102] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1103] [Fourth embodiment]

[1104] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1105] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1106] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1107] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1108] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1109] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1110] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1111] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1112] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1113] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1114] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1115] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1116] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1117] This invention is a system that realizes a "anything-to-anything consultation app" that responds to the everyday worries and questions of elderly users, and features no codes, no prompts, and no text input. This system allows users to obtain information easily, quickly, and accurately using only voice input.

[1118] An embodiment of the system operates as follows.

[1119] User voice input

[1120] The user speaks a question into the device. For example, "My legs have been hurting lately. What should I do?"

[1121] Recording audio on your device

[1122] The terminal has a function for recording the user's voice, records the user's question, and generates the voice data.

[1123] Sending voice data to the server

[1124] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket.

[1125] Server-based speech recognition

[1126] The server inputs the received voice data into a voice recognition engine and converts it into text data. For example, the resulting text data is "My leg has been hurting lately. What should I do?"

[1127] Server-based natural language analysis

[1128] The server sends the obtained text data to a natural language processing engine, which analyzes the user's question, understands the intent of the question, and generates an appropriate answer.

[1129] Speech conversion and transmission of text data

[1130] The generated text data of the answer is converted into voice data by a speech synthesis engine. For example, voice data with the content "If your leg hurts, it is important to first rest and cool it. If the pain persists, consult a doctor" is generated. The server sends this voice data to the device.

[1131] Playing audio on your device

[1132] The device receives the audio data sent from the server and plays it back, telling the user, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor."

[1133] As a concrete example, if a user asks the terminal, "My legs have been hurting lately. What should I do?", the system will operate as follows.

[1134] 1. The user provides voice input.

[1135] 2. The device records the audio and sends the data to the server.

[1136] 3. The server converts the audio data into text.

[1137] 4. The server analyzes the text data and generates an appropriate answer.

[1138] 5. The server converts the answer into audio data and sends it to the device.

[1139] 6. The device plays the audio data and tells the user the answer.

[1140] In this way, users can easily get answers to their concerns and questions through voice input. Also, since it is targeted at elderly users, the series of operations is designed to be intuitive and easy to perform.

[1141] The processing flow will be explained below.

[1142] Step 1:

[1143] The user speaks a question into the device, which then records the user's voice. For example, the user might ask, "My legs have been hurting lately. What should I do?"

[1144] Step 2:

[1145] The device records the audio and prepares the recording, which is saved as an audio file.

[1146] Step 3:

[1147] The device sends the recorded audio data to the server using the HTTP or WebSocket communication protocol.

[1148] Step 4:

[1149] The server inputs the received voice data into a voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. For example, the generated text data is "My leg has been hurting lately. What should I do?"

[1150] Step 5:

[1151] The server sends the obtained text data to a natural language processing engine, which analyzes the text data and understands the meaning of the user's question.

[1152] Step 6:

[1153] The server's natural language processing engine generates an appropriate answer based on the analysis results. For example, a text data answer such as "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor."

[1154] Step 7:

[1155] The server generates a text response, which is then input to a speech synthesis engine and converted into voice data. The speech synthesis engine then converts the text response into voice data.

[1156] Step 8:

[1157] The server transmits the generated voice data to the terminal, and the voice data is transmitted using a communication protocol.

[1158] Step 9:

[1159] The terminal receives the voice data sent from the server, and then acquires and prepares the voice data.

[1160] Step 10:

[1161] The device plays back the audio data and gives the user a spoken response, specifically saying, "If your leg hurts, it's important to rest and cool it down. If the pain persists, consult a doctor."

[1162] Example 1

[1163] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1164] For users who have difficulty using keyboards or touch input, such as the elderly, there is a need to provide a simple and intuitive interface that can quickly and accurately respond to their daily concerns and questions. However, conventional systems are complicated to operate, and the variety of input methods results in a high learning cost. To solve this problem, it is necessary to build a system that can obtain information using only voice input.

[1165] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1166] In this invention, the server includes a means for acquiring the user's voice, a means for converting the recorded voice data into text data, and a means for analyzing the text data and generating appropriate answers, thereby enabling users to quickly obtain answers to their questions and concerns by simply inputting their voice.

[1167] "Means for acquiring user's voice" refers to a device or function that physically detects the voice spoken by the user and acquires it as digital data.

[1168] The "recording terminal means" refers to a terminal or device that electronically records acquired voice and saves it as voice data.

[1169] The "means for transmitting recorded voice data to a server" refers to a function or protocol for transferring voice data recorded on a terminal to a server via a communication network.

[1170] "Means for converting voice data into text data on a server" refers to a process or system that uses voice recognition technology to convert voice data into text information.

[1171] The "means for analyzing text data and generating appropriate answers" is a function that uses natural language processing technology to understand the meaning of text data and generate appropriate answers to user questions.

[1172] The "means for converting the generated answers into voice data" refers to a process or device that converts the text-format answers into voice data using voice synthesis technology.

[1173] The "means for transmitting voice data to a user's terminal" refers to a function or protocol for transferring the generated voice data to a user's terminal via a communication network.

[1174] "Means for reproducing audio data at a terminal" refers to a function or device for reproducing audio data received by a user's terminal and audibly conveying the data to the user.

[1175] A "natural language processing model" is a trained algorithm or framework for understanding the meaning of text data and generating appropriate answers based on that understanding.

[1176] A "communications protocol" is a standardized procedure or rule for sending and receiving data over a network.

[1177] This invention is a system that realizes a "consultation app" that responds to the everyday worries and questions of elderly users. This system features no codes, no prompts, and no text input, allowing users to obtain information easily, quickly, and accurately using only voice input.

[1178] User voice input

[1179] The user speaks a question into the device. For example, they can ask a specific question such as, "My legs have been hurting lately. What should I do?" The user does not need to enter any special text, and can ask questions using only voice input.

[1180] Recording audio on your device

[1181] The device records the user's voice using the built-in microphone. The device uses a voice recorder application built into the device's operating system. When the user finishes answering a question, the recording stops automatically and the voice data is generated.

[1182] Sending voice data to the server

[1183] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket, where it is converted into an appropriate format (e.g., WAV or MP3) and uploaded to a specified endpoint on the server.

[1184] Server-based speech recognition

[1185] The server inputs the received voice data into a voice recognition engine. The voice recognition engine used can be a general-purpose voice recognition API (for example, Google Speech-to-Text API or IBM Watson Speech to Text). This converts the voice data into text data, and specific text data such as "My leg has been hurting lately. What should I do?" is obtained.

[1186] Server-based natural language analysis

[1187] The server sends the text data to a natural language processing engine, such as OpenAI's GPT-3 or Google Natural Language API, to analyze the question, allowing the server to generate an appropriate answer to the user's question.

[1188] Speech conversion and transmission of text data

[1189] The generated text data of the answer is sent by the server to a speech synthesis engine. Examples of speech synthesis engines used include Google Text-to-Speech API and Amazon Polly. For example, specific speech data such as "If your leg hurts, it's important to first rest and ice it. If the pain persists, consult a doctor" is generated. The server then sends this speech data to the device.

[1190] Playing audio on your device

[1191] The device receives the audio data sent from the server and plays it back using a built-in speaker, providing the user with specific advice such as, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor."

[1192] Specific examples

[1193] For example, if a user asks a question by voice, such as "My legs have been hurting lately, what should I do?", the system operates as follows: First, the device records the voice and sends the data to the server. Next, the server converts the voice data into text, analyzes the text data, and generates an appropriate answer. Finally, the generated answer is converted into voice data, sent to the device, and played back to the user.

[1194] In this way, users can easily get answers to their concerns and questions through voice input. Also, since it is targeted at elderly users, the series of operations is designed to be intuitive and easy to perform.

[1195] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1196] Step 1:

[1197] The user speaks a question into the terminal. For example, "My legs have been hurting lately. What should I do?" The input is the user's voice, and the output is the voice data recorded by the terminal.

[1198] Step 2:

[1199] The device uses a built-in microphone to record the user's voice and save it as digital audio data. Specifically, it uses a voice recorder application built into the device's operating system and saves it as an audio file in wav or mp3 format. The input is the user's voice and the output is an audio file.

[1200] Step 3:

[1201] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket. Specifically, the device converts the audio data into an appropriate format and uploads it to a specified endpoint on the server. The input is the audio file, and the output is the audio data sent to the server.

[1202] Step 4:

[1203] The server inputs the received voice data into a voice recognition engine. The voice recognition engine used can be a general voice recognition API (for example, Google Speech-to-Text API). The input is voice data, and the output is converted text data. Specifically, the voice data is sent to the voice recognition API, and text data such as "My leg has been hurting lately, what should I do?" is obtained.

[1204] Step 5:

[1205] The server sends the obtained text data to a natural language processing engine, which analyzes the question. The natural language processing engine uses OpenAI's GPT-3 or equivalent natural language processing technology. The input is text data, and the output is the analyzed answer text. Specifically, the text data is sent to the natural language processing engine, and the answer obtained is, "If your leg hurts, it is important to first rest and ice it. If the pain persists, consult a doctor."

[1206] Step 6:

[1207] The server sends the generated answer text to a speech synthesis engine and converts it into audio data. Examples of speech synthesis engines that can be used include Google Text-to-Speech API and Amazon Polly. The input is the answer text and the output is audio data. Specifically, the answer text is sent to the speech synthesis engine, which generates an audio file (e.g., mp3 format).

[1208] Step 7:

[1209] The server sends the generated audio data to the device. The transmission method uses a communication protocol (HTTP or WebSocket). The input is the audio data, and the output is the audio data sent to the device.

[1210] Step 8:

[1211] The device plays the received voice data. Specifically, it plays the voice data using the device's speaker and provides the answer in an audible format to the user. The input is the voice data, and the output is the voice answer presented to the user.

[1212] (Application example 1)

[1213] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1214] There is a demand for systems that can easily resolve questions and problems that elderly people and users unfamiliar with technology have in their daily lives and while shopping. In particular, physical stores such as supermarkets lack easy ways to obtain information about products and sales areas. Conventional systems that require text input or complex operations are difficult for elderly people to use, so there is a need to develop systems that can be used intuitively with only voice input.

[1215] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1216] In this invention, the server includes a means for converting speech into text data, a means for generating appropriate responses using a generative AI model, and a means for inputting prompt sentences, which enables elderly people and users who are unfamiliar with technology to easily obtain product locations and information in a store using only voice input.

[1217] "User" means a person who uses the system.

[1218] "Means for acquiring audio" refers to devices or software that record the audio emitted by the user and store it as digital data.

[1219] "Server" refers to a computer system for receiving and processing audio data.

[1220] "Means for converting into text data" refers to the technology and software that converts audio data into text information.

[1221] "Means of analysis and generating appropriate answers" refers to the technologies and algorithms used to analyze text data and create optimal answers to users' questions.

[1222] "Means for converting generated answers into audio data" refers to technology or software that converts text-based answers into data that can be played as audio.

[1223] "Terminal" refers to a device (smartphone, smart glasses, etc.) that is directly operated by the user and used to input and output audio.

[1224] "Means for playback" refers to the function for outputting audio data received by the terminal as audio through a speaker or the like.

[1225] "A means of support for users to ask about product locations and information by voice in a store" refers to an auxiliary function that allows users to ask questions about specific products or information in a physical store.

[1226] "Generative AI model" refers to artificial intelligence technology used to generate appropriate answers to user questions.

[1227] "Means for inputting prompt sentences" refers to the technology or method for inputting sentences or data that give operational instructions to the generative AI model.

[1228] "Communication protocol" refers to the rules and procedures for transmitting digital information.

[1229] "Natural language processing engine" refers to technologies and algorithms for understanding, processing, and generating natural language text.

[1230] The system of this invention aims to enable elderly people and users unfamiliar with technology to easily obtain information about products and sales areas in brick-and-mortar stores such as supermarkets using only their voice. The specific system configuration and operating procedure are shown below.

[1231] System configuration

[1232] Device: A smartphone or smart glasses held by a user, used to input voice and play audio output.

[1233] Server: Responsible for processing voice data. Has functions such as speech recognition, natural language processing, and answer generation using generative AI models.

[1234] Hardware and software used

[1235] Speech recognition engine: Used to convert the user's voice into text data.

[1236] Natural language processing engine: An algorithm that analyzes text data and generates appropriate answers using generative AI models.

[1237] Generative AI model: An artificial intelligence technique for generating optimal answers to user questions.

[1238] Speech synthesis engine: Used to convert the generated text response into audio data.

[1239] Communication protocol: A communication rule such as HTTP or WebSocket for sending and receiving data between a terminal and a server.

[1240] Program processing explanation

[1241] 1. User voice input: The user voices a question into the device, for example, "Where are the tomatoes?"

[1242] 2. Voice recording on the device: The device records the user's voice and generates the voice data.

[1243] 3. Sending audio data to the server: The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket.

[1244] 4. Speech recognition on the server: The server inputs the received voice data into a voice recognition engine and converts the voice into text data. For example, the resulting text data is "Where are the tomatoes?"

[1245] 5. Natural language analysis on the server: The server sends the obtained text data to a natural language processing engine, which analyzes the question, understands the intent of the question, and generates an appropriate answer using a generative AI model.

[1246] 6. Text-to-speech conversion and transmission: The generated text answer is converted into voice data by a speech synthesis engine. For example, voice data such as "The tomatoes are in aisle 3" is generated, and the server transmits this voice data to the device.

[1247] 7. Playback on the device: The device receives the audio data sent from the server and plays it back, telling the user the answer: "The tomatoes are in aisle 3."

[1248] Specific examples

[1249] If a user asks a terminal, "Where are the tomatoes?", the system will do the following:

[1250] Speak: "Where are the tomatoes?"

[1251] Request to server: "Where are the tomatoes?"

[1252] Server responds: "Tomatoes are in aisle 3."

[1253] Speech synthesis and playback: The audio response "The tomatoes are in aisle 3" is played.

[1254] Prompt Sentence Examples

[1255] "If a user asks, 'Where are the tomatoes?' generate the right answer."

[1256] This system will enable even elderly people and those unfamiliar with technology to easily obtain information using voice within the supermarket, greatly improving convenience.

[1257] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1258] Step 1:

[1259] The user speaks a question into the device. For example, "Where are the tomatoes?" The input for this step is the user's voice, and the output is the voice data recorded by the device. Specifically, the device's microphone captures the voice and records it in digital format.

[1260] Step 2:

[1261] The device records the user's voice and generates voice data. The generated voice data is saved as a digital file. The input for this step is the user's voice, and the output is the recorded voice data. Specifically, the data is saved in a format required for input to the voice recognition engine.

[1262] Step 3:

[1263] The device sends the recorded audio data to the server. The input of this step is the audio data, and the output is the audio data sent to the server. Specifically, the device sends the audio data to the server using a communication protocol such as HTTP or WebSocket.

[1264] Step 4:

[1265] The server inputs the received voice data into a voice recognition engine and converts the voice into text data. The input for this step is the voice data sent to the server, and the output is the converted text data. Specifically, the voice recognition engine analyzes the sound wave data and generates the text "Where are the tomatoes?"

[1266] Step 5:

[1267] The server sends the obtained text data to a natural language processing engine, which analyzes the user's question. It understands the intent of the question and generates an appropriate answer using a generative AI model. The input to this step is text data, and the output is the text data of the generated answer. Specifically, the natural language processing engine analyzes the text and provides a prompt sentence to the generative AI model to generate an answer.

[1268] Step 6:

[1269] The server converts the generated text data of the answer into voice data using a voice synthesis engine. The input of this step is the generated text data of the answer, and the output is the converted voice data. Specifically, the voice synthesis engine generates the voice saying, "The tomatoes are in aisle 3."

[1270] Step 7:

[1271] The server sends the generated voice data to the user's terminal. The input of this step is the voice data, and the output is the voice data sent to the terminal. In specific operations, the server sends the voice data to the terminal using a communication protocol.

[1272] Step 8:

[1273] The device receives the voice data sent from the server and plays it back. The input of this step is the voice data sent to the device, and the output is the voice response played back to the user. Specifically, the device's speaker plays back the voice, "The tomatoes are in aisle 3."

[1274] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1275] This invention is a system that realizes a "consultation app" that responds to the daily worries and questions of elderly users, and features no codes, no prompts, and no text input. This system allows users to easily, quickly, and accurately obtain information using only voice input. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide answers that correspond to the user's emotional state.

[1276] An embodiment of the system operates as follows.

[1277] User voice input

[1278] The user speaks a question into the device, which then records the user's voice. For example, the user might ask, "My legs have been hurting lately. What should I do?"

[1279] Recording audio on your device

[1280] The terminal has a function for recording the user's voice, records the user's question, and generates the voice data.

[1281] Sending voice data to the server

[1282] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket.

[1283] Server-based speech recognition

[1284] The server inputs the received voice data into a voice recognition engine and converts it into text data. For example, the resulting text data is "My leg has been hurting lately. What should I do?"

[1285] Emotion recognition on the server

[1286] The server sends the acquired voice and text data to the emotion engine to recognize the user's emotion. The emotion engine analyzes the tone, speed, and intensity of the voice to recognize the user's emotional state.

[1287] Server-based natural language analysis

[1288] The server sends the emotion data and text data obtained from the emotion engine to the natural language processing engine, which analyzes the text data, understands the meaning of the user's question, and generates an appropriate answer taking the emotion data into account.

[1289] Speech conversion and transmission of text data

[1290] The generated text data of the answer is converted into voice data by a speech synthesis engine. For example, voice data with the content "If your leg hurts, it is important to first rest and cool it. If the pain persists, consult a doctor" is generated. The server sends this voice data to the device.

[1291] Playing audio on your device

[1292] The device receives the audio data sent from the server and plays it back, telling the user, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor."

[1293] As a concrete example, if a user asks the terminal, "My legs have been hurting lately. What should I do?", the system will operate as follows.

[1294] 1. The user provides voice input.

[1295] 2. The device records the audio and sends the data to the server.

[1296] 3. The server converts the audio data into text.

[1297] 4. The server analyzes the text and voice data using an emotion engine to recognize the user's emotions.

[1298] 5. The server analyzes the text data and generates an appropriate answer.

[1299] 6. The server converts the answer into audio data and sends it to the device.

[1300] 7. The device plays the audio data and tells the user the answer.

[1301] In this way, users can easily get answers to their concerns and questions through voice input. Since the app is targeted at elderly users, the entire process is designed to be intuitive and easy to follow. Furthermore, the introduction of an emotion engine makes it possible to respond in accordance with the user's emotions, providing more personalized support.

[1302] The processing flow will be explained below.

[1303] Step 1:

[1304] The user speaks a question into the device, which then records the user's voice. For example, the user might ask, "My legs have been hurting lately. What should I do?"

[1305] Step 2:

[1306] The device records the user's voice and prepares the recording, which is then saved as an audio file.

[1307] Step 3:

[1308] The device sends the recorded audio data to the server using the HTTP or WebSocket communication protocol.

[1309] Step 4:

[1310] The server inputs the received voice data into a voice recognition engine. The voice recognition engine analyzes the voice data and converts it into text data. For example, the generated text data is "My leg has been hurting lately. What should I do?"

[1311] Step 5:

[1312] The server inputs the acquired voice data into the emotion engine, which analyzes the tone, speed, and intensity of the voice to recognize the user's emotional state. For example, if the voice tone is low and the speed is slow, it will recognize that the user is feeling anxious.

[1313] Step 6:

[1314] The server sends the emotion data and text data obtained from the emotion engine to the natural language processing engine, which analyzes the text data and understands the meaning of the user's question.

[1315] Step 7:

[1316] Based on the analysis results, the server's natural language processing engine takes emotional data into consideration and generates an appropriate answer. For example, a text answer such as "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor" is generated.

[1317] Step 8:

[1318] The server generates a text response, which is then input to a speech synthesis engine and converted into voice data. The speech synthesis engine then converts the text response into voice data.

[1319] Step 9:

[1320] The server transmits the generated voice data to the terminal, and the voice data is transmitted using a communication protocol.

[1321] Step 10:

[1322] The terminal receives the voice data sent from the server, and then acquires and prepares the voice data.

[1323] Step 11:

[1324] The device plays back the audio data and gives the user a response in voice. Specifically, it says, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor." Furthermore, if it recognizes that the user is feeling anxious, it plays back the audio in a calmer tone.

[1325] Example 2

[1326] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1327] In modern society, there is a need for prompt and accurate responses to the worries and questions that elderly people have in their daily lives. However, text input and complex operations are a significant burden for elderly people, which creates a barrier to obtaining information. Furthermore, a lack of responses that respond to emotional changes is one of the factors that amplify the anxiety and stress of elderly people. To solve these problems, a system that allows easy information acquisition through voice input alone and that responds with empathy is needed.

[1328] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1329] In this invention, the server includes means for acquiring a user's voice, means for transmitting the voice to the server, means for converting the voice to text data in the server, means for analyzing the text data and voice data to recognize the user's emotion, means for analyzing the text data and generating an appropriate answer, means for converting the generated answer to voice data, means for transmitting the voice data to the user's terminal, and means for playing back the voice data in the terminal. This enables elderly people to easily obtain answers to their worries and questions through voice input, and realizes responses that are sensitive to their emotions.

[1330] A "user" is a person who wishes to use the system to provide voice input and obtain information.

[1331] "Means for acquiring voice" refers to a device for collecting voice information from a user and recording it as digital data.

[1332] A "server" is a computer system that processes and analyzes voice data over a network and generates appropriate responses.

[1333] The "means for transmitting audio to a server" refers to a device or program that has the function of transferring audio data acquired by a terminal to a server using a communication protocol.

[1334] A "means for converting voice into text data" is a program or engine that uses voice recognition technology to convert the content of voice into text.

[1335] "Means for recognizing a user's emotions by analyzing text data and voice data" refers to software or an engine for determining a user's emotional state from the tone of voice and the content of text.

[1336] "Means for analyzing text data and generating appropriate answers" refers to a program or engine that uses natural language processing technology to understand the meaning of input text and generate answers based on that content.

[1337] The "means for converting the generated answer into voice data" is software or an engine with speech synthesis technology that converts text data into a format that can be played back as voice.

[1338] The "means for transmitting voice data to a user's terminal" refers to a device or program that has the function of transmitting voice data generated by a server to a user's terminal using a communication protocol.

[1339] "Means for reproducing the audio data at the terminal" refers to a device that has the function of reproducing the audio data received at the terminal in an audible format for the user.

[1340] This invention is a system that realizes a "consultation app" that responds to the daily worries and questions of elderly users, and features no codes, no prompts, and no text input. Specifically, users can easily, quickly, and accurately obtain information using only voice input. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to provide answers that correspond to the user's emotional state.

[1341] User voice input

[1342] The user speaks their question into the device. The device is equipped with a built-in microphone, and the user can start voice input by pressing the voice input button. For example, they might ask, "My leg has been hurting lately. What should I do?"

[1343] Recording audio on your device

[1344] The terminal has a function to record the user's voice, and uses a built-in microphone to record the user's questions and generate the voice data. The recording module converts the input voice into digital data.

[1345] Sending voice data to the server

[1346] The device sends the recorded audio data to the server using a communication protocol such as HTTP or WebSocket. For example, the audio data can be sent to the server using an HTTP POST request.

[1347] Server-based speech recognition

[1348] The server inputs the received voice data into a voice recognition engine and converts it into text data. The voice recognition engine uses, for example, the Google Speech-to-Text API. The resulting text data is, "My leg has been hurting lately. What should I do?"

[1349] Emotion recognition on the server

[1350] The server sends the acquired voice and text data to an emotion engine, such as IBM Watson Tone Analyzer, which analyzes the tone, speed, and intensity of the voice to determine whether the user is "confused."

[1351] Server-based natural language analysis

[1352] The server sends the emotion data obtained from the emotion engine and the recognized text data to a natural language processing engine. For example, OpenAI GPT-3 is used for the natural language processing engine. This analyzes the text data, understands the meaning of the user's question, and generates an appropriate answer taking into account the emotion data. The answer it generates is, "If your leg hurts, it's important to first rest and apply ice. If the pain persists, consult a doctor."

[1353] Speech conversion and transmission of text data

[1354] The generated text data of the answer is converted into voice data by a speech synthesis engine. For example, Amazon Polly is used as the speech synthesis engine. This generates voice data that says, "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor." The server then sends this voice data to the device via an HTTP POST request.

[1355] Playing audio on your device

[1356] The device receives the voice data sent from the server and plays it back. Specifically, it tells the user, "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor." The device with a speaker also plays back the voice.

[1357] Specific examples

[1358] When a user asks the terminal, "My legs have been hurting lately. What should I do?", the system operates as follows.

[1359] 1. The user provides voice input.

[1360] 2. The device records the audio and sends the data to the server.

[1361] 3. The server converts the audio data into text.

[1362] 4. The server analyzes the text and voice data using an emotion engine to recognize the user's emotions.

[1363] 5. The server analyzes the text data and generates an appropriate answer.

[1364] 6. The server converts the answer into audio data and sends it to the device.

[1365] 7. The device plays the audio data and tells the user the answer.

[1366] This system allows users to easily obtain answers to their concerns and questions through voice input. As it is targeted at elderly users, the system is designed to be intuitive and easy to operate. Furthermore, the introduction of an emotion engine makes it possible to respond in accordance with the user's emotions, providing more personalized support. This system is designed to be particularly easy to use for the elderly, and is capable of responding appropriately through emotion recognition, which is expected to increase user satisfaction.

[1367] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1368] Step 1: User Speech Input

[1369] The user speaks a question into the device. For example, if the question is, "My legs have been hurting lately, what should I do?", this voice becomes the input for the system. The user starts voice input by pressing the voice input button on the device and speaking.

[1370] Step 2: Record audio on your device

[1371] The device records the user's voice. It uses the built-in microphone to capture the user's question and generates the voice data. This generated voice data becomes the input for the next process. Specifically, the recording module generates the voice data, "My legs have been hurting lately. What should I do?"

[1372] Step 3: Sending audio data to the server

[1373] The device sends the recorded audio data to the server. A communication protocol such as HTTP or WebSocket is used for transmission. This input audio data is transferred to the server via an HTTP POST request. The audio data sent to the server becomes the input for the next process.

[1374] Step 4: Speech recognition on the server

[1375] The server inputs the received voice data into a voice recognition engine and converts it into text data. The voice recognition engine uses the Google Speech-to-Text API or similar. The generated text data, "My leg has been hurting lately, what should I do?", becomes the input for the next process.

[1376] Step 5: Emotion Recognition on the Server

[1377] The server sends the obtained voice and text data to an emotion engine to recognize the user's emotions. The emotion engine uses a system such as IBM Watson Tone Analyzer. It analyzes the tone, speed, and intensity of the voice and determines that the user is "confused." This emotion data and text data become the input for the next process.

[1378] Step 6: Natural Language Analysis on the Server

[1379] The server sends the emotion data obtained from the emotion engine and the recognized text data to a natural language processing engine. The natural language processing engine uses OpenAI GPT-3 and other technologies. This analyzes the text data and generates a response text such as, "If your leg hurts, it's important to first rest and cool it. If the pain persists, consult a doctor." The generated response text becomes the input for the next process.

[1380] Step 7: Convert text to speech and send it

[1381] The generated text response is converted into voice data by a speech synthesis engine. Amazon Polly or another similar engine is used as the voice synthesis engine. Voice data is generated that says, "If your leg hurts, it's important to rest and cool it. If the pain persists, consult a doctor." This voice data becomes the input for the next process and is sent to the device via an HTTP POST request.

[1382] Step 8: Play audio on your device

[1383] The device receives the voice data sent from the server and plays it back. Specifically, the device's speaker plays back the following voice: "If your leg hurts, it's important to rest and cool it down first. If the pain persists, consult a doctor." This voice playback is the system's final output, allowing the user to obtain an appropriate answer.

[1384] This system allows users to easily obtain answers to their concerns and questions through voice input. Since it is targeted at elderly users, the operation is intuitive and easy. Furthermore, the introduction of an emotion engine makes it possible to respond in accordance with the user's emotions, providing more personalized support.

[1385] (Application example 2)

[1386] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1387] In recent years, when elderly users use food delivery services in their daily lives, they are faced with a lack of diverse options and suggestions that take into account their health status. Furthermore, elderly users are often unfamiliar with the operation of these services, and certain emotional states can make it difficult for them to make appropriate choices. Furthermore, while intuitive operation based on voice input is required, existing systems do not take emotion recognition into account, which can result in low user satisfaction.

[1388] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a user's voice, means for transmitting the voice to the server, means for converting the voice to text data in the server, means for analyzing the text data and generating an appropriate response, means for converting the generated response to voice data, means for transmitting the voice data to the user's terminal, means for playing the voice data in the terminal, means for recognizing the user's emotion, and means for generating a response based on the emotion. This enables elderly users to intuitively and easily use food delivery services and receive appropriate suggestions while taking their emotions into consideration.

[1389] The "means for acquiring the user's voice" refers to a combination of hardware and software for recognizing and acquiring the voice uttered by the user via a device.

[1390] The "means for transmitting the voice to the server" refers to a communication protocol and software function for transmitting the acquired voice data to the server via a network.

[1391] The "means for converting the voice into text data in the server" is a function for converting voice data into text data using a voice recognition engine in the server.

[1392] The "means for analyzing the text data and generating an appropriate response" refers to the algorithm and software functions that analyze the text data using a natural language processing engine and generate a response appropriate to the user's intention or question.

[1393] The "means for converting the generated answer into voice data" is a function for converting the generated text-format answer into voice data using a voice synthesis engine.

[1394] The "means for transmitting the voice data to the user's terminal" refers to a communication protocol and software function for transmitting the generated voice data to the user's terminal via a network.

[1395] The "means for reproducing the audio data at the terminal" refers to the hardware and software functions for reproducing the audio data received at the terminal and allowing the user to listen to it.

[1396] The "means for recognizing the user's emotions" is a function that analyzes acquired voice and text data and identifies the user's emotional state using an emotion engine.

[1397] The "means for generating an answer based on emotion" is a function of the natural language processing engine that takes into account the recognized emotional state of the user to generate the most appropriate answer.

[1398] This invention is a "smart food assistant" system that allows elderly people to use food delivery services easily and intuitively. The system uses emotion recognition based on the user's voice input to provide optimal meal suggestions.

[1399] System configuration

[1400] Hardware

[1401] 1. User device: Use a device that allows voice input, such as a smartphone or smart speaker.

[1402] 2. Server: A server equipped with a high-performance processor performs speech recognition, natural language processing, and emotion recognition.

[1403] software

[1404] 1. Speech recognition engine: Used to convert the user's voice into text data. Common examples include Google's speech recognition API.

[1405] 2. Natural Language Processing Engine (NLP Engine): Used to analyze text data using generative AI models and generate appropriate suggestions.

[1406] 3. Emotion Recognition Engine: Identifies the user's emotional state from voice and text data. Transformers, a Python library, is available.

[1407] 4. Speech synthesis engine: Used to convert the generated suggestions into audio data. gTTS is an example of this.

[1408] Processing flow

[1409] Generating a Program

[1410] The program consists of the following steps:

[1411] 1. Voice input acceptance: When a user voice-over asks questions or asks questions about food delivery, the device captures and records the voice.

[1412] 2. Audio data transmission: The acquired audio data is sent to the server via the Internet using HTTP or WebSocket as the communication protocol.

[1413] 3. Speech recognition and text conversion: The server uses a speech recognition engine to convert the voice data into text data.

[1414] 4. Emotion recognition: Using the converted text data and the original audio data, the emotion recognition engine analyzes the user's emotions.

[1415] 5. Natural Language Processing and Suggestion Generation: The NLP engine analyzes the user's intent and sentiment to generate optimal meal suggestions.

[1416] 6. Speech synthesis: Converting the generated suggestions into audio data.

[1417] 7. Sending and playing back audio data: The server sends the generated audio data to the terminal, and the terminal plays back the audio data to convey the suggestion to the user.

[1418] Specific examples

[1419] Example 1: Elderly users have trouble deciding what to eat

[1420] User says: "I'm not sure what to have for dinner tonight. Can you recommend something?"

[1421] The system responds: "The weather is nice today, how about a fresh vegetable salad?" (If the user seems a little depressed: "I recommend a meat dish to energize you today.")

[1422] Example prompt

[1423] "I'm not sure what to have for dinner tonight. What do you recommend?" -> An older user says they're having trouble deciding what to eat. They seem a little depressed. Take their emotions into consideration and offer healthy, uplifting meal suggestions.

[1424] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1425] Step 1:

[1426] Voice input reception

[1427] The user speaks, "I'm not sure what to have for dinner tonight. What do you recommend?" The device records the speech. The recorded speech data undergoes pre-processing before being sent to the speech recognition engine. Specifically, digital signal processing is performed to change the sampling rate of the speech data to an appropriate value and remove noise.

[1428] Input: User's voice

[1429] Output: Preprocessed audio data

[1430] Step 2:

[1431] Audio data transmission

[1432] After preprocessing, the audio data is sent from the device to the server using HTTP or WebSocket communication protocols. The audio data is split into network packets and securely transmitted using a secure transport layer.

[1433] Input: Preprocessed audio data

[1434] Output: Audio data sent to the server

[1435] Step 3:

[1436] Speech recognition and text conversion

[1437] The server inputs the received voice data into a voice recognition engine, which converts the voice data into text data. Here, features are extracted from the voice data and a deep learning model is used to convert the voice into text. As a specific example, the generated text is, "I'm not sure what to have for dinner tonight. What do you recommend?"

[1438] Input: Audio data received by the server

[1439] Output: Text data (e.g., "I'm not sure what to have for dinner tonight. What do you recommend?")

[1440] Step 4:

[1441] emotion recognition

[1442] The server inputs the converted text data and the original voice data into an emotion recognition engine to analyze the user's emotions. The emotion engine analyzes the tone, speed, and intensity of the voice and the content of the text to identify the user's emotional state. For example, it may recognize that the user is depressed.

[1443] Input: Text and audio data

[1444] Output: Emotion data (e.g., depressed)

[1445] Step 5:

[1446] Natural Language Processing and Suggestion Generation

[1447] The server inputs the emotion data and text data into a natural language processing engine (NLP engine) and generates suggestions based on the user's intentions and emotions. The NLP engine uses a generative AI model to generate suggestions such as "I'm not sure what to have for dinner tonight" and "I'm feeling down," based on the information that the user is unsure about what to have for dinner tonight.

[1448] Input: Text data and emotion data

[1449] Output: Suggestion text (e.g. "Today, I recommend a meat dish to energize you.")

[1450] Step 6:

[1451] Speech synthesis

[1452] The server inputs the generated suggested text into a speech synthesis engine and converts it into voice data. The speech synthesis engine converts the text into natural language and outputs it as an audio file. Specifically, software such as "gTTS" is used to convert the suggested text into an audio file.

[1453] Input: Suggested text

[1454] Output: Speech data (e.g., "I recommend a meat dish to energize you today.")

[1455] Step 7:

[1456] Sending and playing audio data

[1457] The server sends the generated audio data to the user's device using HTTP or WebSocket communication protocols. The device plays the received audio data and provides suggestions to the user by decoding the audio data and playing it on the speaker.

[1458] Input: Audio data sent from the server

[1459] Output: Speech played to the user (e.g., "Today, I recommend a meat dish to energize you.")

[1460] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1461] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1462] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1463] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1464] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1465] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1466] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1467] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1468] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1469] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1470] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1471] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1472] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1473] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1474] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1475] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1476] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1477] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1478] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1479] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1480] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1481] The following is further disclosed regarding the above embodiment.

[1482] (Claim 1)

[1483] A means for acquiring a user's voice;

[1484] means for transmitting the audio to a server;

[1485] means for converting the voice into text data in the server;

[1486] means for analyzing the text data and generating an appropriate answer;

[1487] means for converting the generated answer into voice data;

[1488] means for transmitting the voice data to a user terminal;

[1489] means for reproducing the audio data in the terminal;

[1490] A system including:

[1491] (Claim 2)

[1492] 10. The system of claim 1, wherein the means for analyzing and generating an appropriate response uses a natural language processing engine.

[1493] (Claim 3)

[1494] 2. The system according to claim 1, wherein the means for transmitting the voice data to the user terminal uses a communication protocol.

[1495] "Example 1"

[1496] (Claim 1)

[1497] A means for acquiring a user's voice;

[1498] a terminal means for recording the voice;

[1499] means for transmitting the recorded voice data to a server;

[1500] means for converting the voice data into text data in the server;

[1501] means for analyzing the text data and generating an appropriate answer;

[1502] means for converting the generated answer into voice data;

[1503] means for transmitting the voice data to a user terminal;

[1504] means for reproducing the audio data in the terminal;

[1505] A system including:

[1506] (Claim 2)

[1507] 10. The system of claim 1, wherein the means for analyzing and generating an appropriate response uses a natural language processing model.

[1508] (Claim 3)

[1509] 2. The system according to claim 1, wherein the means for transmitting the voice data to the user terminal uses a communication protocol.

[1510] "Application Example 1"

[1511] (Claim 1)

[1512] A means for acquiring a user's voice;

[1513] means for transmitting the audio to a server;

[1514] means for converting the voice into text data in the server;

[1515] means for analyzing the text data and generating an appropriate answer;

[1516] means for converting the generated answer into voice data;

[1517] means for transmitting the voice data to a user terminal;

[1518] means for reproducing the audio data in the terminal;

[1519] A means of assistance for customers to ask about product locations and information by voice in the store;

[1520] a means for using a generative AI model in the means for analyzing and generating an appropriate response;

[1521] means for inputting a prompt sentence to the generative AI model;

[1522] A system including:

[1523] (Claim 2)

[1524] 10. The system of claim 1, wherein the means for analyzing and generating an appropriate response uses a natural language processing engine.

[1525] (Claim 3)

[1526] 2. The system according to claim 1, wherein the means for transmitting the voice data to the user terminal uses a communication protocol.

[1527] "Example 2: Combining Emotion Engines"

[1528] (Claim 1)

[1529] A means for acquiring a user's voice;

[1530] means for transmitting the audio to a server;

[1531] means for converting the voice into text data in the server;

[1532] means for analyzing the text data and voice data to recognize the user's emotions;

[1533] means for analyzing the text data and generating an appropriate answer;

[1534] means for converting the generated answer into voice data;

[1535] means for transmitting the voice data to a user terminal;

[1536] means for reproducing the audio data in the terminal;

[1537] A system including:

[1538] (Claim 2)

[1539] 10. The system of claim 1, wherein the means for analyzing and generating an appropriate response uses a natural language processing engine.

[1540] (Claim 3)

[1541] 2. The system according to claim 1, wherein the means for analyzing the text data and the voice data to recognize the user's emotion uses an emotion engine.

[1542] (Claim 4)

[1543] 2. The system of claim 1, wherein the means for converting the generated answer into voice data uses a voice synthesis engine.

[1544] (Claim 5)

[1545] 2. The system according to claim 1, wherein the means for transmitting the voice data to the user terminal uses a communication protocol.

[1546] "Application example 2 when combining emotion engines"

[1547] (Claim 1)

[1548] A means for acquiring a user's voice;

[1549] means for transmitting the audio to a server;

[1550] means for converting the voice into text data in the server;

[1551] means for analyzing the text data and generating an appropriate answer;

[1552] means for converting the generated answer into voice data;

[1553] means for transmitting the voice data to a user terminal;

[1554] means for reproducing the audio data in the terminal;

[1555] a means for recognizing a user's emotion;

[1556] a means for generating an answer based on the sentiment;

[1557] A system including:

[1558] (Claim 2)

[1559] 10. The system of claim 1, wherein the means for analyzing and generating an appropriate response uses a natural language processing engine.

[1560] (Claim 3)

[1561] 2. The system according to claim 1, wherein the means for transmitting the voice data to the user terminal uses a communication protocol. [Explanation of symbols]

[1562] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for acquiring a user's voice; means for transmitting the audio to a server; means for converting the voice into text data in the server; means for analyzing the text data and generating an appropriate answer; means for converting the generated answer into voice data; means for transmitting the voice data to a user terminal; means for reproducing the audio data in the terminal; A system including:

2. 10. The system of claim 1, wherein the means for analyzing and generating an appropriate response uses a natural language processing engine.

3. 2. The system according to claim 1, wherein the means for transmitting the voice data to the user terminal uses a communication protocol.

Images