system

Abstract

To provide a system with high reliability and efficiency. [Solution] The system includes means for receiving voice input from a user, means for converting the voice input into a digital signal, means for transmitting the digital signal to an external computer, means for converting the digital signal into text data in the external computer, means for analyzing the text data and generating response data based thereon, means for converting the generated response data into voice data, means for transmitting the converted voice data to a terminal, and means for playing the voice data on the terminal.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With current technology, it is difficult to process large-scale language models (LLMs) on edge devices, due to limitations in the computing power and power consumption of the edge devices themselves. Therefore, to achieve more natural conversations between users and AI, real-time speech recognition and response generation are required, but an efficient method for this processing is necessary. In addition, problems such as delays and data loss are easily encountered in the transmission of voice data, so there is a need to solve these problems and provide a highly reliable and efficient system. [Means for solving the problem]

[0005] The present invention provides a system that reduces the load on an edge device by capturing voice input from a user using the edge device and transmitting the voice data to an external computer. Specifically, the system includes a means for receiving voice input from a user, a means for converting the voice input into a digital signal, a means for transmitting the digital signal to an external computer, a means for converting the digital signal into text data on the external computer, a means for analyzing the text data and generating response data based on the text data, a means for converting the generated response data into voice data, a means for transmitting the converted voice data to a terminal, and a means for playing the voice data on the terminal. This offloads much of the voice processing to the external computer, reducing the load on the edge device and enabling real-time responses. Furthermore, the system achieves reliable data communication by packetizing the data when transmitting digital signals and receiving and decompressing the packets on the external computer. Furthermore, the system uses a large-scale language model (LLM) when analyzing text data, enabling the generation of more natural and sophisticated responses.

[0006] A "user" is an entity that utilizes the system to provide voice input.

[0007] A "terminal" is a device that captures voice input from a user and transmits the voice data to an external computer.

[0008] "Voice input" refers to voice information such as verbal instructions or questions uttered by the user.

[0009] A "digital signal" is a signal that has been converted from analog audio input into digital form.

[0010] An "external computer" is a high-performance computer at a remote location that is entrusted with processing voice data.

[0011] "Text data" is character information generated from voice data by a voice recognition engine.

[0012] A "speech recognition engine" is a combination of software or hardware that analyzes speech data and generates corresponding text data.

[0013] "Response data" is text information that is generated by analyzing text data and creating an appropriate response.

[0014] "Audio data" refers to response data in audio format generated using a text-to-speech engine.

[0015] A "packet" is a unit into which data is divided in data communication.

[0016] A "large-scale language model (LLM)" is a machine learning model with a very large number of parameters used in natural language processing. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention is a system that captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. Below, the processing of the program of this system will be explained in natural language.

[0039] Audio capture and transmission

[0040] When a user speaks, the device uses a microphone to capture the voice. The captured voice data is converted into a digital signal and sent to an external computer via the Internet. Here, the data is divided into packets for transmission, ensuring efficient and reliable data transmission.

[0041] Speech recognition and text conversion

[0042] The server receives the voice data packets sent from the device and restores the original voice data. The restored voice data is analyzed by a voice recognition engine and converted into corresponding text data. The voice recognition engine analyzes the voice data in detail and performs highly accurate text conversion.

[0043] Response Generation

[0044] The server inputs the generated text data into a large-scale language model (LLM) to generate an appropriate response. LLM is a natural language processing model with advanced analytical capabilities, allowing it to generate appropriate and natural responses to user questions and instructions. For example, if a user types, "What's the weather forecast?", the LLM will generate a response such as, "Today's weather is sunny, and the temperature is 25 degrees."

[0045] Transcribing and retransmitting responses

[0046] The response data generated by the server is converted into voice data by a text-to-speech engine. This voice data is also packetized and sent to the terminal over the Internet. The packetization and decompression process minimizes data loss and delays.

[0047] Audio playback

[0048] The device then decompresses the received voice data and plays it back to the user through the built-in speaker, allowing the user to receive a response to their question or command as voice. For example, the device provides a natural response to the user, such as "Today's weather is sunny and the temperature is 25 degrees."

[0049] Specific examples

[0050] For example, consider the case where a user speaks to a device, "What's the weather going to be like tomorrow?" In this case, the device captures the user's voice and sends it to the server. The server converts the voice to text, generating the text data "What's the weather going to be like tomorrow?". It then uses LLM to generate an appropriate response, such as "Tomorrow's weather will be rainy and the temperature will be 20 degrees." This response is converted to audio and sent to the device, which ultimately plays it back to the user.

[0051] As described above, this system enables natural conversation between the user and AI. It reduces the load on the edge device and achieves both advanced response generation and real-time performance. This system provides strong support for collaboration between edge devices and external computers, providing users with a high-quality voice interface.

[0052] The processing flow will be explained below.

[0053] Step 1:

[0054] The user performs voice input. The user speaks to the device, saying, "Tell me the weather forecast."

[0055] Step 2:

[0056] The device captures the audio. The device's microphone collects the user's voice and converts the analog audio into a digital signal.

[0057] Step 3:

[0058] The device sends a digital signal to the server, which then divides the signal into packets and sends them over the network to the server.

[0059] Step 4:

[0060] The server receives the packets and restores the digital signal. The server reconstructs the received packets and stores the original digital signal in a buffer.

[0061] Step 5:

[0062] The server uses a speech recognition engine to convert the digital signal into text data, which the server then analyzes and generates the text data "What's the weather forecast?"

[0063] Step 6:

[0064] The server uses a large-scale language model (LLM) to analyze the text data and generate a response. The server inputs the text data "What is the weather forecast?" into the LLM and generates the response text "Today's weather is sunny and the temperature is 25 degrees."

[0065] Step 7:

[0066] The server uses a text-to-speech engine to convert the response text into audio data. The server converts the response text, "Today's weather is sunny and the temperature is 25 degrees," into audio.

[0067] Step 8:

[0068] The server packets the audio data and sends it to the terminal. The server divides the audio data into packets and sends them to the terminal via the network.

[0069] Step 9:

[0070] The terminal receives the packets and reconstructs the voice data. The terminal reconstructs the received packets and restores the original voice data.

[0071] Step 10:

[0072] The device plays the audio data, and the audio response "Today's weather is sunny, and the temperature is 25 degrees" is played to the user through the device's built-in speaker.

[0073] Example 1

[0074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0075] Conventional voice recognition and response generation systems impose a heavy computational load on edge devices and suffer from problems such as network delays and data loss. Furthermore, achieving real-time and highly accurate response generation requires large-scale computational resources, which entails high costs. The present invention aims to solve these problems and provide a fast, high-quality voice response interface while reducing the load on edge devices.

[0076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0077] In this invention, the server includes means including a speech recognition engine for converting voice data into text data, means for analyzing the text data and using a large-scale language model to generate response data based on the text data, and means including a speech synthesis engine for converting the generated response data into voice data, thereby reducing the load on the edge device and enabling highly accurate response generation in real time while minimizing problems such as network delays and data loss.

[0078] "User" refers to a person who utilizes the speech recognition and response generation system of the present invention.

[0079] "Terminal" refers to a device that captures voice input from a user and transmits a digital signal to a server. Examples include smartphones and smart speakers.

[0080] The term "server" refers to a computer that receives data sent from a terminal and performs speech recognition, response generation, and speech synthesis.

[0081] "Voice input" refers to information spoken by a user into a terminal.

[0082] "Digital signal" refers to a signal obtained by converting analog audio data into digital format.

[0083] A "network" refers to a communications infrastructure that interconnects multiple computers and terminals. A specific example is the Internet.

[0084] The "external computer" refers to a computer such as a server that receives and processes data sent from a terminal.

[0085] "Audio data" refers to data that represents an audio signal in digital form.

[0086] "Text data" refers to character information converted from voice data by a voice recognition engine.

[0087] A "voice recognition engine" refers to software or hardware that takes voice data as input, analyzes it, and converts it into text data.

[0088] A "speech synthesis engine" refers to software or hardware that takes text data as input and generates speech data.

[0089] A "large-scale language model" is a natural language processing model that uses deep learning and learns from large amounts of text data to generate and analyze text.

[0090] A "packet" refers to the small units of data that are divided when sending and receiving data over a network.

[0091] "Packetization" refers to the process of breaking down digital signals or data into smaller units (packets) and converting them into a form that can be sent over a network.

[0092] "Reconstruction" refers to the process of reconstructing the received packets into the original voice data.

[0093] "Playback" refers to the act of outputting audio data from a speaker device so that the user can hear it.

[0094] The present invention is a system that receives voice input from a user, processes the input, generates a response, and provides it to the user again in voice. How this system is specifically implemented will be described below.

[0095] Audio capture and conversion to digital signal

[0096] When a user speaks, the device captures the voice using a built-in or connected microphone. Examples include smartphones and smart speakers. The captured voice is converted into a digital signal. Here, the device uses, for example, an ADC (analog-to-digital converter).

[0097] Digital signal transmission

[0098] The converted digital signal is divided into packets and sent over a network (e.g., the Internet) to an external computer, i.e., a server, using a network protocol such as TCP / IP.

[0099] Processing on the server

[0100] The server receives the packets sent from the device, restores them to their original digital signals, and then converts them into text data using a speech recognition engine (e.g., Google® Cloud Speech-to-Text or IBM Watson® Speech to Text).

[0101] Response Generation

[0102] The text data is analyzed using a large-scale language model (e.g., OpenAI's GPT-4®) to generate response data, which is then made appropriate and highly accurate using natural language processing technology.

[0103] Conversion to audio data and transmission

[0104] The generated response data is converted into voice data using a speech synthesis engine (e.g., Amazon Polly or Google Cloud Text-to-Speech). The converted voice data is again divided into packets and sent to the device over the network.

[0105] Audio playback

[0106] The terminal receives the voice data packets sent from the server and restores them to the original voice data. The restored voice data is then played back to the user through the speaker, allowing the user to receive voice responses to their questions and instructions.

[0107] Specific examples

[0108] For example, consider the case where a user speaks to a terminal and asks, "What's the weather going to be like tomorrow?"

[0109] 1. The device captures the user's voice, converts it into a digital signal, and sends it to the server.

[0110] 2. The server receives the voice and uses a speech recognition engine to generate text data such as "What's the weather like tomorrow?"

[0111] 3. The generated text data is analyzed using a large-scale language model, and the response data, "Tomorrow's weather will be rainy and the temperature will be 20 degrees," is generated.

[0112] 4. This response data is converted into voice data using a voice synthesis engine and sent to the terminal.

[0113] 5. The device plays the received voice data, and the user hears the response, "Tomorrow's weather will be rainy and the temperature will be 20 degrees."

[0114] Prompt Sentence Examples

[0115] User: What's the weather like tomorrow?

[0116] Device: (Capture audio and send to server)

[0117] Server: (Receives voice data and converts it to text)

[0118] Server: (Generates response using LLM)

[0119] Server: (Converts the response into audio and sends it to the device)

[0120] Terminal: (receives and plays audio data)

[0121] Through the above procedure, the system of the present invention can realize a natural voice interface with the user, and can provide highly accurate responses while reducing the load on the edge device.

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] Step 1: Capture audio

[0124] The user makes a voice input (e.g., "What's the weather going to be like tomorrow?"). The microphone built into the device captures this voice and converts the analog voice signal into a digital signal. Specifically, the microphone picks up the voice and converts it into digital data via an ADC (analog-to-digital converter).

[0125] (Input) User's voice

[0126] (Output) Digital audio signal

[0127] Step 2: Sending a digital signal

[0128] The terminal breaks the converted digital voice signal into packets, and then uses a network protocol (e.g., TCP / IP) to send the data over the Internet to a server.

[0129] (Input) Digital audio signal

[0130] (Output) Packet data sent over the network

[0131] Step 3: Receiving and restoring audio data

[0132] The server receives the packet data sent from the terminal, reassembles the received packets, and restores the original digital audio signal.

[0133] (Input) Packet data

[0134] (Output) Reconstructed digital audio signal

[0135] Step 4: Speech recognition and text conversion

[0136] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to analyze the digital voice signal and convert it into corresponding text data. Specifically, the server inputs the voice data into the speech recognition engine and obtains the text data as its output.

[0137] (Input) Digital audio signal

[0138] (Output) Text data (e.g., "What's the weather like tomorrow?")

[0139] Step 5: Parsing text data and generating responses

[0140] The server inputs the generated text data into a large-scale language model (e.g., OpenAI's GPT-4) to generate an appropriate response. Specifically, the text data is input into the LLM, and the response text is obtained as the output (e.g., "Tomorrow's weather will be rainy, and the temperature will be 20 degrees").

[0141] (Input) Text data (e.g., "What's the weather like tomorrow?")

[0142] (Output) Response text (e.g. "Tomorrow's weather will be rainy and the temperature will be 20 degrees")

[0143] Step 6: Convert response text to speech

[0144] The server inputs the generated response text into a speech synthesis engine (e.g., Amazon Polly) and converts it into voice data. Specifically, the server inputs the response text and obtains voice data as output.

[0145] (Input) Response text (e.g., "Tomorrow's weather will be rainy and the temperature will be 20 degrees.")

[0146] (Output) Audio data

[0147] Step 7: Sending audio data

[0148] The server divides the voice data into packets and transmits them to the terminal via the network. Specifically, the server packetizes the voice data and transmits them using a network protocol.

[0149] (Input) Audio data

[0150] (Output) Packet data sent over the network

[0151] Step 8: Receiving and Playing Audio Data

[0152] The terminal receives the packet data sent from the server, restores it to the original voice data, and then plays the restored voice data through the built-in speaker.

[0153] (Input) Packet data

[0154] (Output) Audio to be played (e.g. "Tomorrow's weather will be rainy and the temperature will be 20 degrees")

[0155] This is the flow of the program processing for this system, which allows users to use the voice interface in a natural way.

[0156] (Application example 1)

[0157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0158] Current voice input systems require real-time responses and highly accurate recognition when users voice-order or customize their food. However, in many cases, the load on edge devices is high, resulting in delayed responses. Another problem is that low-accuracy recognition may occur, resulting in results that differ from the user's request. These issues are particularly serious in fields such as food delivery, which require fast and accurate processing. The present invention aims to solve these issues and provide a highly accurate, real-time voice input system while reducing the load on edge devices.

[0159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0160] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into a digital signal, means for transmitting the digital signal to an external computer, means for inputting text data into a large-scale language model and generating response data, means for converting the generated response data into voice data, means for transmitting the converted voice data to a terminal, means for the user to place an order by voice input, means for confirming the order details in real time, and means for giving voice instructions for customizing dishes. This allows the user to receive highly accurate voice responses in real time. Furthermore, since orders and customizations are accurately executed based on the user's voice input, an advanced user experience can be provided.

[0161] A "means for receiving audio input from a user" is a device or method for capturing audio produced by a user.

[0162] A "means for converting audio input to a digital signal" is a device or method for converting analog audio data into digital form.

[0163] "Means for transmitting digital signals to an external computer" refers to a device or method for transmitting digitized voice data to an external server via a network such as the Internet.

[0164] The "means for converting digital signals into text data using an external computer" refers to a device or method, such as a voice recognition engine, for converting transmitted digital voice data into text information.

[0165] The "means for inputting the generated text data into a large-scale language model and generating response data" refers to a device or method for inputting text data obtained by a speech recognition engine into a natural language processing model and generating an appropriate response to a user's inquiry or instruction.

[0166] The "means for converting the generated response data into voice data" refers to a device or method for converting the generated text-format response data into synthetic voice.

[0167] The "means for transmitting the converted voice data to the terminal" refers to a device or method for retransmitting the synthesized voice data to the user terminal over the network.

[0168] "Means for reproducing audio data at a terminal" refers to a device or method for reproducing audio data transmitted to a user terminal using an audio device such as a speaker.

[0169] The "means for users to order by voice input" refers to a device or method that allows users to order products or services by voice.

[0170] "Means for confirming order details in real time" refers to a device or method for analyzing and confirming order details in real time based on voice input.

[0171] "Means for giving voice instructions for customizing dishes" refers to a device or method that allows a user to give voice instructions for customizing desired dishes, and that the system understands and reflects these instructions.

[0172] The present invention is a system that captures voice input from users with an edge device and transmits the voice data to an external computer for processing. The system enables voice ordering, real-time confirmation, and customization instructions in the field of food delivery.

[0173] User voice input

[0174] When a user places an order by voice using a food delivery application, the smartphone's microphone is used to capture voice input. For example, the user might say, "I'd like to order one pizza." This voice is converted into a digital signal and sent over the Internet to an external computer, or server.

[0175] Speech data conversion and response generation

[0176] The server converts the received digital voice signal into text data. High-precision text conversion is performed using a speech recognition engine such as Google Cloud Speech-to-Text. The converted text data is input into a large-scale language model (LLM), which generates response data based on the user's question or instruction. Using a generative AI model, a natural response such as, "We've received your pizza order. Is there anything else you'd like to order?" can be obtained.

[0177] Transcribing and retransmitting response data

[0178] The generated response data is converted into voice data using a speech synthesis engine such as Google Text-to-Speech (gTTS), which is then sent back to the user's smartphone via the Internet.

[0179] Audio playback on the user device

[0180] The user's smartphone then plays the received voice data through its built-in speaker, allowing the user to receive natural voice responses to confirm their order or to ask additional questions. For example, a user can customize their order by voice input, such as "more tomatoes, less cheese, please," and the response will be returned in real time.

[0181] Specific examples

[0182] If the user says the following prompt:

[0183] "I'd like to order a pizza."

[0184] "More tomatoes, less cheese, please."

[0185] Based on the prompt above, the system will proceed through the following steps:

[0186] 1. Converts audio input into a digital signal.

[0187] 2. Send the digital signal to the server.

[0188] 3. The server converts the voice signal into text.

[0189] 4. Generate response data using a large-scale language model.

[0190] 5. Convert the response data into audio data.

[0191] 6. The voice data is sent to the user terminal.

[0192] 7. The user device plays the received audio.

[0193] This system allows users to smoothly order food delivery using voice. The main hardware used is a smartphone and a microphone. The software includes Google Cloud Speech-to-Text and Google Text-to-Speech (gTTS), which enables a high-quality voice interface.

[0194] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0195] Step 1:

[0196] A user speaks to a smartphone to place an order or ask a question. For example, they might say, "I'd like to order one pizza." This voice data is captured by a microphone. The input is an analog voice signal, and the output is the captured digital voice data. Specifically, the smartphone's microphone captures the voice and the built-in ADC (analog-to-digital converter) converts it into a digital signal.

[0197] Step 2:

[0198] The terminal transmits the captured digital audio data to a server via the Internet. The input is the captured digital audio data, and the output is the audio data transmitted via the Internet. Specifically, the terminal generates an HTTP request, divides the audio data into packets, and transmits them to the server.

[0199] Step 3:

[0200] The server reconstructs the received voice data and converts it into text data using a speech recognition engine. The input is the transmitted digital voice data, and the output is the converted text data. Specifically, the server restores the received packets to the original voice data and converts the voice data into text using Google Cloud Speech-to-Text.

[0201] Step 4:

[0202] The server uses a large-scale language model (LLM) to analyze the text data and generate response data. The input is the converted text data (e.g., "I would like to order one pizza"), and the output is the response data (e.g., "We have received your pizza order. Is there anything else you would like to order?"). Specifically, the server inputs a prompt sentence into the generative AI model and generates an appropriate response.

[0203] Step 5:

[0204] The server converts the generated response data into voice data. The input is the response data (in text format), and the output is voice data. Specifically, Google Text-to-Speech (gTTS) converts the text into synthetic voice.

[0205] Step 6:

[0206] The server divides the voice data into packets again and sends them to the user terminal via the Internet. The input is voice data, and the output is voice data sent via the Internet. Specifically, the server divides the voice data into packets and sends them as an HTTP response.

[0207] Step 7:

[0208] The terminal restores the received voice data and plays it back to the user through the built-in speaker. The input is the transmitted voice data, and the output is the played voice. Specifically, the terminal restores the received packets to the original voice data and plays back a response using the speaker. For example, the response played back to the user is, "We've received your pizza order. Is there anything else you'd like to order?"

[0209] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0210] This invention is a system that captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. Furthermore, the external computer is equipped with an emotion engine that recognizes the user's emotions and generates responses linked to the emotions, enabling more natural and emotionally appropriate dialogue.

[0211] Audio capture and transmission

[0212] When a user speaks, the device captures the voice using a microphone. The captured voice data is converted into a digital signal and sent to a server over the Internet. Here, the data is divided into packets for transmission, ensuring efficient and reliable data transmission.

[0213] Speech recognition and text conversion

[0214] The server receives the voice data packets sent from the device and restores the original voice data. The restored voice data is analyzed by a voice recognition engine and converted into corresponding text data. The voice recognition engine analyzes the voice data in detail and performs highly accurate text conversion.

[0215] emotion recognition

[0216] Next, the server inputs the text data into the emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from the input text data based on multiple emotion categories and outputs them in the form of numbers or tags. For example, if a user says, "I'm feeling terrible today," the emotion engine will classify it into emotion categories such as "negative" or "frustration."

[0217] Response Generation

[0218] The server inputs the generated text data and emotional information into a large-scale language model (LLM) to generate an appropriate response. LLM is a natural language processing model with advanced analytical capabilities, allowing it to generate an appropriate response tailored to the user's emotions. For example, if a user inputs "I'm feeling terrible today," the LLM will generate a response in the form of "I'm sorry, that's terrible. Is there anything I can help you with?"

[0219] Transcribing and retransmitting responses

[0220] The response data generated by the server is converted into voice data by a text-to-speech engine. This voice data is also packetized and sent to the terminal over the Internet. The packetization and decompression process minimizes data loss and delays.

[0221] Audio playback

[0222] The device decompresses the received voice data and plays it back to the user through the built-in speaker. This allows the user to receive a response to their question or instruction as voice. For example, a response such as "I'm sorry, that's a problem. Is there anything I can help you with?" is provided to the user in a natural way.

[0223] Specific examples

[0224] For example, consider a situation where a user says to a device, "I'm feeling incredibly miserable today." In this case, the device captures the user's voice and sends it to the server. The server converts the voice to text, generating the text data "I'm feeling incredibly miserable today." The emotion engine then recognizes emotions such as "negative" or "sad" from this text data. It then uses the LLM to generate an appropriate response such as "That's really miserable. Let's talk a bit and hopefully make you feel better." This response is converted into audio and sent to the device, which ultimately plays it back to the user.

[0225] As described above, this system enables natural conversation between the user and AI. It reduces the load on the edge device and simultaneously generates sophisticated, emotionally relevant responses in real time. This system provides strong support for collaboration between edge devices and external computers, providing users with a high-quality voice interface.

[0226] The processing flow will be explained below.

[0227] Step 1:

[0228] The user performs voice input. For example, the user speaks to the terminal, "Today is so hard."

[0229] Step 2:

[0230] The device captures the audio. The device's microphone collects the user's voice and converts the analog audio into a digital signal.

[0231] Step 3:

[0232] The terminal sends a digital signal to the server, which then divides the digital signal into packets and sends them over the network to the server.

[0233] Step 4:

[0234] The server receives the packets and restores the digital signal. The server reconstructs the received packets and stores the restored digital signal in a buffer.

[0235] Step 5:

[0236] The server uses a speech recognition engine to convert the digital signal into text data. The server analyzes the digital signal and generates the text data "Today is extremely difficult."

[0237] Step 6:

[0238] The server inputs the text data into the emotion engine and analyzes the emotions. The server recognizes emotions such as "negative" or "sad" from the text data and outputs the results in the form of numbers or tags.

[0239] Step 7:

[0240] The server inputs text data and emotional information into a large-scale language model (LLM) to generate a response. The server passes the text "I'm feeling incredibly miserable today" along with the emotional information "negative" and "sad" to the LLM, generating a response such as "That's really hard on you. I hope we can talk a bit and make you feel better."

[0241] Step 8:

[0242] The server converts the response text into audio data using a text-to-speech engine. The server converts the generated response text into audio data.

[0243] Step 9:

[0244] The server packets the voice data and sends it to the terminal. The server divides the voice data into packets and sends them to the terminal over the network.

[0245] Step 10:

[0246] The terminal receives the packets and reconstructs the voice data. The terminal reconstructs the received packets and restores the original voice data.

[0247] Step 11:

[0248] The device plays the audio data, and through the device's built-in speaker, plays a voice response to the user saying, "That must be really hard on you. Let's talk a bit and hopefully you'll feel better."

[0249] Example 2

[0250] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0251] Conventional speech recognition systems simply convert a user's speech into text and generate simple responses based on that text. This makes it difficult to understand the user's emotions and generate responses that take those emotions into consideration, resulting in a less natural and emotionally rich dialogue. In particular, in situations where the user is emotionally charged, the system is unable to respond appropriately, making it difficult to provide a satisfying user experience. The present invention aims to solve this problem by providing a system that analyzes a user's emotions and provides natural-sounding voice responses based on those emotions.

[0252] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing the user's emotions and generating response data based on the emotions, a means for converting a digital signal into text data, and a means for analyzing the text data and generating response data based on the text data. This makes it possible to generate natural responses that are in line with the user's emotions, enabling more satisfying interactions.

[0253] "User" refers to a person who interacts with the system by inputting voice through a terminal.

[0254] "Terminal" refers to a device that receives voice input from a user, converts it into a digital signal, transmits it to a server, and plays back a voice response from the server to the user.

[0255] "Voice input" refers to the voice that the user utters to the terminal, and is the voice signal that is processed by the system.

[0256] A "digital signal" is a signal obtained by converting an audio input into a digital format, and refers to data in a format that can be processed by a computer.

[0257] "External computer" refers to a computing device that processes digital signals sent from the terminal and performs voice recognition, emotion analysis, and response generation.

[0258] "Text data" refers to character string data generated by an external computer analyzing voice data.

[0259] An "emotion recognition engine" refers to a software function that analyzes text data to recognize a user's emotions and output them in numerical or tag format.

[0260] "Response data" refers to data containing appropriate responses to users, generated using an emotion recognition engine or a large-scale language model.

[0261] "Large-scale language model (LLM)" refers to an advanced language model that uses natural language processing to generate appropriate responses from text data.

[0262] "Text-to-speech engine" means the software functionality for converting generated response data into audio data.

[0263] "Packet" refers to the unit of data into which digital signals are divided and sent over the Internet.

[0264] A "microphone" refers to an audio device for capturing the user's voice.

[0265] "Speaker" refers to an audio device that plays back audio data sent from the server and lets the user hear it.

[0266] This clarifies each element of the system and its function.

[0267] The present invention is a system that captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. Furthermore, the external computer is equipped with an emotion engine that recognizes the user's emotions and generates responses linked to the emotions, enabling more natural and emotionally appropriate dialogue.

[0268] Audio capture and transmission

[0269] When a user speaks, the device captures the voice using a microphone. For example, the user might say, "I'm feeling incredibly miserable today." This voice is converted into a digital signal and sent to a server over the Internet. The voice data is divided into packets for transmission, ensuring efficient and reliable transmission.

[0270] Speech recognition and text conversion

[0271] The server receives the voice data packets sent from the device and restores the original voice data. The restored voice data is analyzed by a voice recognition engine (e.g., an engine using a deep learning model) and converted into corresponding text data. The voice recognition engine analyzes the voice data in detail and performs highly accurate text conversion.

[0272] emotion recognition

[0273] Next, the server inputs the text data into an emotion engine (e.g., an engine using natural language processing technology) to analyze the user's emotions. The emotion engine recognizes the user's emotions from the input text data based on multiple emotion categories and outputs them in the form of numbers or tags. For example, if a user says, "I'm feeling terrible today," the emotion engine will classify it into emotion categories such as "negative" or "frustration."

[0274] Response Generation

[0275] The server inputs the generated text data and emotional information into a large-scale language model (LLM) to generate an appropriate response. LLM is a natural language processing model with advanced analytical capabilities, so it generates an appropriate response that matches the user's emotions. For example, if a user inputs "I'm feeling terrible today," the LLM will generate a response in the form of "I'm sorry, that's terrible. Is there anything I can help you with?" In this case, the LLM can be given a prompt sentence such as:

[0276] "Example prompt statement:"

[0277] User: I'm having a really hard time today

[0278] System: That's really tough. Let's talk a bit and hopefully it will help ease your mind.

[0279] Transcribing and retransmitting responses

[0280] The response data generated by the server is converted into voice data by a text-to-speech engine (e.g., a text-to-speech engine using natural language processing). This voice data is also packetized and sent to the terminal over the Internet. The packetization and decompression process minimizes data loss and delays.

[0281] Audio playback

[0282] The device decompresses the received voice data and plays it back to the user through the built-in speaker. This allows the user to receive a response to their question or instruction as voice. For example, a response such as "I'm sorry, that's a problem. Is there anything I can help you with?" is provided to the user in a natural way.

[0283] These components and processing steps enable dialogue that reflects the user's emotions, providing a more human-like and natural voice interface, which reduces the load on edge devices and enables both emotionally-sensitive and real-time response generation.

[0284] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0285] Step 1: Capture audio

[0286] Input: User's spoken utterance

[0287] How it works: The user speaks into the device's microphone, which captures the audio signal.

[0288] Output: Digitized audio data

[0289] Specific operation: The device's microphone converts analog audio signals into digital signals and temporarily stores the data in the device's memory.

[0290] Step 2: Sending audio data

[0291] Input: Digitized audio data

[0292] How it works: The device splits the captured audio data into packets and sends them over the Internet to a server.

[0293] Output: Packetized voice data sent over the internet

[0294] Specific operation: Audio data is divided into packets of a fixed size, and header information is added to each packet before transmission.

[0295] Step 3: Audio data recovery

[0296] Input: Packetized audio data

[0297] Operation: The server receives packets sent from the device and restores them to the original audio data.

[0298] Output: Recovered audio data

[0299] Specific operation: The received packets are rearranged in order and combined to restore the original audio data.

[0300] Step 4: Speech recognition and text conversion

[0301] Input: Recovered audio data

[0302] How it works: The server inputs the recovered voice data into a speech recognition engine and converts it into highly accurate text data.

[0303] Output: Text data

[0304] Specific operation: A speech recognition engine (e.g., an engine using a deep learning model) analyzes the voice data and generates text data such as "Today is incredibly difficult."

[0305] Step 5: Emotion Recognition

[0306] Input: Generated text data

[0307] How it works: The server inputs text data into an emotion recognition engine to analyze the user's emotions.

[0308] Output: Emotion information (in numerical and tag formats)

[0309] Specific operation: An emotion recognition engine using natural language processing technology tags emotions in the text and assigns labels such as "negative" or "sad."

[0310] Step 6: Response Generation

[0311] Input: Text data and emotion information

[0312] How it works: The server inputs text data and emotion information into a large-scale language model (LLM) to generate an appropriate response.

[0313] Output: Response data

[0314] Specific operation: The user's utterances and emotional data are input as prompts into a large-scale language model (e.g., GPT-4), and a response such as "That's really painful. Let's talk for a bit and hopefully you'll feel better" is generated.

[0315] Step 7: Transcribing responses

[0316] Input: Response data

[0317] How it works: The server inputs the generated response data into a text-to-speech engine and converts it into audio data.

[0318] Output: Audio data

[0319] Specific operation: A text-to-speech engine (e.g., an engine using natural language processing) converts the response data into audio data.

[0320] Step 8: Resend the response audio

[0321] Input: Audio data

[0322] How it works: The server breaks the audio data back into packets and sends them over the Internet to the device.

[0323] Output: Packetized audio data

[0324] Specific operation: Audio data is packetized and sent to the terminal. Packets contain sequence information, making it easy to restore the data at the terminal.

[0325] Step 9: Playing Audio

[0326] Input: Packetized audio data

[0327] Operation: The device reconstructs the received audio packets and plays them through the built-in speaker.

[0328] Output: A spoken response to the user

[0329] Specific operation: The device receives the packets and restores them as voice data. The restored voice data is played over the speaker, providing a response to the user. For example, "That must be really painful. I hope that talking about it will help you feel better."

[0330] (Application example 2)

[0331] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0332] Conventional speech recognition systems have difficulty in accurately recognizing user emotions and generating responses based on them. As a result, especially in customer service and complaint handling, systems can only provide mechanical responses that ignore the user's emotions, making it difficult to improve customer satisfaction. In addition, edge devices themselves have limited processing power, making it difficult to perform highly accurate emotion recognition and response generation in real time.

[0333] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into a digital signal, means for transmitting the digital signal to an external computer, means for converting the digital signal into text data in the external computer, means for analyzing the text data and generating response data based on the text data, means for converting the generated response data into voice data, means for transmitting the converted voice data to a terminal, means for playing the voice data on the terminal, and means for recognizing the user's emotions in a conversation with a customer and generating an appropriate response based on the emotions. This makes it possible to provide more natural and appropriate responses in real time that correspond to the user's emotions.

[0334] "User" refers to any person or subject that uses the system to provide voice input.

[0335] The "means for receiving audio input" refers to a device for capturing audio data from a user, and specifically refers to an audio capture device such as a microphone.

[0336] "Means for converting audio input into a digital signal" refers to a process or device that converts received audio into electronic digital data, typically an analog-to-digital converter (ADC).

[0337] "Means for transmitting digital signals to an external computing device" refers to means for transmitting digital data over the Internet or other communications network to a remote server or computing resource.

[0338] "External computers" refer to servers or cloud computing environments that exist separately from edge devices and are used for advanced computational processing and analysis.

[0339] "Means for converting digital signals into text data" refers to software or a system that uses a speech recognition engine to convert digital voice data into a corresponding text format.

[0340] "Means for analyzing text data and generating response data based on it" refers to a program or system for interpreting received text data and generating optimal responses or instructions in response to it.

[0341] The "means for converting the generated response data into voice data" refers to a text-to-speech engine (TTS) for outputting the text-format response data as voice again.

[0342] "Means for transmitting converted voice data to a terminal" refers to a communication means for returning the generated voice data to the edge device.

[0343] "Means for reproducing audio data on a terminal" refers to means for reproducing received audio data through an audio device such as a speaker.

[0344] "Means for recognizing a user's emotions in customer interactions and generating appropriate responses based on those emotions" refers to large-scale language models and other artificial intelligence technologies that use emotion recognition engines to understand a user's emotions and generate natural-sounding responses accordingly.

[0345] MODE FOR CARRYING OUT THE INVENTION

[0346] A system embodying this invention captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. A specific implementation of this system is described below.

[0347] Audio capture and transmission

[0348] When a user speaks, the smart glasses used as a terminal capture the voice using a microphone. The captured voice data is converted into a digital signal by a voice capture device. This converted digital signal is transmitted to an external computer server via the Internet. Specifically, the data is transmitted in packets, enabling efficient data transmission.

[0349] Speech recognition and text conversion

[0350] The server receives the voice data packets sent from the device and restores them to the original voice data. The restored voice data is analyzed using a voice recognition engine and converted into corresponding text data. The voice recognition engine is typically a highly accurate software.

[0351] emotion recognition

[0352] Next, the server inputs the text data into an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from the input text data based on multiple emotion categories. This emotion information is output in the form of numbers and tags.

[0353] Response Generation

[0354] The server uses the generated text data and emotion information to generate appropriate responses using a large-scale language model (LLM), which has the ability to provide natural responses that are in line with the user's input.

[0355] Transcribing and retransmitting responses

[0356] The generated response data is converted into voice data by a text-to-speech engine, which is also packetized and transmitted to the smart glasses via the internet.

[0357] Audio playback

[0358] The smart glasses, which are the terminals, decompress the received voice data and play it back to the user through the built-in speaker, allowing the user to receive voice responses to their questions and instructions.

[0359] For example, if a store staff member asks, "Hey, is there something I can help you with?" and the customer replies, "Hey, I don't know how to use this product," the system captures the voice and sends it to the server. The server analyzes the voice and recognizes the emotion of "the customer is having trouble." It then generates an appropriate response, such as "That's a problem. Can you explain in more detail?" and provides it to the staff member through the smart glasses.

[0360] An example of a specific prompt is, "Capture the customer's voice and generate an appropriate response based on their emotion."

[0361] This invention makes it possible to provide a more natural and appropriate response in real time according to the user's emotions.

[0362] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0363] Step 1:

[0364] The user speaks to the device

[0365] What it does: A user speaks into the smart glasses, and the smart glasses' built-in microphone captures the audio.

[0366] Input: User's voice

[0367] Output: Captured audio data

[0368] Step 2:

[0369] The device converts the voice data into a digital signal

[0370] Specific operation: Audio data captured by the microphone is converted into a digital signal by an analog-to-digital converter (ADC) inside the smart glasses.

[0371] Input: Captured audio data

[0372] Output: Digital audio signal

[0373] Step 3:

[0374] The terminal sends a digital signal to an external computer

[0375] What it does: Digital audio signals are packetized and sent over the internet to a remote server.

[0376] Input: Digital audio signal

[0377] Output: Packetized digital signal

[0378] Step 4:

[0379] The server receives and decompresses the packets and converts the audio data into text data.

[0380] Specific operation: The server receives and decompresses the packetized digital signal and converts it into text data using a speech recognition engine.

[0381] Input: Packetized digital signal

[0382] Output: Text data

[0383] Step 5:

[0384] The server inputs the text data into the emotion engine and analyzes the user's emotions.

[0385] Specific operation: The received text data is input into the emotion engine, which recognizes the user's emotion based on multiple emotion categories. The emotion engine outputs the results in the form of numbers and tags.

[0386] Input: Text data

[0387] Output: Emotion information (number, tag)

[0388] Step 6:

[0389] The server uses the text data and emotional information to generate an appropriate response.

[0390] Specific operation: The server inputs text data and emotional information into a large-scale language model (LLM) and generates an appropriate response based on the user's input.

[0391] Input: Text data, emotion information

[0392] Output: Response data

[0393] Step 7:

[0394] The response data generated by the server is converted into voice data, repacketized, and sent to the terminal.

[0395] Specific operation: The response data is converted into voice data by a text-to-speech engine (TTS), repacketized, and sent to the terminal via the Internet.

[0396] Input: Response data

[0397] Output: Packetized audio data

[0398] Step 8:

[0399] The device recovers the audio data it receives and plays it through the built-in speaker.

[0400] Specific operation: The smart glasses decompress the packetized audio data they receive and play the audio through their built-in speakers.

[0401] Input: Packetized audio data

[0402] Output: Playback of the recovered audio data

[0403] The user is then able to interact through the responses that are played back.

[0404] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0405] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0406] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0407] [Second embodiment]

[0408] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0409] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0410] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0411] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0412] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0413] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0414] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0415] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0416] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0417] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0418] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0419] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0420] The present invention is a system that captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. Below, the processing of the program of this system will be explained in natural language.

[0421] Audio capture and transmission

[0422] When a user speaks, the device uses a microphone to capture the voice. The captured voice data is converted into a digital signal and sent to an external computer via the Internet. Here, the data is divided into packets for transmission, ensuring efficient and reliable data transmission.

[0423] Speech recognition and text conversion

[0424] The server receives the voice data packets sent from the device and restores the original voice data. The restored voice data is analyzed by a voice recognition engine and converted into corresponding text data. The voice recognition engine analyzes the voice data in detail and performs highly accurate text conversion.

[0425] Response Generation

[0426] The server inputs the generated text data into a large-scale language model (LLM) to generate an appropriate response. LLM is a natural language processing model with advanced analytical capabilities, allowing it to generate appropriate and natural responses to user questions and instructions. For example, if a user types, "What's the weather forecast?", the LLM will generate a response such as, "Today's weather is sunny, and the temperature is 25 degrees."

[0427] Transcribing and retransmitting responses

[0428] The response data generated by the server is converted into voice data by a text-to-speech engine. This voice data is also packetized and sent to the terminal over the Internet. The packetization and decompression process minimizes data loss and delays.

[0429] Audio playback

[0430] The device then decompresses the received voice data and plays it back to the user through the built-in speaker, allowing the user to receive a response to their question or command as voice. For example, the device provides a natural response to the user, such as "Today's weather is sunny and the temperature is 25 degrees."

[0431] Specific examples

[0432] For example, consider the case where a user speaks to a device, "What's the weather going to be like tomorrow?" In this case, the device captures the user's voice and sends it to the server. The server converts the voice to text, generating the text data "What's the weather going to be like tomorrow?". It then uses LLM to generate an appropriate response, such as "Tomorrow's weather will be rainy and the temperature will be 20 degrees." This response is converted to audio and sent to the device, which ultimately plays it back to the user.

[0433] As described above, this system enables natural conversation between the user and AI. It reduces the load on the edge device and achieves both advanced response generation and real-time performance. This system provides strong support for collaboration between edge devices and external computers, providing users with a high-quality voice interface.

[0434] The processing flow will be explained below.

[0435] Step 1:

[0436] The user performs voice input. The user speaks to the device, saying, "Tell me the weather forecast."

[0437] Step 2:

[0438] The device captures the audio. The device's microphone collects the user's voice and converts the analog audio into a digital signal.

[0439] Step 3:

[0440] The device sends a digital signal to the server, which then divides the signal into packets and sends them over the network to the server.

[0441] Step 4:

[0442] The server receives the packets and restores the digital signal. The server reconstructs the received packets and stores the original digital signal in a buffer.

[0443] Step 5:

[0444] The server uses a speech recognition engine to convert the digital signal into text data, which the server then analyzes and generates the text data "What's the weather forecast?"

[0445] Step 6:

[0446] The server uses a large-scale language model (LLM) to analyze the text data and generate a response. The server inputs the text data "What is the weather forecast?" into the LLM and generates the response text "Today's weather is sunny and the temperature is 25 degrees."

[0447] Step 7:

[0448] The server uses a text-to-speech engine to convert the response text into audio data. The server converts the response text, "Today's weather is sunny and the temperature is 25 degrees," into audio.

[0449] Step 8:

[0450] The server packets the audio data and sends it to the terminal. The server divides the audio data into packets and sends them to the terminal via the network.

[0451] Step 9:

[0452] The terminal receives the packets and reconstructs the voice data. The terminal reconstructs the received packets and restores the original voice data.

[0453] Step 10:

[0454] The device plays the audio data, and the audio response "Today's weather is sunny, and the temperature is 25 degrees" is played to the user through the device's built-in speaker.

[0455] Example 1

[0456] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0457] Conventional voice recognition and response generation systems impose a heavy computational load on edge devices and suffer from problems such as network delays and data loss. Furthermore, achieving real-time and highly accurate response generation requires large-scale computational resources, which entails high costs. The present invention aims to solve these problems and provide a fast, high-quality voice response interface while reducing the load on edge devices.

[0458] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0459] In this invention, the server includes means including a speech recognition engine for converting voice data into text data, means for analyzing the text data and using a large-scale language model to generate response data based on the text data, and means including a speech synthesis engine for converting the generated response data into voice data, thereby reducing the load on the edge device and enabling highly accurate response generation in real time while minimizing problems such as network delays and data loss.

[0460] "User" refers to a person who utilizes the speech recognition and response generation system of the present invention.

[0461] "Terminal" refers to a device that captures voice input from a user and transmits a digital signal to a server. Examples include smartphones and smart speakers.

[0462] The term "server" refers to a computer that receives data sent from a terminal and performs speech recognition, response generation, and speech synthesis.

[0463] "Voice input" refers to information spoken by a user into a terminal.

[0464] "Digital signal" refers to a signal obtained by converting analog audio data into digital format.

[0465] A "network" refers to a communications infrastructure that interconnects multiple computers and terminals. A specific example is the Internet.

[0466] The "external computer" refers to a computer such as a server that receives and processes data sent from a terminal.

[0467] "Audio data" refers to data that represents an audio signal in digital form.

[0468] "Text data" refers to character information converted from voice data by a voice recognition engine.

[0469] A "voice recognition engine" refers to software or hardware that takes voice data as input, analyzes it, and converts it into text data.

[0470] A "speech synthesis engine" refers to software or hardware that takes text data as input and generates speech data.

[0471] A "large-scale language model" is a natural language processing model that uses deep learning and learns from large amounts of text data to generate and analyze text.

[0472] A "packet" refers to the small units of data that are divided when sending and receiving data over a network.

[0473] "Packetization" refers to the process of breaking down digital signals or data into smaller units (packets) and converting them into a form that can be sent over a network.

[0474] "Reconstruction" refers to the process of reconstructing the received packets into the original voice data.

[0475] "Playback" refers to the act of outputting audio data from a speaker device so that the user can hear it.

[0476] The present invention is a system that receives voice input from a user, processes the input, generates a response, and provides it to the user again in voice. How this system is specifically implemented will be described below.

[0477] Audio capture and conversion to digital signal

[0478] When a user speaks, the device captures the voice using a built-in or connected microphone. Examples include smartphones and smart speakers. The captured voice is converted into a digital signal. Here, the device uses, for example, an ADC (analog-to-digital converter).

[0479] Digital signal transmission

[0480] The converted digital signal is divided into packets and sent over a network (e.g., the Internet) to an external computer, i.e., a server, using a network protocol such as TCP / IP.

[0481] Processing on the server

[0482] The server receives the packets sent from the device, restores them to their original digital signals, and then converts them into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text or IBM Watson Speech to Text).

[0483] Response Generation

[0484] The text data is analyzed using a large-scale language model (e.g., OpenAI's GPT-4) to generate response data, which is then made appropriate and highly accurate using natural language processing technology.

[0485] Conversion to audio data and transmission

[0486] The generated response data is converted into voice data using a speech synthesis engine (e.g., Amazon Polly or Google Cloud Text-to-Speech). The converted voice data is again divided into packets and sent to the device over the network.

[0487] Audio playback

[0488] The terminal receives the voice data packets sent from the server and restores them to the original voice data. The restored voice data is then played back to the user through the speaker, allowing the user to receive voice responses to their questions and instructions.

[0489] Specific examples

[0490] For example, consider the case where a user speaks to a terminal and asks, "What's the weather going to be like tomorrow?"

[0491] 1. The device captures the user's voice, converts it into a digital signal, and sends it to the server.

[0492] 2. The server receives the voice and uses a speech recognition engine to generate text data such as "What's the weather like tomorrow?"

[0493] 3. The generated text data is analyzed using a large-scale language model, and the response data, "Tomorrow's weather will be rainy and the temperature will be 20 degrees," is generated.

[0494] 4. This response data is converted into voice data using a voice synthesis engine and sent to the terminal.

[0495] 5. The device plays the received voice data, and the user hears the response, "Tomorrow's weather will be rainy and the temperature will be 20 degrees."

[0496] Prompt Sentence Examples

[0497] User: What's the weather like tomorrow?

[0498] Device: (Capture audio and send to server)

[0499] Server: (Receives voice data and converts it to text)

[0500] Server: (Generates response using LLM)

[0501] Server: (Converts the response into audio and sends it to the device)

[0502] Terminal: (receives and plays audio data)

[0503] Through the above procedure, the system of the present invention can realize a natural voice interface with the user, and can provide highly accurate responses while reducing the load on the edge device.

[0504] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0505] Step 1: Capture audio

[0506] The user makes a voice input (e.g., "What's the weather going to be like tomorrow?"). The microphone built into the device captures this voice and converts the analog voice signal into a digital signal. Specifically, the microphone picks up the voice and converts it into digital data via an ADC (analog-to-digital converter).

[0507] (Input) User's voice

[0508] (Output) Digital audio signal

[0509] Step 2: Sending a digital signal

[0510] The terminal breaks the converted digital voice signal into packets, and then uses a network protocol (e.g., TCP / IP) to send the data over the Internet to a server.

[0511] (Input) Digital audio signal

[0512] (Output) Packet data sent over the network

[0513] Step 3: Receiving and restoring audio data

[0514] The server receives the packet data sent from the terminal, reassembles the received packets, and restores the original digital audio signal.

[0515] (Input) Packet data

[0516] (Output) Reconstructed digital audio signal

[0517] Step 4: Speech recognition and text conversion

[0518] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to analyze the digital voice signal and convert it into corresponding text data. Specifically, the server inputs the voice data into the speech recognition engine and obtains the text data as its output.

[0519] (Input) Digital audio signal

[0520] (Output) Text data (e.g., "What's the weather like tomorrow?")

[0521] Step 5: Parsing text data and generating responses

[0522] The server inputs the generated text data into a large-scale language model (e.g., OpenAI's GPT-4) to generate an appropriate response. Specifically, the text data is input into the LLM, and the response text is obtained as the output (e.g., "Tomorrow's weather will be rainy, and the temperature will be 20 degrees").

[0523] (Input) Text data (e.g., "What's the weather like tomorrow?")

[0524] (Output) Response text (e.g. "Tomorrow's weather will be rainy and the temperature will be 20 degrees")

[0525] Step 6: Convert response text to speech

[0526] The server inputs the generated response text into a speech synthesis engine (e.g., Amazon Polly) and converts it into voice data. Specifically, the server inputs the response text and obtains voice data as output.

[0527] (Input) Response text (e.g., "Tomorrow's weather will be rainy and the temperature will be 20 degrees.")

[0528] (Output) Audio data

[0529] Step 7: Sending audio data

[0530] The server divides the voice data into packets and transmits them to the terminal via the network. Specifically, the server packetizes the voice data and transmits them using a network protocol.

[0531] (Input) Audio data

[0532] (Output) Packet data sent over the network

[0533] Step 8: Receiving and Playing Audio Data

[0534] The terminal receives the packet data sent from the server, restores it to the original voice data, and then plays the restored voice data through the built-in speaker.

[0535] (Input) Packet data

[0536] (Output) Audio to be played (e.g. "Tomorrow's weather will be rainy and the temperature will be 20 degrees")

[0537] This is the flow of the program processing for this system, which allows users to use the voice interface in a natural way.

[0538] (Application example 1)

[0539] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0540] Current voice input systems require real-time responses and highly accurate recognition when users voice-order or customize their food. However, in many cases, the load on edge devices is high, resulting in delayed responses. Another problem is that low-accuracy recognition may occur, resulting in results that differ from the user's request. These issues are particularly serious in fields such as food delivery, which require fast and accurate processing. The present invention aims to solve these issues and provide a highly accurate, real-time voice input system while reducing the load on edge devices.

[0541] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0542] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into a digital signal, means for transmitting the digital signal to an external computer, means for inputting text data into a large-scale language model and generating response data, means for converting the generated response data into voice data, means for transmitting the converted voice data to a terminal, means for the user to place an order by voice input, means for confirming the order details in real time, and means for giving voice instructions for customizing dishes. This allows the user to receive highly accurate voice responses in real time. Furthermore, since orders and customizations are accurately executed based on the user's voice input, an advanced user experience can be provided.

[0543] A "means for receiving audio input from a user" is a device or method for capturing audio produced by a user.

[0544] A "means for converting audio input to a digital signal" is a device or method for converting analog audio data into digital form.

[0545] "Means for transmitting digital signals to an external computer" refers to a device or method for transmitting digitized voice data to an external server via a network such as the Internet.

[0546] The "means for converting digital signals into text data using an external computer" refers to a device or method, such as a voice recognition engine, for converting transmitted digital voice data into text information.

[0547] The "means for inputting the generated text data into a large-scale language model and generating response data" refers to a device or method for inputting text data obtained by a speech recognition engine into a natural language processing model and generating an appropriate response to a user's inquiry or instruction.

[0548] The "means for converting the generated response data into voice data" refers to a device or method for converting the generated text-format response data into synthetic voice.

[0549] The "means for transmitting the converted voice data to the terminal" refers to a device or method for retransmitting the synthesized voice data to the user terminal over the network.

[0550] "Means for reproducing audio data at a terminal" refers to a device or method for reproducing audio data transmitted to a user terminal using an audio device such as a speaker.

[0551] The "means for users to order by voice input" refers to a device or method that allows users to order products or services by voice.

[0552] "Means for confirming order details in real time" refers to a device or method for analyzing and confirming order details in real time based on voice input.

[0553] "Means for giving voice instructions for customizing dishes" refers to a device or method that allows a user to give voice instructions for customizing desired dishes, and that the system understands and reflects these instructions.

[0554] The present invention is a system that captures voice input from users with an edge device and transmits the voice data to an external computer for processing. The system enables voice ordering, real-time confirmation, and customization instructions in the field of food delivery.

[0555] User voice input

[0556] When a user places an order by voice using a food delivery application, the smartphone's microphone is used to capture voice input. For example, the user might say, "I'd like to order one pizza." This voice is converted into a digital signal and sent over the Internet to an external computer, or server.

[0557] Speech data conversion and response generation

[0558] The server converts the received digital voice signal into text data. High-precision text conversion is performed using a speech recognition engine such as Google Cloud Speech-to-Text. The converted text data is input into a large-scale language model (LLM), which generates response data based on the user's question or instruction. Using a generative AI model, a natural response such as, "We've received your pizza order. Is there anything else you'd like to order?" can be obtained.

[0559] Transcribing and retransmitting response data

[0560] The generated response data is converted into voice data using a speech synthesis engine such as Google Text-to-Speech (gTTS), which is then sent back to the user's smartphone via the Internet.

[0561] Audio playback on the user device

[0562] The user's smartphone then plays the received voice data through its built-in speaker, allowing the user to receive natural voice responses to confirm their order or to ask additional questions. For example, a user can customize their order by voice input, such as "more tomatoes, less cheese, please," and the response will be returned in real time.

[0563] Specific examples

[0564] If the user says the following prompt:

[0565] "I'd like to order a pizza."

[0566] "More tomatoes, less cheese, please."

[0567] Based on the prompt above, the system will proceed through the following steps:

[0568] 1. Converts audio input into a digital signal.

[0569] 2. Send the digital signal to the server.

[0570] 3. The server converts the voice signal into text.

[0571] 4. Generate response data using a large-scale language model.

[0572] 5. Convert the response data into audio data.

[0573] 6. The voice data is sent to the user terminal.

[0574] 7. The user device plays the received audio.

[0575] This system allows users to smoothly order food delivery using voice. The main hardware used is a smartphone and a microphone. The software includes Google Cloud Speech-to-Text and Google Text-to-Speech (gTTS), which enables a high-quality voice interface.

[0576] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0577] Step 1:

[0578] A user speaks to a smartphone to place an order or ask a question. For example, they might say, "I'd like to order one pizza." This voice data is captured by a microphone. The input is an analog voice signal, and the output is the captured digital voice data. Specifically, the smartphone's microphone captures the voice and the built-in ADC (analog-to-digital converter) converts it into a digital signal.

[0579] Step 2:

[0580] The terminal transmits the captured digital audio data to a server via the Internet. The input is the captured digital audio data, and the output is the audio data transmitted via the Internet. Specifically, the terminal generates an HTTP request, divides the audio data into packets, and transmits them to the server.

[0581] Step 3:

[0582] The server reconstructs the received voice data and converts it into text data using a speech recognition engine. The input is the transmitted digital voice data, and the output is the converted text data. Specifically, the server restores the received packets to the original voice data and converts the voice data into text using Google Cloud Speech-to-Text.

[0583] Step 4:

[0584] The server uses a large-scale language model (LLM) to analyze the text data and generate response data. The input is the converted text data (e.g., "I would like to order one pizza"), and the output is the response data (e.g., "We have received your pizza order. Is there anything else you would like to order?"). Specifically, the server inputs a prompt sentence into the generative AI model and generates an appropriate response.

[0585] Step 5:

[0586] The server converts the generated response data into voice data. The input is the response data (in text format), and the output is voice data. Specifically, Google Text-to-Speech (gTTS) converts the text into synthetic voice.

[0587] Step 6:

[0588] The server divides the voice data into packets again and sends them to the user terminal via the Internet. The input is voice data, and the output is voice data sent via the Internet. Specifically, the server divides the voice data into packets and sends them as an HTTP response.

[0589] Step 7:

[0590] The terminal restores the received voice data and plays it back to the user through the built-in speaker. The input is the transmitted voice data, and the output is the played voice. Specifically, the terminal restores the received packets to the original voice data and plays back a response using the speaker. For example, the response played back to the user is, "We've received your pizza order. Is there anything else you'd like to order?"

[0591] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0592] This invention is a system that captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. Furthermore, the external computer is equipped with an emotion engine that recognizes the user's emotions and generates responses linked to the emotions, enabling more natural and emotionally appropriate dialogue.

[0593] Audio capture and transmission

[0594] When a user speaks, the device captures the voice using a microphone. The captured voice data is converted into a digital signal and sent to a server over the Internet. Here, the data is divided into packets for transmission, ensuring efficient and reliable data transmission.

[0595] Speech recognition and text conversion

[0596] The server receives the voice data packets sent from the device and restores the original voice data. The restored voice data is analyzed by a voice recognition engine and converted into corresponding text data. The voice recognition engine analyzes the voice data in detail and performs highly accurate text conversion.

[0597] emotion recognition

[0598] Next, the server inputs the text data into the emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from the input text data based on multiple emotion categories and outputs them in the form of numbers or tags. For example, if a user says, "I'm feeling terrible today," the emotion engine will classify it into emotion categories such as "negative" or "frustration."

[0599] Response Generation

[0600] The server inputs the generated text data and emotional information into a large-scale language model (LLM) to generate an appropriate response. LLM is a natural language processing model with advanced analytical capabilities, allowing it to generate an appropriate response tailored to the user's emotions. For example, if a user inputs "I'm feeling terrible today," the LLM will generate a response in the form of "I'm sorry, that's terrible. Is there anything I can help you with?"

[0601] Transcribing and retransmitting responses

[0602] The response data generated by the server is converted into voice data by a text-to-speech engine. This voice data is also packetized and sent to the terminal over the Internet. The packetization and decompression process minimizes data loss and delays.

[0603] Audio playback

[0604] The device decompresses the received voice data and plays it back to the user through the built-in speaker. This allows the user to receive a response to their question or instruction as voice. For example, a response such as "I'm sorry, that's a problem. Is there anything I can help you with?" is provided to the user in a natural way.

[0605] Specific examples

[0606] For example, consider a situation where a user says to a device, "I'm feeling incredibly miserable today." In this case, the device captures the user's voice and sends it to the server. The server converts the voice to text, generating the text data "I'm feeling incredibly miserable today." The emotion engine then recognizes emotions such as "negative" or "sad" from this text data. It then uses the LLM to generate an appropriate response such as "That's really miserable. Let's talk a bit and hopefully make you feel better." This response is converted into audio and sent to the device, which ultimately plays it back to the user.

[0607] As described above, this system enables natural conversation between the user and AI. It reduces the load on the edge device and simultaneously generates sophisticated, emotionally relevant responses in real time. This system provides strong support for collaboration between edge devices and external computers, providing users with a high-quality voice interface.

[0608] The processing flow will be explained below.

[0609] Step 1:

[0610] The user performs voice input. For example, the user speaks to the terminal, "Today is so hard."

[0611] Step 2:

[0612] The device captures the audio. The device's microphone collects the user's voice and converts the analog audio into a digital signal.

[0613] Step 3:

[0614] The terminal sends a digital signal to the server, which then divides the digital signal into packets and sends them over the network to the server.

[0615] Step 4:

[0616] The server receives the packets and restores the digital signal. The server reconstructs the received packets and stores the restored digital signal in a buffer.

[0617] Step 5:

[0618] The server uses a speech recognition engine to convert the digital signal into text data. The server analyzes the digital signal and generates the text data "Today is extremely difficult."

[0619] Step 6:

[0620] The server inputs the text data into the emotion engine and analyzes the emotions. The server recognizes emotions such as "negative" or "sad" from the text data and outputs the results in the form of numbers or tags.

[0621] Step 7:

[0622] The server inputs text data and emotional information into a large-scale language model (LLM) to generate a response. The server passes the text "I'm feeling incredibly miserable today" along with the emotional information "negative" and "sad" to the LLM, generating a response such as "That's really hard on you. I hope we can talk a bit and make you feel better."

[0623] Step 8:

[0624] The server converts the response text into audio data using a text-to-speech engine. The server converts the generated response text into audio data.

[0625] Step 9:

[0626] The server packets the voice data and sends it to the terminal. The server divides the voice data into packets and sends them to the terminal over the network.

[0627] Step 10:

[0628] The terminal receives the packets and reconstructs the voice data. The terminal reconstructs the received packets and restores the original voice data.

[0629] Step 11:

[0630] The device plays the audio data, and through the device's built-in speaker, plays a voice response to the user saying, "That must be really hard on you. Let's talk a bit and hopefully you'll feel better."

[0631] Example 2

[0632] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0633] Conventional speech recognition systems simply convert a user's speech into text and generate simple responses based on that text. This makes it difficult to understand the user's emotions and generate responses that take those emotions into consideration, resulting in a less natural and emotionally rich dialogue. In particular, in situations where the user is emotionally charged, the system is unable to respond appropriately, making it difficult to provide a satisfying user experience. The present invention aims to solve this problem by providing a system that analyzes a user's emotions and provides natural-sounding voice responses based on those emotions.

[0634] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing the user's emotions and generating response data based on the emotions, a means for converting a digital signal into text data, and a means for analyzing the text data and generating response data based on the text data. This makes it possible to generate natural responses that are in line with the user's emotions, enabling more satisfying interactions.

[0635] "User" refers to a person who interacts with the system by inputting voice through a terminal.

[0636] "Terminal" refers to a device that receives voice input from a user, converts it into a digital signal, transmits it to a server, and plays back a voice response from the server to the user.

[0637] "Voice input" refers to the voice that the user utters to the terminal, and is the voice signal that is processed by the system.

[0638] A "digital signal" is a signal obtained by converting an audio input into a digital format, and refers to data in a format that can be processed by a computer.

[0639] "External computer" refers to a computing device that processes digital signals sent from the terminal and performs voice recognition, emotion analysis, and response generation.

[0640] "Text data" refers to character string data generated by an external computer analyzing voice data.

[0641] An "emotion recognition engine" refers to a software function that analyzes text data to recognize a user's emotions and output them in numerical or tag format.

[0642] "Response data" refers to data containing appropriate responses to users, generated using an emotion recognition engine or a large-scale language model.

[0643] "Large-scale language model (LLM)" refers to an advanced language model that uses natural language processing to generate appropriate responses from text data.

[0644] "Text-to-speech engine" means the software functionality for converting generated response data into audio data.

[0645] "Packet" refers to the unit of data into which digital signals are divided and sent over the Internet.

[0646] A "microphone" refers to an audio device for capturing the user's voice.

[0647] "Speaker" refers to an audio device that plays back audio data sent from the server and lets the user hear it.

[0648] This clarifies each element of the system and its function.

[0649] The present invention is a system that captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. Furthermore, the external computer is equipped with an emotion engine that recognizes the user's emotions and generates responses linked to the emotions, enabling more natural and emotionally appropriate dialogue.

[0650] Audio capture and transmission

[0651] When a user speaks, the device captures the voice using a microphone. For example, the user might say, "I'm feeling incredibly miserable today." This voice is converted into a digital signal and sent to a server over the Internet. The voice data is divided into packets for transmission, ensuring efficient and reliable transmission.

[0652] Speech recognition and text conversion

[0653] The server receives the voice data packets sent from the device and restores the original voice data. The restored voice data is analyzed by a voice recognition engine (e.g., an engine using a deep learning model) and converted into corresponding text data. The voice recognition engine analyzes the voice data in detail and performs highly accurate text conversion.

[0654] emotion recognition

[0655] Next, the server inputs the text data into an emotion engine (e.g., an engine using natural language processing technology) to analyze the user's emotions. The emotion engine recognizes the user's emotions from the input text data based on multiple emotion categories and outputs them in the form of numbers or tags. For example, if a user says, "I'm feeling terrible today," the emotion engine will classify it into emotion categories such as "negative" or "frustration."

[0656] Response Generation

[0657] The server inputs the generated text data and emotional information into a large-scale language model (LLM) to generate an appropriate response. LLM is a natural language processing model with advanced analytical capabilities, so it generates an appropriate response that matches the user's emotions. For example, if a user inputs "I'm feeling terrible today," the LLM will generate a response in the form of "I'm sorry, that's terrible. Is there anything I can help you with?" In this case, the LLM can be given a prompt sentence such as:

[0658] "Example prompt statement:"

[0659] User: I'm having a really hard time today

[0660] System: That's really tough. Let's talk a bit and hopefully it will help ease your mind.

[0661] Transcribing and retransmitting responses

[0662] The response data generated by the server is converted into voice data by a text-to-speech engine (e.g., a text-to-speech engine using natural language processing). This voice data is also packetized and sent to the terminal over the Internet. The packetization and decompression process minimizes data loss and delays.

[0663] Audio playback

[0664] The device decompresses the received voice data and plays it back to the user through the built-in speaker. This allows the user to receive a response to their question or instruction as voice. For example, a response such as "I'm sorry, that's a problem. Is there anything I can help you with?" is provided to the user in a natural way.

[0665] These components and processing steps enable dialogue that reflects the user's emotions, providing a more human-like and natural voice interface, which reduces the load on edge devices and enables both emotionally-sensitive and real-time response generation.

[0666] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0667] Step 1: Capture audio

[0668] Input: User's spoken utterance

[0669] How it works: The user speaks into the device's microphone, which captures the audio signal.

[0670] Output: Digitized audio data

[0671] Specific operation: The device's microphone converts analog audio signals into digital signals and temporarily stores the data in the device's memory.

[0672] Step 2: Sending audio data

[0673] Input: Digitized audio data

[0674] How it works: The device splits the captured audio data into packets and sends them over the Internet to a server.

[0675] Output: Packetized voice data sent over the internet

[0676] Specific operation: Audio data is divided into packets of a fixed size, and header information is added to each packet before transmission.

[0677] Step 3: Audio data recovery

[0678] Input: Packetized audio data

[0679] Operation: The server receives packets sent from the device and restores them to the original audio data.

[0680] Output: Recovered audio data

[0681] Specific operation: The received packets are rearranged in order and combined to restore the original audio data.

[0682] Step 4: Speech recognition and text conversion

[0683] Input: Recovered audio data

[0684] How it works: The server inputs the recovered voice data into a speech recognition engine and converts it into highly accurate text data.

[0685] Output: Text data

[0686] Specific operation: A speech recognition engine (e.g., an engine using a deep learning model) analyzes the voice data and generates text data such as "Today is incredibly difficult."

[0687] Step 5: Emotion Recognition

[0688] Input: Generated text data

[0689] How it works: The server inputs text data into an emotion recognition engine to analyze the user's emotions.

[0690] Output: Emotion information (in numerical and tag formats)

[0691] Specific operation: An emotion recognition engine using natural language processing technology tags emotions in the text and assigns labels such as "negative" or "sad."

[0692] Step 6: Response Generation

[0693] Input: Text data and emotion information

[0694] How it works: The server inputs text data and emotion information into a large-scale language model (LLM) to generate an appropriate response.

[0695] Output: Response data

[0696] Specific operation: The user's utterances and emotional data are input as prompts into a large-scale language model (e.g., GPT-4), and a response such as "That's really painful. Let's talk for a bit and hopefully you'll feel better" is generated.

[0697] Step 7: Transcribing responses

[0698] Input: Response data

[0699] How it works: The server inputs the generated response data into a text-to-speech engine and converts it into audio data.

[0700] Output: Audio data

[0701] Specific operation: A text-to-speech engine (e.g., an engine using natural language processing) converts the response data into audio data.

[0702] Step 8: Resend the response audio

[0703] Input: Audio data

[0704] How it works: The server breaks the audio data back into packets and sends them over the Internet to the device.

[0705] Output: Packetized audio data

[0706] Specific operation: Audio data is packetized and sent to the terminal. Packets contain sequence information, making it easy to restore the data at the terminal.

[0707] Step 9: Playing Audio

[0708] Input: Packetized audio data

[0709] Operation: The device reconstructs the received audio packets and plays them through the built-in speaker.

[0710] Output: A spoken response to the user

[0711] Specific operation: The device receives the packets and restores them as voice data. The restored voice data is played over the speaker, providing a response to the user. For example, "That must be really painful. I hope that talking about it will help you feel better."

[0712] (Application example 2)

[0713] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0714] Conventional speech recognition systems have difficulty in accurately recognizing user emotions and generating responses based on them. As a result, especially in customer service and complaint handling, systems can only provide mechanical responses that ignore the user's emotions, making it difficult to improve customer satisfaction. In addition, edge devices themselves have limited processing power, making it difficult to perform highly accurate emotion recognition and response generation in real time.

[0715] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into a digital signal, means for transmitting the digital signal to an external computer, means for converting the digital signal into text data in the external computer, means for analyzing the text data and generating response data based on the text data, means for converting the generated response data into voice data, means for transmitting the converted voice data to a terminal, means for playing the voice data on the terminal, and means for recognizing the user's emotions in a conversation with a customer and generating an appropriate response based on the emotions. This makes it possible to provide more natural and appropriate responses in real time that correspond to the user's emotions.

[0716] "User" refers to any person or subject that uses the system to provide voice input.

[0717] The "means for receiving audio input" refers to a device for capturing audio data from a user, and specifically refers to an audio capture device such as a microphone.

[0718] "Means for converting audio input into a digital signal" refers to a process or device that converts received audio into electronic digital data, typically an analog-to-digital converter (ADC).

[0719] "Means for transmitting digital signals to an external computing device" refers to means for transmitting digital data over the Internet or other communications network to a remote server or computing resource.

[0720] "External computers" refer to servers or cloud computing environments that exist separately from edge devices and are used for advanced computational processing and analysis.

[0721] "Means for converting digital signals into text data" refers to software or a system that uses a speech recognition engine to convert digital voice data into a corresponding text format.

[0722] "Means for analyzing text data and generating response data based on it" refers to a program or system for interpreting received text data and generating optimal responses or instructions in response to it.

[0723] The "means for converting the generated response data into voice data" refers to a text-to-speech engine (TTS) for outputting the text-format response data as voice again.

[0724] "Means for transmitting converted voice data to a terminal" refers to a communication means for returning the generated voice data to the edge device.

[0725] "Means for reproducing audio data on a terminal" refers to means for reproducing received audio data through an audio device such as a speaker.

[0726] "Means for recognizing a user's emotions in customer interactions and generating appropriate responses based on those emotions" refers to large-scale language models and other artificial intelligence technologies that use emotion recognition engines to understand a user's emotions and generate natural-sounding responses accordingly.

[0727] MODE FOR CARRYING OUT THE INVENTION

[0728] A system embodying this invention captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. A specific implementation of this system is described below.

[0729] Audio capture and transmission

[0730] When a user speaks, the smart glasses used as a terminal capture the voice using a microphone. The captured voice data is converted into a digital signal by a voice capture device. This converted digital signal is transmitted to an external computer server via the Internet. Specifically, the data is transmitted in packets, enabling efficient data transmission.

[0731] Speech recognition and text conversion

[0732] The server receives the voice data packets sent from the device and restores them to the original voice data. The restored voice data is analyzed using a voice recognition engine and converted into corresponding text data. The voice recognition engine is typically a highly accurate software.

[0733] emotion recognition

[0734] Next, the server inputs the text data into an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from the input text data based on multiple emotion categories. This emotion information is output in the form of numbers and tags.

[0735] Response Generation

[0736] The server uses the generated text data and emotion information to generate appropriate responses using a large-scale language model (LLM), which has the ability to provide natural responses that are in line with the user's input.

[0737] Transcribing and retransmitting responses

[0738] The generated response data is converted into voice data by a text-to-speech engine, which is also packetized and transmitted to the smart glasses via the internet.

[0739] Audio playback

[0740] The smart glasses, which are the terminals, decompress the received voice data and play it back to the user through the built-in speaker, allowing the user to receive voice responses to their questions and instructions.

[0741] For example, if a store staff member asks, "Hey, is there something I can help you with?" and the customer replies, "Hey, I don't know how to use this product," the system captures the voice and sends it to the server. The server analyzes the voice and recognizes the emotion of "the customer is having trouble." It then generates an appropriate response, such as "That's a problem. Can you explain in more detail?" and provides it to the staff member through the smart glasses.

[0742] An example of a specific prompt is, "Capture the customer's voice and generate an appropriate response based on their emotion."

[0743] This invention makes it possible to provide a more natural and appropriate response in real time according to the user's emotions.

[0744] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0745] Step 1:

[0746] The user speaks to the device

[0747] What it does: A user speaks into the smart glasses, and the smart glasses' built-in microphone captures the audio.

[0748] Input: User's voice

[0749] Output: Captured audio data

[0750] Step 2:

[0751] The device converts the voice data into a digital signal

[0752] Specific operation: Audio data captured by the microphone is converted into a digital signal by an analog-to-digital converter (ADC) inside the smart glasses.

[0753] Input: Captured audio data

[0754] Output: Digital audio signal

[0755] Step 3:

[0756] The terminal sends a digital signal to an external computer

[0757] What it does: Digital audio signals are packetized and sent over the internet to a remote server.

[0758] Input: Digital audio signal

[0759] Output: Packetized digital signal

[0760] Step 4:

[0761] The server receives and decompresses the packets and converts the audio data into text data.

[0762] Specific operation: The server receives and decompresses the packetized digital signal and converts it into text data using a speech recognition engine.

[0763] Input: Packetized digital signal

[0764] Output: Text data

[0765] Step 5:

[0766] The server inputs the text data into the emotion engine and analyzes the user's emotions.

[0767] Specific operation: The received text data is input into the emotion engine, which recognizes the user's emotion based on multiple emotion categories. The emotion engine outputs the results in the form of numbers and tags.

[0768] Input: Text data

[0769] Output: Emotion information (number, tag)

[0770] Step 6:

[0771] The server uses the text data and emotional information to generate an appropriate response.

[0772] Specific operation: The server inputs text data and emotional information into a large-scale language model (LLM) and generates an appropriate response based on the user's input.

[0773] Input: Text data, emotion information

[0774] Output: Response data

[0775] Step 7:

[0776] The response data generated by the server is converted into voice data, repacketized, and sent to the terminal.

[0777] Specific operation: The response data is converted into voice data by a text-to-speech engine (TTS), repacketized, and sent to the terminal via the Internet.

[0778] Input: Response data

[0779] Output: Packetized audio data

[0780] Step 8:

[0781] The device recovers the audio data it receives and plays it through the built-in speaker.

[0782] Specific operation: The smart glasses decompress the packetized audio data they receive and play the audio through their built-in speakers.

[0783] Input: Packetized audio data

[0784] Output: Playback of the recovered audio data

[0785] The user is then able to interact through the responses that are played back.

[0786] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0787] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0788] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0789] [Third embodiment]

[0790] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0791] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0792] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0793] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0794] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0795] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0796] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0797] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0798] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0799] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0800] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0801] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0802] The present invention is a system that captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. Below, the processing of the program of this system will be explained in natural language.

[0803] Audio capture and transmission

[0804] When a user speaks, the device uses a microphone to capture the voice. The captured voice data is converted into a digital signal and sent to an external computer via the Internet. Here, the data is divided into packets for transmission, ensuring efficient and reliable data transmission.

[0805] Speech recognition and text conversion

[0806] The server receives the voice data packets sent from the device and restores the original voice data. The restored voice data is analyzed by a voice recognition engine and converted into corresponding text data. The voice recognition engine analyzes the voice data in detail and performs highly accurate text conversion.

[0807] Response Generation

[0808] The server inputs the generated text data into a large-scale language model (LLM) to generate an appropriate response. LLM is a natural language processing model with advanced analytical capabilities, allowing it to generate appropriate and natural responses to user questions and instructions. For example, if a user types, "What's the weather forecast?", the LLM will generate a response such as, "Today's weather is sunny, and the temperature is 25 degrees."

[0809] Transcribing and retransmitting responses

[0810] The response data generated by the server is converted into voice data by a text-to-speech engine. This voice data is also packetized and sent to the terminal over the Internet. The packetization and decompression process minimizes data loss and delays.

[0811] Audio playback

[0812] The device then decompresses the received voice data and plays it back to the user through the built-in speaker, allowing the user to receive a response to their question or command as voice. For example, the device provides a natural response to the user, such as "Today's weather is sunny and the temperature is 25 degrees."

[0813] Specific examples

[0814] For example, consider the case where a user speaks to a device, "What's the weather going to be like tomorrow?" In this case, the device captures the user's voice and sends it to the server. The server converts the voice to text, generating the text data "What's the weather going to be like tomorrow?". It then uses LLM to generate an appropriate response, such as "Tomorrow's weather will be rainy and the temperature will be 20 degrees." This response is converted to audio and sent to the device, which ultimately plays it back to the user.

[0815] As described above, this system enables natural conversation between the user and AI. It reduces the load on the edge device and achieves both advanced response generation and real-time performance. This system provides strong support for collaboration between edge devices and external computers, providing users with a high-quality voice interface.

[0816] The processing flow will be explained below.

[0817] Step 1:

[0818] The user performs voice input. The user speaks to the device, saying, "Tell me the weather forecast."

[0819] Step 2:

[0820] The device captures the audio. The device's microphone collects the user's voice and converts the analog audio into a digital signal.

[0821] Step 3:

[0822] The device sends a digital signal to the server, which then divides the signal into packets and sends them over the network to the server.

[0823] Step 4:

[0824] The server receives the packets and restores the digital signal. The server reconstructs the received packets and stores the original digital signal in a buffer.

[0825] Step 5:

[0826] The server uses a speech recognition engine to convert the digital signal into text data, which the server then analyzes and generates the text data "What's the weather forecast?"

[0827] Step 6:

[0828] The server uses a large-scale language model (LLM) to analyze the text data and generate a response. The server inputs the text data "What is the weather forecast?" into the LLM and generates the response text "Today's weather is sunny and the temperature is 25 degrees."

[0829] Step 7:

[0830] The server uses a text-to-speech engine to convert the response text into audio data. The server converts the response text, "Today's weather is sunny and the temperature is 25 degrees," into audio.

[0831] Step 8:

[0832] The server packets the audio data and sends it to the terminal. The server divides the audio data into packets and sends them to the terminal via the network.

[0833] Step 9:

[0834] The terminal receives the packets and reconstructs the voice data. The terminal reconstructs the received packets and restores the original voice data.

[0835] Step 10:

[0836] The device plays the audio data, and the audio response "Today's weather is sunny, and the temperature is 25 degrees" is played to the user through the device's built-in speaker.

[0837] Example 1

[0838] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0839] Conventional voice recognition and response generation systems impose a heavy computational load on edge devices and suffer from problems such as network delays and data loss. Furthermore, achieving real-time and highly accurate response generation requires large-scale computational resources, which entails high costs. The present invention aims to solve these problems and provide a fast, high-quality voice response interface while reducing the load on edge devices.

[0840] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0841] In this invention, the server includes means including a speech recognition engine for converting voice data into text data, means for analyzing the text data and using a large-scale language model to generate response data based on the text data, and means including a speech synthesis engine for converting the generated response data into voice data, thereby reducing the load on the edge device and enabling highly accurate response generation in real time while minimizing problems such as network delays and data loss.

[0842] "User" refers to a person who utilizes the speech recognition and response generation system of the present invention.

[0843] "Terminal" refers to a device that captures voice input from a user and transmits a digital signal to a server. Examples include smartphones and smart speakers.

[0844] The term "server" refers to a computer that receives data sent from a terminal and performs speech recognition, response generation, and speech synthesis.

[0845] "Voice input" refers to information spoken by a user into a terminal.

[0846] "Digital signal" refers to a signal obtained by converting analog audio data into digital format.

[0847] A "network" refers to a communications infrastructure that interconnects multiple computers and terminals. A specific example is the Internet.

[0848] The "external computer" refers to a computer such as a server that receives and processes data sent from a terminal.

[0849] "Audio data" refers to data that represents an audio signal in digital form.

[0850] "Text data" refers to character information converted from voice data by a voice recognition engine.

[0851] A "voice recognition engine" refers to software or hardware that takes voice data as input, analyzes it, and converts it into text data.

[0852] A "speech synthesis engine" refers to software or hardware that takes text data as input and generates speech data.

[0853] A "large-scale language model" is a natural language processing model that uses deep learning and learns from large amounts of text data to generate and analyze text.

[0854] A "packet" refers to the small units of data that are divided when sending and receiving data over a network.

[0855] "Packetization" refers to the process of breaking down digital signals or data into smaller units (packets) and converting them into a form that can be sent over a network.

[0856] "Reconstruction" refers to the process of reconstructing the received packets into the original voice data.

[0857] "Playback" refers to the act of outputting audio data from a speaker device so that the user can hear it.

[0858] The present invention is a system that receives voice input from a user, processes the input, generates a response, and provides it to the user again in voice. How this system is specifically implemented will be described below.

[0859] Audio capture and conversion to digital signal

[0860] When a user speaks, the device captures the voice using a built-in or connected microphone. Examples include smartphones and smart speakers. The captured voice is converted into a digital signal. Here, the device uses, for example, an ADC (analog-to-digital converter).

[0861] Digital signal transmission

[0862] The converted digital signal is divided into packets and sent over a network (e.g., the Internet) to an external computer, i.e., a server, using a network protocol such as TCP / IP.

[0863] Processing on the server

[0864] The server receives the packets sent from the device, restores them to their original digital signals, and then converts them into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text or IBM Watson Speech to Text).

[0865] Response Generation

[0866] The text data is analyzed using a large-scale language model (e.g., OpenAI's GPT-4) to generate response data, which is then made appropriate and highly accurate using natural language processing technology.

[0867] Conversion to audio data and transmission

[0868] The generated response data is converted into voice data using a speech synthesis engine (e.g., Amazon Polly or Google Cloud Text-to-Speech). The converted voice data is again divided into packets and sent to the device over the network.

[0869] Audio playback

[0870] The terminal receives the voice data packets sent from the server and restores them to the original voice data. The restored voice data is then played back to the user through the speaker, allowing the user to receive voice responses to their questions and instructions.

[0871] Specific examples

[0872] For example, consider the case where a user speaks to a terminal and asks, "What's the weather going to be like tomorrow?"

[0873] 1. The device captures the user's voice, converts it into a digital signal, and sends it to the server.

[0874] 2. The server receives the voice and uses a speech recognition engine to generate text data such as "What's the weather like tomorrow?"

[0875] 3. The generated text data is analyzed using a large-scale language model, and the response data, "Tomorrow's weather will be rainy and the temperature will be 20 degrees," is generated.

[0876] 4. This response data is converted into voice data using a voice synthesis engine and sent to the terminal.

[0877] 5. The device plays the received voice data, and the user hears the response, "Tomorrow's weather will be rainy and the temperature will be 20 degrees."

[0878] Prompt Sentence Examples

[0879] User: What's the weather like tomorrow?

[0880] Device: (Capture audio and send to server)

[0881] Server: (Receives voice data and converts it to text)

[0882] Server: (Generates response using LLM)

[0883] Server: (Converts the response into audio and sends it to the device)

[0884] Terminal: (receives and plays audio data)

[0885] Through the above procedure, the system of the present invention can realize a natural voice interface with the user, and can provide highly accurate responses while reducing the load on the edge device.

[0886] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0887] Step 1: Capture audio

[0888] The user makes a voice input (e.g., "What's the weather going to be like tomorrow?"). The microphone built into the device captures this voice and converts the analog voice signal into a digital signal. Specifically, the microphone picks up the voice and converts it into digital data via an ADC (analog-to-digital converter).

[0889] (Input) User's voice

[0890] (Output) Digital audio signal

[0891] Step 2: Sending a digital signal

[0892] The terminal breaks the converted digital voice signal into packets, and then uses a network protocol (e.g., TCP / IP) to send the data over the Internet to a server.

[0893] (Input) Digital audio signal

[0894] (Output) Packet data sent over the network

[0895] Step 3: Receiving and restoring audio data

[0896] The server receives the packet data sent from the terminal, reassembles the received packets, and restores the original digital audio signal.

[0897] (Input) Packet data

[0898] (Output) Reconstructed digital audio signal

[0899] Step 4: Speech recognition and text conversion

[0900] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to analyze the digital voice signal and convert it into corresponding text data. Specifically, the server inputs the voice data into the speech recognition engine and obtains the text data as its output.

[0901] (Input) Digital audio signal

[0902] (Output) Text data (e.g., "What's the weather like tomorrow?")

[0903] Step 5: Parsing text data and generating responses

[0904] The server inputs the generated text data into a large-scale language model (e.g., OpenAI's GPT-4) to generate an appropriate response. Specifically, the text data is input into the LLM, and the response text is obtained as the output (e.g., "Tomorrow's weather will be rainy, and the temperature will be 20 degrees").

[0905] (Input) Text data (e.g., "What's the weather like tomorrow?")

[0906] (Output) Response text (e.g. "Tomorrow's weather will be rainy and the temperature will be 20 degrees")

[0907] Step 6: Convert response text to speech

[0908] The server inputs the generated response text into a speech synthesis engine (e.g., Amazon Polly) and converts it into voice data. Specifically, the server inputs the response text and obtains voice data as output.

[0909] (Input) Response text (e.g., "Tomorrow's weather will be rainy and the temperature will be 20 degrees.")

[0910] (Output) Audio data

[0911] Step 7: Sending audio data

[0912] The server divides the voice data into packets and transmits them to the terminal via the network. Specifically, the server packetizes the voice data and transmits them using a network protocol.

[0913] (Input) Audio data

[0914] (Output) Packet data sent over the network

[0915] Step 8: Receiving and Playing Audio Data

[0916] The terminal receives the packet data sent from the server, restores it to the original voice data, and then plays the restored voice data through the built-in speaker.

[0917] (Input) Packet data

[0918] (Output) Audio to be played (e.g. "Tomorrow's weather will be rainy and the temperature will be 20 degrees")

[0919] This is the flow of the program processing for this system, which allows users to use the voice interface in a natural way.

[0920] (Application example 1)

[0921] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0922] Current voice input systems require real-time responses and highly accurate recognition when users voice-order or customize their food. However, in many cases, the load on edge devices is high, resulting in delayed responses. Another problem is that low-accuracy recognition may occur, resulting in results that differ from the user's request. These issues are particularly serious in fields such as food delivery, which require fast and accurate processing. The present invention aims to solve these issues and provide a highly accurate, real-time voice input system while reducing the load on edge devices.

[0923] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0924] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into a digital signal, means for transmitting the digital signal to an external computer, means for inputting text data into a large-scale language model and generating response data, means for converting the generated response data into voice data, means for transmitting the converted voice data to a terminal, means for the user to place an order by voice input, means for confirming the order details in real time, and means for giving voice instructions for customizing dishes. This allows the user to receive highly accurate voice responses in real time. Furthermore, since orders and customizations are accurately executed based on the user's voice input, an advanced user experience can be provided.

[0925] A "means for receiving audio input from a user" is a device or method for capturing audio produced by a user.

[0926] A "means for converting audio input to a digital signal" is a device or method for converting analog audio data into digital form.

[0927] "Means for transmitting digital signals to an external computer" refers to a device or method for transmitting digitized voice data to an external server via a network such as the Internet.

[0928] The "means for converting digital signals into text data using an external computer" refers to a device or method, such as a voice recognition engine, for converting transmitted digital voice data into text information.

[0929] The "means for inputting the generated text data into a large-scale language model and generating response data" refers to a device or method for inputting text data obtained by a speech recognition engine into a natural language processing model and generating an appropriate response to a user's inquiry or instruction.

[0930] The "means for converting the generated response data into voice data" refers to a device or method for converting the generated text-format response data into synthetic voice.

[0931] The "means for transmitting the converted voice data to the terminal" refers to a device or method for retransmitting the synthesized voice data to the user terminal over the network.

[0932] "Means for reproducing audio data at a terminal" refers to a device or method for reproducing audio data transmitted to a user terminal using an audio device such as a speaker.

[0933] The "means for users to order by voice input" refers to a device or method that allows users to order products or services by voice.

[0934] "Means for confirming order details in real time" refers to a device or method for analyzing and confirming order details in real time based on voice input.

[0935] "Means for giving voice instructions for customizing dishes" refers to a device or method that allows a user to give voice instructions for customizing desired dishes, and that the system understands and reflects these instructions.

[0936] The present invention is a system that captures voice input from users with an edge device and transmits the voice data to an external computer for processing. The system enables voice ordering, real-time confirmation, and customization instructions in the field of food delivery.

[0937] User voice input

[0938] When a user places an order by voice using a food delivery application, the smartphone's microphone is used to capture voice input. For example, the user might say, "I'd like to order one pizza." This voice is converted into a digital signal and sent over the Internet to an external computer, or server.

[0939] Speech data conversion and response generation

[0940] The server converts the received digital voice signal into text data. High-precision text conversion is performed using a speech recognition engine such as Google Cloud Speech-to-Text. The converted text data is input into a large-scale language model (LLM), which generates response data based on the user's question or instruction. Using a generative AI model, a natural response such as, "We've received your pizza order. Is there anything else you'd like to order?" can be obtained.

[0941] Transcribing and retransmitting response data

[0942] The generated response data is converted into voice data using a speech synthesis engine such as Google Text-to-Speech (gTTS), which is then sent back to the user's smartphone via the Internet.

[0943] Audio playback on the user device

[0944] The user's smartphone then plays the received voice data through its built-in speaker, allowing the user to receive natural voice responses to confirm their order or to ask additional questions. For example, a user can customize their order by voice input, such as "more tomatoes, less cheese, please," and the response will be returned in real time.

[0945] Specific examples

[0946] If the user says the following prompt:

[0947] "I'd like to order a pizza."

[0948] "More tomatoes, less cheese, please."

[0949] Based on the prompt above, the system will proceed through the following steps:

[0950] 1. Converts audio input into a digital signal.

[0951] 2. Send the digital signal to the server.

[0952] 3. The server converts the voice signal into text.

[0953] 4. Generate response data using a large-scale language model.

[0954] 5. Convert the response data into audio data.

[0955] 6. The voice data is sent to the user terminal.

[0956] 7. The user device plays the received audio.

[0957] This system allows users to smoothly order food delivery using voice. The main hardware used is a smartphone and a microphone. The software includes Google Cloud Speech-to-Text and Google Text-to-Speech (gTTS), which enables a high-quality voice interface.

[0958] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0959] Step 1:

[0960] A user speaks to a smartphone to place an order or ask a question. For example, they might say, "I'd like to order one pizza." This voice data is captured by a microphone. The input is an analog voice signal, and the output is the captured digital voice data. Specifically, the smartphone's microphone captures the voice and the built-in ADC (analog-to-digital converter) converts it into a digital signal.

[0961] Step 2:

[0962] The terminal transmits the captured digital audio data to a server via the Internet. The input is the captured digital audio data, and the output is the audio data transmitted via the Internet. Specifically, the terminal generates an HTTP request, divides the audio data into packets, and transmits them to the server.

[0963] Step 3:

[0964] The server reconstructs the received voice data and converts it into text data using a speech recognition engine. The input is the transmitted digital voice data, and the output is the converted text data. Specifically, the server restores the received packets to the original voice data and converts the voice data into text using Google Cloud Speech-to-Text.

[0965] Step 4:

[0966] The server uses a large-scale language model (LLM) to analyze the text data and generate response data. The input is the converted text data (e.g., "I would like to order one pizza"), and the output is the response data (e.g., "We have received your pizza order. Is there anything else you would like to order?"). Specifically, the server inputs a prompt sentence into the generative AI model and generates an appropriate response.

[0967] Step 5:

[0968] The server converts the generated response data into voice data. The input is the response data (in text format), and the output is voice data. Specifically, Google Text-to-Speech (gTTS) converts the text into synthetic voice.

[0969] Step 6:

[0970] The server divides the voice data into packets again and sends them to the user terminal via the Internet. The input is voice data, and the output is voice data sent via the Internet. Specifically, the server divides the voice data into packets and sends them as an HTTP response.

[0971] Step 7:

[0972] The terminal restores the received voice data and plays it back to the user through the built-in speaker. The input is the transmitted voice data, and the output is the played voice. Specifically, the terminal restores the received packets to the original voice data and plays back a response using the speaker. For example, the response played back to the user is, "We've received your pizza order. Is there anything else you'd like to order?"

[0973] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0974] This invention is a system that captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. Furthermore, the external computer is equipped with an emotion engine that recognizes the user's emotions and generates responses linked to the emotions, enabling more natural and emotionally appropriate dialogue.

[0975] Audio capture and transmission

[0976] When a user speaks, the device captures the voice using a microphone. The captured voice data is converted into a digital signal and sent to a server over the Internet. Here, the data is divided into packets for transmission, ensuring efficient and reliable data transmission.

[0977] Speech recognition and text conversion

[0978] The server receives the voice data packets sent from the device and restores the original voice data. The restored voice data is analyzed by a voice recognition engine and converted into corresponding text data. The voice recognition engine analyzes the voice data in detail and performs highly accurate text conversion.

[0979] emotion recognition

[0980] Next, the server inputs the text data into the emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from the input text data based on multiple emotion categories and outputs them in the form of numbers or tags. For example, if a user says, "I'm feeling terrible today," the emotion engine will classify it into emotion categories such as "negative" or "frustration."

[0981] Response Generation

[0982] The server inputs the generated text data and emotional information into a large-scale language model (LLM) to generate an appropriate response. LLM is a natural language processing model with advanced analytical capabilities, allowing it to generate an appropriate response tailored to the user's emotions. For example, if a user inputs "I'm feeling terrible today," the LLM will generate a response in the form of "I'm sorry, that's terrible. Is there anything I can help you with?"

[0983] Transcribing and retransmitting responses

[0984] The response data generated by the server is converted into voice data by a text-to-speech engine. This voice data is also packetized and sent to the terminal over the Internet. The packetization and decompression process minimizes data loss and delays.

[0985] Audio playback

[0986] The device decompresses the received voice data and plays it back to the user through the built-in speaker. This allows the user to receive a response to their question or instruction as voice. For example, a response such as "I'm sorry, that's a problem. Is there anything I can help you with?" is provided to the user in a natural way.

[0987] Specific examples

[0988] For example, consider a situation where a user says to a device, "I'm feeling incredibly miserable today." In this case, the device captures the user's voice and sends it to the server. The server converts the voice to text, generating the text data "I'm feeling incredibly miserable today." The emotion engine then recognizes emotions such as "negative" or "sad" from this text data. It then uses the LLM to generate an appropriate response such as "That's really miserable. Let's talk a bit and hopefully make you feel better." This response is converted into audio and sent to the device, which ultimately plays it back to the user.

[0989] As described above, this system enables natural conversation between the user and AI. It reduces the load on the edge device and simultaneously generates sophisticated, emotionally relevant responses in real time. This system provides strong support for collaboration between edge devices and external computers, providing users with a high-quality voice interface.

[0990] The processing flow will be explained below.

[0991] Step 1:

[0992] The user performs voice input. For example, the user speaks to the terminal, "Today is so hard."

[0993] Step 2:

[0994] The device captures the audio. The device's microphone collects the user's voice and converts the analog audio into a digital signal.

[0995] Step 3:

[0996] The terminal sends a digital signal to the server, which then divides the digital signal into packets and sends them over the network to the server.

[0997] Step 4:

[0998] The server receives the packets and restores the digital signal. The server reconstructs the received packets and stores the restored digital signal in a buffer.

[0999] Step 5:

[1000] The server uses a speech recognition engine to convert the digital signal into text data. The server analyzes the digital signal and generates the text data "Today is extremely difficult."

[1001] Step 6:

[1002] The server inputs the text data into the emotion engine and analyzes the emotions. The server recognizes emotions such as "negative" or "sad" from the text data and outputs the results in the form of numbers or tags.

[1003] Step 7:

[1004] The server inputs text data and emotional information into a large-scale language model (LLM) to generate a response. The server passes the text "I'm feeling incredibly miserable today" along with the emotional information "negative" and "sad" to the LLM, generating a response such as "That's really hard on you. I hope we can talk a bit and make you feel better."

[1005] Step 8:

[1006] The server converts the response text into audio data using a text-to-speech engine. The server converts the generated response text into audio data.

[1007] Step 9:

[1008] The server packets the voice data and sends it to the terminal. The server divides the voice data into packets and sends them to the terminal over the network.

[1009] Step 10:

[1010] The terminal receives the packets and reconstructs the voice data. The terminal reconstructs the received packets and restores the original voice data.

[1011] Step 11:

[1012] The device plays the audio data, and through the device's built-in speaker, plays a voice response to the user saying, "That must be really hard on you. Let's talk a bit and hopefully you'll feel better."

[1013] Example 2

[1014] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1015] Conventional speech recognition systems simply convert a user's speech into text and generate simple responses based on that text. This makes it difficult to understand the user's emotions and generate responses that take those emotions into consideration, resulting in a less natural and emotionally rich dialogue. In particular, in situations where the user is emotionally charged, the system is unable to respond appropriately, making it difficult to provide a satisfying user experience. The present invention aims to solve this problem by providing a system that analyzes a user's emotions and provides natural-sounding voice responses based on those emotions.

[1016] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing the user's emotions and generating response data based on the emotions, a means for converting a digital signal into text data, and a means for analyzing the text data and generating response data based on the text data. This makes it possible to generate natural responses that are in line with the user's emotions, enabling more satisfying interactions.

[1017] "User" refers to a person who interacts with the system by inputting voice through a terminal.

[1018] "Terminal" refers to a device that receives voice input from a user, converts it into a digital signal, transmits it to a server, and plays back a voice response from the server to the user.

[1019] "Voice input" refers to the voice that the user utters to the terminal, and is the voice signal that is processed by the system.

[1020] A "digital signal" is a signal obtained by converting an audio input into a digital format, and refers to data in a format that can be processed by a computer.

[1021] "External computer" refers to a computing device that processes digital signals sent from the terminal and performs voice recognition, emotion analysis, and response generation.

[1022] "Text data" refers to character string data generated by an external computer analyzing voice data.

[1023] An "emotion recognition engine" refers to a software function that analyzes text data to recognize a user's emotions and output them in numerical or tag format.

[1024] "Response data" refers to data containing appropriate responses to users, generated using an emotion recognition engine or a large-scale language model.

[1025] "Large-scale language model (LLM)" refers to an advanced language model that uses natural language processing to generate appropriate responses from text data.

[1026] "Text-to-speech engine" means the software functionality for converting generated response data into audio data.

[1027] "Packet" refers to the unit of data into which digital signals are divided and sent over the Internet.

[1028] A "microphone" refers to an audio device for capturing the user's voice.

[1029] "Speaker" refers to an audio device that plays back audio data sent from the server and lets the user hear it.

[1030] This clarifies each element of the system and its function.

[1031] The present invention is a system that captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. Furthermore, the external computer is equipped with an emotion engine that recognizes the user's emotions and generates responses linked to the emotions, enabling more natural and emotionally appropriate dialogue.

[1032] Audio capture and transmission

[1033] When a user speaks, the device captures the voice using a microphone. For example, the user might say, "I'm feeling incredibly miserable today." This voice is converted into a digital signal and sent to a server over the Internet. The voice data is divided into packets for transmission, ensuring efficient and reliable transmission.

[1034] Speech recognition and text conversion

[1035] The server receives the voice data packets sent from the device and restores the original voice data. The restored voice data is analyzed by a voice recognition engine (e.g., an engine using a deep learning model) and converted into corresponding text data. The voice recognition engine analyzes the voice data in detail and performs highly accurate text conversion.

[1036] emotion recognition

[1037] Next, the server inputs the text data into an emotion engine (e.g., an engine using natural language processing technology) to analyze the user's emotions. The emotion engine recognizes the user's emotions from the input text data based on multiple emotion categories and outputs them in the form of numbers or tags. For example, if a user says, "I'm feeling terrible today," the emotion engine will classify it into emotion categories such as "negative" or "frustration."

[1038] Response Generation

[1039] The server inputs the generated text data and emotional information into a large-scale language model (LLM) to generate an appropriate response. LLM is a natural language processing model with advanced analytical capabilities, so it generates an appropriate response that matches the user's emotions. For example, if a user inputs "I'm feeling terrible today," the LLM will generate a response in the form of "I'm sorry, that's terrible. Is there anything I can help you with?" In this case, the LLM can be given a prompt sentence such as:

[1040] "Example prompt statement:"

[1041] User: I'm having a really hard time today

[1042] System: That's really tough. Let's talk a bit and hopefully it will help ease your mind.

[1043] Transcribing and retransmitting responses

[1044] The response data generated by the server is converted into voice data by a text-to-speech engine (e.g., a text-to-speech engine using natural language processing). This voice data is also packetized and sent to the terminal over the Internet. The packetization and decompression process minimizes data loss and delays.

[1045] Audio playback

[1046] The device decompresses the received voice data and plays it back to the user through the built-in speaker. This allows the user to receive a response to their question or instruction as voice. For example, a response such as "I'm sorry, that's a problem. Is there anything I can help you with?" is provided to the user in a natural way.

[1047] These components and processing steps enable dialogue that reflects the user's emotions, providing a more human-like and natural voice interface, which reduces the load on edge devices and enables both emotionally-sensitive and real-time response generation.

[1048] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1049] Step 1: Capture audio

[1050] Input: User's spoken utterance

[1051] How it works: The user speaks into the device's microphone, which captures the audio signal.

[1052] Output: Digitized audio data

[1053] Specific operation: The device's microphone converts analog audio signals into digital signals and temporarily stores the data in the device's memory.

[1054] Step 2: Sending audio data

[1055] Input: Digitized audio data

[1056] How it works: The device splits the captured audio data into packets and sends them over the Internet to a server.

[1057] Output: Packetized voice data sent over the internet

[1058] Specific operation: Audio data is divided into packets of a fixed size, and header information is added to each packet before transmission.

[1059] Step 3: Audio data recovery

[1060] Input: Packetized audio data

[1061] Operation: The server receives packets sent from the device and restores them to the original audio data.

[1062] Output: Recovered audio data

[1063] Specific operation: The received packets are rearranged in order and combined to restore the original audio data.

[1064] Step 4: Speech recognition and text conversion

[1065] Input: Recovered audio data

[1066] How it works: The server inputs the recovered voice data into a speech recognition engine and converts it into highly accurate text data.

[1067] Output: Text data

[1068] Specific operation: A speech recognition engine (e.g., an engine using a deep learning model) analyzes the voice data and generates text data such as "Today is incredibly difficult."

[1069] Step 5: Emotion Recognition

[1070] Input: Generated text data

[1071] How it works: The server inputs text data into an emotion recognition engine to analyze the user's emotions.

[1072] Output: Emotion information (in numerical and tag formats)

[1073] Specific operation: An emotion recognition engine using natural language processing technology tags emotions in the text and assigns labels such as "negative" or "sad."

[1074] Step 6: Response Generation

[1075] Input: Text data and emotion information

[1076] How it works: The server inputs text data and emotion information into a large-scale language model (LLM) to generate an appropriate response.

[1077] Output: Response data

[1078] Specific operation: The user's utterances and emotional data are input as prompts into a large-scale language model (e.g., GPT-4), and a response such as "That's really painful. Let's talk for a bit and hopefully you'll feel better" is generated.

[1079] Step 7: Transcribing responses

[1080] Input: Response data

[1081] How it works: The server inputs the generated response data into a text-to-speech engine and converts it into audio data.

[1082] Output: Audio data

[1083] Specific operation: A text-to-speech engine (e.g., an engine using natural language processing) converts the response data into audio data.

[1084] Step 8: Resend the response audio

[1085] Input: Audio data

[1086] How it works: The server breaks the audio data back into packets and sends them over the Internet to the device.

[1087] Output: Packetized audio data

[1088] Specific operation: Audio data is packetized and sent to the terminal. Packets contain sequence information, making it easy to restore the data at the terminal.

[1089] Step 9: Playing Audio

[1090] Input: Packetized audio data

[1091] Operation: The device reconstructs the received audio packets and plays them through the built-in speaker.

[1092] Output: A spoken response to the user

[1093] Specific operation: The device receives the packets and restores them as voice data. The restored voice data is played over the speaker, providing a response to the user. For example, "That must be really painful. I hope that talking about it will help you feel better."

[1094] (Application example 2)

[1095] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1096] Conventional speech recognition systems have difficulty in accurately recognizing user emotions and generating responses based on them. As a result, especially in customer service and complaint handling, systems can only provide mechanical responses that ignore the user's emotions, making it difficult to improve customer satisfaction. In addition, edge devices themselves have limited processing power, making it difficult to perform highly accurate emotion recognition and response generation in real time.

[1097] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into a digital signal, means for transmitting the digital signal to an external computer, means for converting the digital signal into text data in the external computer, means for analyzing the text data and generating response data based on the text data, means for converting the generated response data into voice data, means for transmitting the converted voice data to a terminal, means for playing the voice data on the terminal, and means for recognizing the user's emotions in a conversation with a customer and generating an appropriate response based on the emotions. This makes it possible to provide more natural and appropriate responses in real time that correspond to the user's emotions.

[1098] "User" refers to any person or subject that uses the system to provide voice input.

[1099] The "means for receiving audio input" refers to a device for capturing audio data from a user, and specifically refers to an audio capture device such as a microphone.

[1100] "Means for converting audio input into a digital signal" refers to a process or device that converts received audio into electronic digital data, typically an analog-to-digital converter (ADC).

[1101] "Means for transmitting digital signals to an external computing device" refers to means for transmitting digital data over the Internet or other communications network to a remote server or computing resource.

[1102] "External computers" refer to servers or cloud computing environments that exist separately from edge devices and are used for advanced computational processing and analysis.

[1103] "Means for converting digital signals into text data" refers to software or a system that uses a speech recognition engine to convert digital voice data into a corresponding text format.

[1104] "Means for analyzing text data and generating response data based on it" refers to a program or system for interpreting received text data and generating optimal responses or instructions in response to it.

[1105] The "means for converting the generated response data into voice data" refers to a text-to-speech engine (TTS) for outputting the text-format response data as voice again.

[1106] "Means for transmitting converted voice data to a terminal" refers to a communication means for returning the generated voice data to the edge device.

[1107] "Means for reproducing audio data on a terminal" refers to means for reproducing received audio data through an audio device such as a speaker.

[1108] "Means for recognizing a user's emotions in customer interactions and generating appropriate responses based on those emotions" refers to large-scale language models and other artificial intelligence technologies that use emotion recognition engines to understand a user's emotions and generate natural-sounding responses accordingly.

[1109] MODE FOR CARRYING OUT THE INVENTION

[1110] A system embodying this invention captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. A specific implementation of this system is described below.

[1111] Audio capture and transmission

[1112] When a user speaks, the smart glasses used as a terminal capture the voice using a microphone. The captured voice data is converted into a digital signal by a voice capture device. This converted digital signal is transmitted to an external computer server via the Internet. Specifically, the data is transmitted in packets, enabling efficient data transmission.

[1113] Speech recognition and text conversion

[1114] The server receives the voice data packets sent from the device and restores them to the original voice data. The restored voice data is analyzed using a voice recognition engine and converted into corresponding text data. The voice recognition engine is typically a highly accurate software.

[1115] emotion recognition

[1116] Next, the server inputs the text data into an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from the input text data based on multiple emotion categories. This emotion information is output in the form of numbers and tags.

[1117] Response Generation

[1118] The server uses the generated text data and emotion information to generate appropriate responses using a large-scale language model (LLM), which has the ability to provide natural responses that are in line with the user's input.

[1119] Transcribing and retransmitting responses

[1120] The generated response data is converted into voice data by a text-to-speech engine, which is also packetized and transmitted to the smart glasses via the internet.

[1121] Audio playback

[1122] The smart glasses, which are the terminals, decompress the received voice data and play it back to the user through the built-in speaker, allowing the user to receive voice responses to their questions and instructions.

[1123] For example, if a store staff member asks, "Hey, is there something I can help you with?" and the customer replies, "Hey, I don't know how to use this product," the system captures the voice and sends it to the server. The server analyzes the voice and recognizes the emotion of "the customer is having trouble." It then generates an appropriate response, such as "That's a problem. Can you explain in more detail?" and provides it to the staff member through the smart glasses.

[1124] An example of a specific prompt is, "Capture the customer's voice and generate an appropriate response based on their emotion."

[1125] This invention makes it possible to provide a more natural and appropriate response in real time according to the user's emotions.

[1126] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1127] Step 1:

[1128] The user speaks to the device

[1129] What it does: A user speaks into the smart glasses, and the smart glasses' built-in microphone captures the audio.

[1130] Input: User's voice

[1131] Output: Captured audio data

[1132] Step 2:

[1133] The device converts the voice data into a digital signal

[1134] Specific operation: Audio data captured by the microphone is converted into a digital signal by an analog-to-digital converter (ADC) inside the smart glasses.

[1135] Input: Captured audio data

[1136] Output: Digital audio signal

[1137] Step 3:

[1138] The terminal sends a digital signal to an external computer

[1139] What it does: Digital audio signals are packetized and sent over the internet to a remote server.

[1140] Input: Digital audio signal

[1141] Output: Packetized digital signal

[1142] Step 4:

[1143] The server receives and decompresses the packets and converts the audio data into text data.

[1144] Specific operation: The server receives and decompresses the packetized digital signal and converts it into text data using a speech recognition engine.

[1145] Input: Packetized digital signal

[1146] Output: Text data

[1147] Step 5:

[1148] The server inputs the text data into the emotion engine and analyzes the user's emotions.

[1149] Specific operation: The received text data is input into the emotion engine, which recognizes the user's emotion based on multiple emotion categories. The emotion engine outputs the results in the form of numbers and tags.

[1150] Input: Text data

[1151] Output: Emotion information (number, tag)

[1152] Step 6:

[1153] The server uses the text data and emotional information to generate an appropriate response.

[1154] Specific operation: The server inputs text data and emotional information into a large-scale language model (LLM) and generates an appropriate response based on the user's input.

[1155] Input: Text data, emotion information

[1156] Output: Response data

[1157] Step 7:

[1158] The response data generated by the server is converted into voice data, repacketized, and sent to the terminal.

[1159] Specific operation: The response data is converted into voice data by a text-to-speech engine (TTS), repacketized, and sent to the terminal via the Internet.

[1160] Input: Response data

[1161] Output: Packetized audio data

[1162] Step 8:

[1163] The device recovers the audio data it receives and plays it through the built-in speaker.

[1164] Specific operation: The smart glasses decompress the packetized audio data they receive and play the audio through their built-in speakers.

[1165] Input: Packetized audio data

[1166] Output: Playback of the recovered audio data

[1167] The user is then able to interact through the responses that are played back.

[1168] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1169] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1170] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1171] [Fourth embodiment]

[1172] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1173] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1174] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1175] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1176] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1177] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1178] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1179] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1180] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1181] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1182] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1183] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1184] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1185] The present invention is a system that captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. Below, the processing of the program of this system will be explained in natural language.

[1186] Audio capture and transmission

[1187] When a user speaks, the device uses a microphone to capture the voice. The captured voice data is converted into a digital signal and sent to an external computer via the Internet. Here, the data is divided into packets for transmission, ensuring efficient and reliable data transmission.

[1188] Speech recognition and text conversion

[1189] The server receives the voice data packets sent from the device and restores the original voice data. The restored voice data is analyzed by a voice recognition engine and converted into corresponding text data. The voice recognition engine analyzes the voice data in detail and performs highly accurate text conversion.

[1190] Response Generation

[1191] The server inputs the generated text data into a large-scale language model (LLM) to generate an appropriate response. LLM is a natural language processing model with advanced analytical capabilities, allowing it to generate appropriate and natural responses to user questions and instructions. For example, if a user types, "What's the weather forecast?", the LLM will generate a response such as, "Today's weather is sunny, and the temperature is 25 degrees."

[1192] Transcribing and retransmitting responses

[1193] The response data generated by the server is converted into voice data by a text-to-speech engine. This voice data is also packetized and sent to the terminal over the Internet. The packetization and decompression process minimizes data loss and delays.

[1194] Audio playback

[1195] The device then decompresses the received voice data and plays it back to the user through the built-in speaker, allowing the user to receive a response to their question or command as voice. For example, the device provides a natural response to the user, such as "Today's weather is sunny and the temperature is 25 degrees."

[1196] Specific examples

[1197] For example, consider the case where a user speaks to a device, "What's the weather going to be like tomorrow?" In this case, the device captures the user's voice and sends it to the server. The server converts the voice to text, generating the text data "What's the weather going to be like tomorrow?". It then uses LLM to generate an appropriate response, such as "Tomorrow's weather will be rainy and the temperature will be 20 degrees." This response is converted to audio and sent to the device, which ultimately plays it back to the user.

[1198] As described above, this system enables natural conversation between the user and AI. It reduces the load on the edge device and achieves both advanced response generation and real-time performance. This system provides strong support for collaboration between edge devices and external computers, providing users with a high-quality voice interface.

[1199] The processing flow will be explained below.

[1200] Step 1:

[1201] The user performs voice input. The user speaks to the device, saying, "Tell me the weather forecast."

[1202] Step 2:

[1203] The device captures the audio. The device's microphone collects the user's voice and converts the analog audio into a digital signal.

[1204] Step 3:

[1205] The device sends a digital signal to the server, which then divides the signal into packets and sends them over the network to the server.

[1206] Step 4:

[1207] The server receives the packets and restores the digital signal. The server reconstructs the received packets and stores the original digital signal in a buffer.

[1208] Step 5:

[1209] The server uses a speech recognition engine to convert the digital signal into text data, which the server then analyzes and generates the text data "What's the weather forecast?"

[1210] Step 6:

[1211] The server uses a large-scale language model (LLM) to analyze the text data and generate a response. The server inputs the text data "What is the weather forecast?" into the LLM and generates the response text "Today's weather is sunny and the temperature is 25 degrees."

[1212] Step 7:

[1213] The server uses a text-to-speech engine to convert the response text into audio data. The server converts the response text, "Today's weather is sunny and the temperature is 25 degrees," into audio.

[1214] Step 8:

[1215] The server packets the audio data and sends it to the terminal. The server divides the audio data into packets and sends them to the terminal via the network.

[1216] Step 9:

[1217] The terminal receives the packets and reconstructs the voice data. The terminal reconstructs the received packets and restores the original voice data.

[1218] Step 10:

[1219] The device plays the audio data, and the audio response "Today's weather is sunny, and the temperature is 25 degrees" is played to the user through the device's built-in speaker.

[1220] Example 1

[1221] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1222] Conventional voice recognition and response generation systems impose a heavy computational load on edge devices and suffer from problems such as network delays and data loss. Furthermore, achieving real-time and highly accurate response generation requires large-scale computational resources, which entails high costs. The present invention aims to solve these problems and provide a fast, high-quality voice response interface while reducing the load on edge devices.

[1223] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1224] In this invention, the server includes means including a speech recognition engine for converting voice data into text data, means for analyzing the text data and using a large-scale language model to generate response data based on the text data, and means including a speech synthesis engine for converting the generated response data into voice data, thereby reducing the load on the edge device and enabling highly accurate response generation in real time while minimizing problems such as network delays and data loss.

[1225] "User" refers to a person who utilizes the speech recognition and response generation system of the present invention.

[1226] "Terminal" refers to a device that captures voice input from a user and transmits a digital signal to a server. Examples include smartphones and smart speakers.

[1227] The term "server" refers to a computer that receives data sent from a terminal and performs speech recognition, response generation, and speech synthesis.

[1228] "Voice input" refers to information spoken by a user into a terminal.

[1229] "Digital signal" refers to a signal obtained by converting analog audio data into digital format.

[1230] A "network" refers to a communications infrastructure that interconnects multiple computers and terminals. A specific example is the Internet.

[1231] The "external computer" refers to a computer such as a server that receives and processes data sent from a terminal.

[1232] "Audio data" refers to data that represents an audio signal in digital form.

[1233] "Text data" refers to character information converted from voice data by a voice recognition engine.

[1234] A "voice recognition engine" refers to software or hardware that takes voice data as input, analyzes it, and converts it into text data.

[1235] A "speech synthesis engine" refers to software or hardware that takes text data as input and generates speech data.

[1236] A "large-scale language model" is a natural language processing model that uses deep learning and learns from large amounts of text data to generate and analyze text.

[1237] A "packet" refers to the small units of data that are divided when sending and receiving data over a network.

[1238] "Packetization" refers to the process of breaking down digital signals or data into smaller units (packets) and converting them into a form that can be sent over a network.

[1239] "Reconstruction" refers to the process of reconstructing the received packets into the original voice data.

[1240] "Playback" refers to the act of outputting audio data from a speaker device so that the user can hear it.

[1241] The present invention is a system that receives voice input from a user, processes the input, generates a response, and provides it to the user again in voice. How this system is specifically implemented will be described below.

[1242] Audio capture and conversion to digital signal

[1243] When a user speaks, the device captures the voice using a built-in or connected microphone. Examples include smartphones and smart speakers. The captured voice is converted into a digital signal. Here, the device uses, for example, an ADC (analog-to-digital converter).

[1244] Digital signal transmission

[1245] The converted digital signal is divided into packets and sent over a network (e.g., the Internet) to an external computer, i.e., a server, using a network protocol such as TCP / IP.

[1246] Processing on the server

[1247] The server receives the packets sent from the device, restores them to their original digital signals, and then converts them into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text or IBM Watson Speech to Text).

[1248] Response Generation

[1249] The text data is analyzed using a large-scale language model (e.g., OpenAI's GPT-4) to generate response data, which is then made appropriate and highly accurate using natural language processing technology.

[1250] Conversion to audio data and transmission

[1251] The generated response data is converted into voice data using a speech synthesis engine (e.g., Amazon Polly or Google Cloud Text-to-Speech). The converted voice data is again divided into packets and sent to the device over the network.

[1252] Audio playback

[1253] The terminal receives the voice data packets sent from the server and restores them to the original voice data. The restored voice data is then played back to the user through the speaker, allowing the user to receive voice responses to their questions and instructions.

[1254] Specific examples

[1255] For example, consider the case where a user speaks to a terminal and asks, "What's the weather going to be like tomorrow?"

[1256] 1. The device captures the user's voice, converts it into a digital signal, and sends it to the server.

[1257] 2. The server receives the voice and uses a speech recognition engine to generate text data such as "What's the weather like tomorrow?"

[1258] 3. The generated text data is analyzed using a large-scale language model, and the response data, "Tomorrow's weather will be rainy and the temperature will be 20 degrees," is generated.

[1259] 4. This response data is converted into voice data using a voice synthesis engine and sent to the terminal.

[1260] 5. The device plays the received voice data, and the user hears the response, "Tomorrow's weather will be rainy and the temperature will be 20 degrees."

[1261] Prompt Sentence Examples

[1262] User: What's the weather like tomorrow?

[1263] Device: (Capture audio and send to server)

[1264] Server: (Receives voice data and converts it to text)

[1265] Server: (Generates response using LLM)

[1266] Server: (Converts the response into audio and sends it to the device)

[1267] Terminal: (receives and plays audio data)

[1268] Through the above procedure, the system of the present invention can realize a natural voice interface with the user, and can provide highly accurate responses while reducing the load on the edge device.

[1269] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1270] Step 1: Capture audio

[1271] The user makes a voice input (e.g., "What's the weather going to be like tomorrow?"). The microphone built into the device captures this voice and converts the analog voice signal into a digital signal. Specifically, the microphone picks up the voice and converts it into digital data via an ADC (analog-to-digital converter).

[1272] (Input) User's voice

[1273] (Output) Digital audio signal

[1274] Step 2: Sending a digital signal

[1275] The terminal breaks the converted digital voice signal into packets, and then uses a network protocol (e.g., TCP / IP) to send the data over the Internet to a server.

[1276] (Input) Digital audio signal

[1277] (Output) Packet data sent over the network

[1278] Step 3: Receiving and restoring audio data

[1279] The server receives the packet data sent from the terminal, reassembles the received packets, and restores the original digital audio signal.

[1280] (Input) Packet data

[1281] (Output) Reconstructed digital audio signal

[1282] Step 4: Speech recognition and text conversion

[1283] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to analyze the digital voice signal and convert it into corresponding text data. Specifically, the server inputs the voice data into the speech recognition engine and obtains the text data as its output.

[1284] (Input) Digital audio signal

[1285] (Output) Text data (e.g., "What's the weather like tomorrow?")

[1286] Step 5: Parsing text data and generating responses

[1287] The server inputs the generated text data into a large-scale language model (e.g., OpenAI's GPT-4) to generate an appropriate response. Specifically, the text data is input into the LLM, and the response text is obtained as the output (e.g., "Tomorrow's weather will be rainy, and the temperature will be 20 degrees").

[1288] (Input) Text data (e.g., "What's the weather like tomorrow?")

[1289] (Output) Response text (e.g. "Tomorrow's weather will be rainy and the temperature will be 20 degrees")

[1290] Step 6: Convert response text to speech

[1291] The server inputs the generated response text into a speech synthesis engine (e.g., Amazon Polly) and converts it into voice data. Specifically, the server inputs the response text and obtains voice data as output.

[1292] (Input) Response text (e.g., "Tomorrow's weather will be rainy and the temperature will be 20 degrees.")

[1293] (Output) Audio data

[1294] Step 7: Sending audio data

[1295] The server divides the voice data into packets and transmits them to the terminal via the network. Specifically, the server packetizes the voice data and transmits them using a network protocol.

[1296] (Input) Audio data

[1297] (Output) Packet data sent over the network

[1298] Step 8: Receiving and Playing Audio Data

[1299] The terminal receives the packet data sent from the server, restores it to the original voice data, and then plays the restored voice data through the built-in speaker.

[1300] (Input) Packet data

[1301] (Output) Audio to be played (e.g. "Tomorrow's weather will be rainy and the temperature will be 20 degrees")

[1302] This is the flow of the program processing for this system, which allows users to use the voice interface in a natural way.

[1303] (Application example 1)

[1304] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1305] Current voice input systems require real-time responses and highly accurate recognition when users voice-order or customize their food. However, in many cases, the load on edge devices is high, resulting in delayed responses. Another problem is that low-accuracy recognition may occur, resulting in results that differ from the user's request. These issues are particularly serious in fields such as food delivery, which require fast and accurate processing. The present invention aims to solve these issues and provide a highly accurate, real-time voice input system while reducing the load on edge devices.

[1306] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1307] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into a digital signal, means for transmitting the digital signal to an external computer, means for inputting text data into a large-scale language model and generating response data, means for converting the generated response data into voice data, means for transmitting the converted voice data to a terminal, means for the user to place an order by voice input, means for confirming the order details in real time, and means for giving voice instructions for customizing dishes. This allows the user to receive highly accurate voice responses in real time. Furthermore, since orders and customizations are accurately executed based on the user's voice input, an advanced user experience can be provided.

[1308] A "means for receiving audio input from a user" is a device or method for capturing audio produced by a user.

[1309] A "means for converting audio input to a digital signal" is a device or method for converting analog audio data into digital form.

[1310] "Means for transmitting digital signals to an external computer" refers to a device or method for transmitting digitized voice data to an external server via a network such as the Internet.

[1311] The "means for converting digital signals into text data using an external computer" refers to a device or method, such as a voice recognition engine, for converting transmitted digital voice data into text information.

[1312] The "means for inputting the generated text data into a large-scale language model and generating response data" refers to a device or method for inputting text data obtained by a speech recognition engine into a natural language processing model and generating an appropriate response to a user's inquiry or instruction.

[1313] The "means for converting the generated response data into voice data" refers to a device or method for converting the generated text-format response data into synthetic voice.

[1314] The "means for transmitting the converted voice data to the terminal" refers to a device or method for retransmitting the synthesized voice data to the user terminal over the network.

[1315] "Means for reproducing audio data at a terminal" refers to a device or method for reproducing audio data transmitted to a user terminal using an audio device such as a speaker.

[1316] The "means for users to order by voice input" refers to a device or method that allows users to order products or services by voice.

[1317] "Means for confirming order details in real time" refers to a device or method for analyzing and confirming order details in real time based on voice input.

[1318] "Means for giving voice instructions for customizing dishes" refers to a device or method that allows a user to give voice instructions for customizing desired dishes, and that the system understands and reflects these instructions.

[1319] The present invention is a system that captures voice input from users with an edge device and transmits the voice data to an external computer for processing. The system enables voice ordering, real-time confirmation, and customization instructions in the field of food delivery.

[1320] User voice input

[1321] When a user places an order by voice using a food delivery application, the smartphone's microphone is used to capture voice input. For example, the user might say, "I'd like to order one pizza." This voice is converted into a digital signal and sent over the Internet to an external computer, or server.

[1322] Speech data conversion and response generation

[1323] The server converts the received digital voice signal into text data. High-precision text conversion is performed using a speech recognition engine such as Google Cloud Speech-to-Text. The converted text data is input into a large-scale language model (LLM), which generates response data based on the user's question or instruction. Using a generative AI model, a natural response such as, "We've received your pizza order. Is there anything else you'd like to order?" can be obtained.

[1324] Transcribing and retransmitting response data

[1325] The generated response data is converted into voice data using a speech synthesis engine such as Google Text-to-Speech (gTTS), which is then sent back to the user's smartphone via the Internet.

[1326] Audio playback on the user device

[1327] The user's smartphone then plays the received voice data through its built-in speaker, allowing the user to receive natural voice responses to confirm their order or to ask additional questions. For example, a user can customize their order by voice input, such as "more tomatoes, less cheese, please," and the response will be returned in real time.

[1328] Specific examples

[1329] If the user says the following prompt:

[1330] "I'd like to order a pizza."

[1331] "More tomatoes, less cheese, please."

[1332] Based on the prompt above, the system will proceed through the following steps:

[1333] 1. Converts audio input into a digital signal.

[1334] 2. Send the digital signal to the server.

[1335] 3. The server converts the voice signal into text.

[1336] 4. Generate response data using a large-scale language model.

[1337] 5. Convert the response data into audio data.

[1338] 6. The voice data is sent to the user terminal.

[1339] 7. The user device plays the received audio.

[1340] This system allows users to smoothly order food delivery using voice. The main hardware used is a smartphone and a microphone. The software includes Google Cloud Speech-to-Text and Google Text-to-Speech (gTTS), which enables a high-quality voice interface.

[1341] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1342] Step 1:

[1343] A user speaks to a smartphone to place an order or ask a question. For example, they might say, "I'd like to order one pizza." This voice data is captured by a microphone. The input is an analog voice signal, and the output is the captured digital voice data. Specifically, the smartphone's microphone captures the voice and the built-in ADC (analog-to-digital converter) converts it into a digital signal.

[1344] Step 2:

[1345] The terminal transmits the captured digital audio data to a server via the Internet. The input is the captured digital audio data, and the output is the audio data transmitted via the Internet. Specifically, the terminal generates an HTTP request, divides the audio data into packets, and transmits them to the server.

[1346] Step 3:

[1347] The server reconstructs the received voice data and converts it into text data using a speech recognition engine. The input is the transmitted digital voice data, and the output is the converted text data. Specifically, the server restores the received packets to the original voice data and converts the voice data into text using Google Cloud Speech-to-Text.

[1348] Step 4:

[1349] The server uses a large-scale language model (LLM) to analyze the text data and generate response data. The input is the converted text data (e.g., "I would like to order one pizza"), and the output is the response data (e.g., "We have received your pizza order. Is there anything else you would like to order?"). Specifically, the server inputs a prompt sentence into the generative AI model and generates an appropriate response.

[1350] Step 5:

[1351] The server converts the generated response data into voice data. The input is the response data (in text format), and the output is voice data. Specifically, Google Text-to-Speech (gTTS) converts the text into synthetic voice.

[1352] Step 6:

[1353] The server divides the voice data into packets again and sends them to the user terminal via the Internet. The input is voice data, and the output is voice data sent via the Internet. Specifically, the server divides the voice data into packets and sends them as an HTTP response.

[1354] Step 7:

[1355] The terminal restores the received voice data and plays it back to the user through the built-in speaker. The input is the transmitted voice data, and the output is the played voice. Specifically, the terminal restores the received packets to the original voice data and plays back a response using the speaker. For example, the response played back to the user is, "We've received your pizza order. Is there anything else you'd like to order?"

[1356] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1357] This invention is a system that captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. Furthermore, the external computer is equipped with an emotion engine that recognizes the user's emotions and generates responses linked to the emotions, enabling more natural and emotionally appropriate dialogue.

[1358] Audio capture and transmission

[1359] When a user speaks, the device captures the voice using a microphone. The captured voice data is converted into a digital signal and sent to a server over the Internet. Here, the data is divided into packets for transmission, ensuring efficient and reliable data transmission.

[1360] Speech recognition and text conversion

[1361] The server receives the voice data packets sent from the device and restores the original voice data. The restored voice data is analyzed by a voice recognition engine and converted into corresponding text data. The voice recognition engine analyzes the voice data in detail and performs highly accurate text conversion.

[1362] emotion recognition

[1363] Next, the server inputs the text data into the emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from the input text data based on multiple emotion categories and outputs them in the form of numbers or tags. For example, if a user says, "I'm feeling terrible today," the emotion engine will classify it into emotion categories such as "negative" or "frustration."

[1364] Response Generation

[1365] The server inputs the generated text data and emotional information into a large-scale language model (LLM) to generate an appropriate response. LLM is a natural language processing model with advanced analytical capabilities, allowing it to generate an appropriate response tailored to the user's emotions. For example, if a user inputs "I'm feeling terrible today," the LLM will generate a response in the form of "I'm sorry, that's terrible. Is there anything I can help you with?"

[1366] Transcribing and retransmitting responses

[1367] The response data generated by the server is converted into voice data by a text-to-speech engine. This voice data is also packetized and sent to the terminal over the Internet. The packetization and decompression process minimizes data loss and delays.

[1368] Audio playback

[1369] The device decompresses the received voice data and plays it back to the user through the built-in speaker. This allows the user to receive a response to their question or instruction as voice. For example, a response such as "I'm sorry, that's a problem. Is there anything I can help you with?" is provided to the user in a natural way.

[1370] Specific examples

[1371] For example, consider a situation where a user says to a device, "I'm feeling incredibly miserable today." In this case, the device captures the user's voice and sends it to the server. The server converts the voice to text, generating the text data "I'm feeling incredibly miserable today." The emotion engine then recognizes emotions such as "negative" or "sad" from this text data. It then uses the LLM to generate an appropriate response such as "That's really miserable. Let's talk a bit and hopefully make you feel better." This response is converted into audio and sent to the device, which ultimately plays it back to the user.

[1372] As described above, this system enables natural conversation between the user and AI. It reduces the load on the edge device and simultaneously generates sophisticated, emotionally relevant responses in real time. This system provides strong support for collaboration between edge devices and external computers, providing users with a high-quality voice interface.

[1373] The processing flow will be explained below.

[1374] Step 1:

[1375] The user performs voice input. For example, the user speaks to the terminal, "Today is so hard."

[1376] Step 2:

[1377] The device captures the audio. The device's microphone collects the user's voice and converts the analog audio into a digital signal.

[1378] Step 3:

[1379] The terminal sends a digital signal to the server, which then divides the digital signal into packets and sends them over the network to the server.

[1380] Step 4:

[1381] The server receives the packets and restores the digital signal. The server reconstructs the received packets and stores the restored digital signal in a buffer.

[1382] Step 5:

[1383] The server uses a speech recognition engine to convert the digital signal into text data. The server analyzes the digital signal and generates the text data "Today is extremely difficult."

[1384] Step 6:

[1385] The server inputs the text data into the emotion engine and analyzes the emotions. The server recognizes emotions such as "negative" or "sad" from the text data and outputs the results in the form of numbers or tags.

[1386] Step 7:

[1387] The server inputs text data and emotional information into a large-scale language model (LLM) to generate a response. The server passes the text "I'm feeling incredibly miserable today" along with the emotional information "negative" and "sad" to the LLM, generating a response such as "That's really hard on you. I hope we can talk a bit and make you feel better."

[1388] Step 8:

[1389] The server converts the response text into audio data using a text-to-speech engine. The server converts the generated response text into audio data.

[1390] Step 9:

[1391] The server packets the voice data and sends it to the terminal. The server divides the voice data into packets and sends them to the terminal over the network.

[1392] Step 10:

[1393] The terminal receives the packets and reconstructs the voice data. The terminal reconstructs the received packets and restores the original voice data.

[1394] Step 11:

[1395] The device plays the audio data, and through the device's built-in speaker, plays a voice response to the user saying, "That must be really hard on you. Let's talk a bit and hopefully you'll feel better."

[1396] Example 2

[1397] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1398] Conventional speech recognition systems simply convert a user's speech into text and generate simple responses based on that text. This makes it difficult to understand the user's emotions and generate responses that take those emotions into consideration, resulting in a less natural and emotionally rich dialogue. In particular, in situations where the user is emotionally charged, the system is unable to respond appropriately, making it difficult to provide a satisfying user experience. The present invention aims to solve this problem by providing a system that analyzes a user's emotions and provides natural-sounding voice responses based on those emotions.

[1399] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing the user's emotions and generating response data based on the emotions, a means for converting a digital signal into text data, and a means for analyzing the text data and generating response data based on the text data. This makes it possible to generate natural responses that are in line with the user's emotions, enabling more satisfying interactions.

[1400] "User" refers to a person who interacts with the system by inputting voice through a terminal.

[1401] "Terminal" refers to a device that receives voice input from a user, converts it into a digital signal, transmits it to a server, and plays back a voice response from the server to the user.

[1402] "Voice input" refers to the voice that the user utters to the terminal, and is the voice signal that is processed by the system.

[1403] A "digital signal" is a signal obtained by converting an audio input into a digital format, and refers to data in a format that can be processed by a computer.

[1404] "External computer" refers to a computing device that processes digital signals sent from the terminal and performs voice recognition, emotion analysis, and response generation.

[1405] "Text data" refers to character string data generated by an external computer analyzing voice data.

[1406] An "emotion recognition engine" refers to a software function that analyzes text data to recognize a user's emotions and output them in numerical or tag format.

[1407] "Response data" refers to data containing appropriate responses to users, generated using an emotion recognition engine or a large-scale language model.

[1408] "Large-scale language model (LLM)" refers to an advanced language model that uses natural language processing to generate appropriate responses from text data.

[1409] "Text-to-speech engine" means the software functionality for converting generated response data into audio data.

[1410] "Packet" refers to the unit of data into which digital signals are divided and sent over the Internet.

[1411] A "microphone" refers to an audio device for capturing the user's voice.

[1412] "Speaker" refers to an audio device that plays back audio data sent from the server and lets the user hear it.

[1413] This clarifies each element of the system and its function.

[1414] The present invention is a system that captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. Furthermore, the external computer is equipped with an emotion engine that recognizes the user's emotions and generates responses linked to the emotions, enabling more natural and emotionally appropriate dialogue.

[1415] Audio capture and transmission

[1416] When a user speaks, the device captures the voice using a microphone. For example, the user might say, "I'm feeling incredibly miserable today." This voice is converted into a digital signal and sent to a server over the Internet. The voice data is divided into packets for transmission, ensuring efficient and reliable transmission.

[1417] Speech recognition and text conversion

[1418] The server receives the voice data packets sent from the device and restores the original voice data. The restored voice data is analyzed by a voice recognition engine (e.g., an engine using a deep learning model) and converted into corresponding text data. The voice recognition engine analyzes the voice data in detail and performs highly accurate text conversion.

[1419] emotion recognition

[1420] Next, the server inputs the text data into an emotion engine (e.g., an engine using natural language processing technology) to analyze the user's emotions. The emotion engine recognizes the user's emotions from the input text data based on multiple emotion categories and outputs them in the form of numbers or tags. For example, if a user says, "I'm feeling terrible today," the emotion engine will classify it into emotion categories such as "negative" or "frustration."

[1421] Response Generation

[1422] The server inputs the generated text data and emotional information into a large-scale language model (LLM) to generate an appropriate response. LLM is a natural language processing model with advanced analytical capabilities, so it generates an appropriate response that matches the user's emotions. For example, if a user inputs "I'm feeling terrible today," the LLM will generate a response in the form of "I'm sorry, that's terrible. Is there anything I can help you with?" In this case, the LLM can be given a prompt sentence such as:

[1423] "Example prompt statement:"

[1424] User: I'm having a really hard time today

[1425] System: That's really tough. Let's talk a bit and hopefully it will help ease your mind.

[1426] Transcribing and retransmitting responses

[1427] The response data generated by the server is converted into voice data by a text-to-speech engine (e.g., a text-to-speech engine using natural language processing). This voice data is also packetized and sent to the terminal over the Internet. The packetization and decompression process minimizes data loss and delays.

[1428] Audio playback

[1429] The device decompresses the received voice data and plays it back to the user through the built-in speaker. This allows the user to receive a response to their question or instruction as voice. For example, a response such as "I'm sorry, that's a problem. Is there anything I can help you with?" is provided to the user in a natural way.

[1430] These components and processing steps enable dialogue that reflects the user's emotions, providing a more human-like and natural voice interface, which reduces the load on edge devices and enables both emotionally-sensitive and real-time response generation.

[1431] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1432] Step 1: Capture audio

[1433] Input: User's spoken utterance

[1434] How it works: The user speaks into the device's microphone, which captures the audio signal.

[1435] Output: Digitized audio data

[1436] Specific operation: The device's microphone converts analog audio signals into digital signals and temporarily stores the data in the device's memory.

[1437] Step 2: Sending audio data

[1438] Input: Digitized audio data

[1439] How it works: The device splits the captured audio data into packets and sends them over the Internet to a server.

[1440] Output: Packetized voice data sent over the internet

[1441] Specific operation: Audio data is divided into packets of a fixed size, and header information is added to each packet before transmission.

[1442] Step 3: Audio data recovery

[1443] Input: Packetized audio data

[1444] Operation: The server receives packets sent from the device and restores them to the original audio data.

[1445] Output: Recovered audio data

[1446] Specific operation: The received packets are rearranged in order and combined to restore the original audio data.

[1447] Step 4: Speech recognition and text conversion

[1448] Input: Recovered audio data

[1449] How it works: The server inputs the recovered voice data into a speech recognition engine and converts it into highly accurate text data.

[1450] Output: Text data

[1451] Specific operation: A speech recognition engine (e.g., an engine using a deep learning model) analyzes the voice data and generates text data such as "Today is incredibly difficult."

[1452] Step 5: Emotion Recognition

[1453] Input: Generated text data

[1454] How it works: The server inputs text data into an emotion recognition engine to analyze the user's emotions.

[1455] Output: Emotion information (in numerical and tag formats)

[1456] Specific operation: An emotion recognition engine using natural language processing technology tags emotions in the text and assigns labels such as "negative" or "sad."

[1457] Step 6: Response Generation

[1458] Input: Text data and emotion information

[1459] How it works: The server inputs text data and emotion information into a large-scale language model (LLM) to generate an appropriate response.

[1460] Output: Response data

[1461] Specific operation: The user's utterances and emotional data are input as prompts into a large-scale language model (e.g., GPT-4), and a response such as "That's really painful. Let's talk for a bit and hopefully you'll feel better" is generated.

[1462] Step 7: Transcribing responses

[1463] Input: Response data

[1464] How it works: The server inputs the generated response data into a text-to-speech engine and converts it into audio data.

[1465] Output: Audio data

[1466] Specific operation: A text-to-speech engine (e.g., an engine using natural language processing) converts the response data into audio data.

[1467] Step 8: Resend the response audio

[1468] Input: Audio data

[1469] How it works: The server breaks the audio data back into packets and sends them over the Internet to the device.

[1470] Output: Packetized audio data

[1471] Specific operation: Audio data is packetized and sent to the terminal. Packets contain sequence information, making it easy to restore the data at the terminal.

[1472] Step 9: Playing Audio

[1473] Input: Packetized audio data

[1474] Operation: The device reconstructs the received audio packets and plays them through the built-in speaker.

[1475] Output: A spoken response to the user

[1476] Specific operation: The device receives the packets and restores them as voice data. The restored voice data is played over the speaker, providing a response to the user. For example, "That must be really painful. I hope that talking about it will help you feel better."

[1477] (Application example 2)

[1478] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1479] Conventional speech recognition systems have difficulty in accurately recognizing user emotions and generating responses based on them. As a result, especially in customer service and complaint handling, systems can only provide mechanical responses that ignore the user's emotions, making it difficult to improve customer satisfaction. In addition, edge devices themselves have limited processing power, making it difficult to perform highly accurate emotion recognition and response generation in real time.

[1480] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into a digital signal, means for transmitting the digital signal to an external computer, means for converting the digital signal into text data in the external computer, means for analyzing the text data and generating response data based on the text data, means for converting the generated response data into voice data, means for transmitting the converted voice data to a terminal, means for playing the voice data on the terminal, and means for recognizing the user's emotions in a conversation with a customer and generating an appropriate response based on the emotions. This makes it possible to provide more natural and appropriate responses in real time that correspond to the user's emotions.

[1481] "User" refers to any person or subject that uses the system to provide voice input.

[1482] The "means for receiving audio input" refers to a device for capturing audio data from a user, and specifically refers to an audio capture device such as a microphone.

[1483] "Means for converting audio input into a digital signal" refers to a process or device that converts received audio into electronic digital data, typically an analog-to-digital converter (ADC).

[1484] "Means for transmitting digital signals to an external computing device" refers to means for transmitting digital data over the Internet or other communications network to a remote server or computing resource.

[1485] "External computers" refer to servers or cloud computing environments that exist separately from edge devices and are used for advanced computational processing and analysis.

[1486] "Means for converting digital signals into text data" refers to software or a system that uses a speech recognition engine to convert digital voice data into a corresponding text format.

[1487] "Means for analyzing text data and generating response data based on it" refers to a program or system for interpreting received text data and generating optimal responses or instructions in response to it.

[1488] The "means for converting the generated response data into voice data" refers to a text-to-speech engine (TTS) for outputting the text-format response data as voice again.

[1489] "Means for transmitting converted voice data to a terminal" refers to a communication means for returning the generated voice data to the edge device.

[1490] "Means for reproducing audio data on a terminal" refers to means for reproducing received audio data through an audio device such as a speaker.

[1491] "Means for recognizing a user's emotions in customer interactions and generating appropriate responses based on those emotions" refers to large-scale language models and other artificial intelligence technologies that use emotion recognition engines to understand a user's emotions and generate natural-sounding responses accordingly.

[1492] MODE FOR CARRYING OUT THE INVENTION

[1493] A system embodying this invention captures voice input from a user using an edge device and transmits the voice data to an external computer for processing. A specific implementation of this system is described below.

[1494] Audio capture and transmission

[1495] When a user speaks, the smart glasses used as a terminal capture the voice using a microphone. The captured voice data is converted into a digital signal by a voice capture device. This converted digital signal is transmitted to an external computer server via the Internet. Specifically, the data is transmitted in packets, enabling efficient data transmission.

[1496] Speech recognition and text conversion

[1497] The server receives the voice data packets sent from the device and restores them to the original voice data. The restored voice data is analyzed using a voice recognition engine and converted into corresponding text data. The voice recognition engine is typically a highly accurate software.

[1498] emotion recognition

[1499] Next, the server inputs the text data into an emotion engine to analyze the user's emotions. The emotion engine recognizes the user's emotions from the input text data based on multiple emotion categories. This emotion information is output in the form of numbers and tags.

[1500] Response Generation

[1501] The server uses the generated text data and emotion information to generate appropriate responses using a large-scale language model (LLM), which has the ability to provide natural responses that are in line with the user's input.

[1502] Transcribing and retransmitting responses

[1503] The generated response data is converted into voice data by a text-to-speech engine, which is also packetized and transmitted to the smart glasses via the internet.

[1504] Audio playback

[1505] The smart glasses, which are the terminals, decompress the received voice data and play it back to the user through the built-in speaker, allowing the user to receive voice responses to their questions and instructions.

[1506] For example, if a store staff member asks, "Hey, is there something I can help you with?" and the customer replies, "Hey, I don't know how to use this product," the system captures the voice and sends it to the server. The server analyzes the voice and recognizes the emotion of "the customer is having trouble." It then generates an appropriate response, such as "That's a problem. Can you explain in more detail?" and provides it to the staff member through the smart glasses.

[1507] An example of a specific prompt is, "Capture the customer's voice and generate an appropriate response based on their emotion."

[1508] This invention makes it possible to provide a more natural and appropriate response in real time according to the user's emotions.

[1509] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1510] Step 1:

[1511] The user speaks to the device

[1512] What it does: A user speaks into the smart glasses, and the smart glasses' built-in microphone captures the audio.

[1513] Input: User's voice

[1514] Output: Captured audio data

[1515] Step 2:

[1516] The device converts the voice data into a digital signal

[1517] Specific operation: Audio data captured by the microphone is converted into a digital signal by an analog-to-digital converter (ADC) inside the smart glasses.

[1518] Input: Captured audio data

[1519] Output: Digital audio signal

[1520] Step 3:

[1521] The terminal sends a digital signal to an external computer

[1522] What it does: Digital audio signals are packetized and sent over the internet to a remote server.

[1523] Input: Digital audio signal

[1524] Output: Packetized digital signal

[1525] Step 4:

[1526] The server receives and decompresses the packets and converts the audio data into text data.

[1527] Specific operation: The server receives and decompresses the packetized digital signal and converts it into text data using a speech recognition engine.

[1528] Input: Packetized digital signal

[1529] Output: Text data

[1530] Step 5:

[1531] The server inputs the text data into the emotion engine and analyzes the user's emotions.

[1532] Specific operation: The received text data is input into the emotion engine, which recognizes the user's emotion based on multiple emotion categories. The emotion engine outputs the results in the form of numbers and tags.

[1533] Input: Text data

[1534] Output: Emotion information (number, tag)

[1535] Step 6:

[1536] The server uses the text data and emotional information to generate an appropriate response.

[1537] Specific operation: The server inputs text data and emotional information into a large-scale language model (LLM) and generates an appropriate response based on the user's input.

[1538] Input: Text data, emotion information

[1539] Output: Response data

[1540] Step 7:

[1541] The response data generated by the server is converted into voice data, repacketized, and sent to the terminal.

[1542] Specific operation: The response data is converted into voice data by a text-to-speech engine (TTS), repacketized, and sent to the terminal via the Internet.

[1543] Input: Response data

[1544] Output: Packetized audio data

[1545] Step 8:

[1546] The device recovers the audio data it receives and plays it through the built-in speaker.

[1547] Specific operation: The smart glasses decompress the packetized audio data they receive and play the audio through their built-in speakers.

[1548] Input: Packetized audio data

[1549] Output: Playback of the recovered audio data

[1550] The user is then able to interact through the responses that are played back.

[1551] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1552] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1553] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1554] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1555] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1556] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1557] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1558] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1559] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1560] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1561] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1562] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1563] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1564] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1565] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1566] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1567] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1568] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1569] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1570] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1571] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1572] The following is further disclosed regarding the above embodiment.

[1573] (Claim 1)

[1574] means for receiving voice input from a user;

[1575] means for converting an audio input into a digital signal;

[1576] means for transmitting the digital signal to an external computer;

[1577] means for converting the digital signal into text data by an external computer;

[1578] means for analyzing the text data and generating response data based thereon;

[1579] means for converting the generated response data into voice data;

[1580] means for transmitting the converted voice data to a terminal;

[1581] means for playing audio data on the terminal;

[1582] A system including:

[1583] (Claim 2)

[1584] means for packetizing data when transmitting the digital signal;

[1585] a means for an external computer to receive and decompress the packets;

[1586] 10. The system of claim 1, comprising:

[1587] (Claim 3)

[1588] 10. The system of claim 1, further comprising means for using a large-scale language model as a means for analyzing the text data and generating response data.

[1589] "Example 1"

[1590] (Claim 1)

[1591] means for receiving voice input from a user;

[1592] means for converting an audio input into a digital signal;

[1593] means for transmitting the digital signal to an external computer via a network;

[1594] A means for restoring the digital signal to the original audio data by an external computer;

[1595] means including a speech recognition engine for converting voice data into text data;

[1596] means for analyzing the text data and generating response data based thereon;

[1597] means including a speech synthesis engine for converting the generated response data into speech data;

[1598] means for transmitting the converted voice data to a terminal via a network;

[1599] means for playing audio data on the terminal;

[1600] A system including:

[1601] (Claim 2)

[1602] means for packetizing data when transmitting the digital signal;

[1603] A means for an external computer to receive the packets and restore the original voice data;

[1604] 10. The system of claim 1, comprising:

[1605] (Claim 3)

[1606] 10. The system of claim 1, further comprising means for using a large-scale language model to analyze the text data and generate the response data.

[1607] "Application Example 1"

[1608] (Claim 1)

[1609] means for receiving voice input from a user;

[1610] means for converting an audio input into a digital signal;

[1611] means for transmitting the digital signal to an external computer;

[1612] means for converting the digital signal into text data by an external computer;

[1613] a means for inputting the generated text data into a large-scale language model to generate response data;

[1614] means for converting the generated response data into voice data;

[1615] means for transmitting the converted voice data to a terminal;

[1616] means for playing audio data on the terminal;

[1617] A means for a user to place an order by voice input;

[1618] A way to check order details in real time,

[1619] A means of voice command to customize your meal;

[1620] A system including:

[1621] (Claim 2)

[1622] means for packetizing data when transmitting the digital signal;

[1623] a means for an external computer to receive and decompress the packets;

[1624] means for using at least one speech synthesis engine in converting the response data into speech;

[1625] means for transmitting text data including order details to a server;

[1626] 10. The system of claim 1, comprising:

[1627] (Claim 3)

[1628] means for using a large-scale language model in analyzing the text data and generating the response data;

[1629] means for generating customized product information in real time based on a user order;

[1630] A means for generating appropriate prompts using a generative AI model; and

[1631] 10. The system of claim 1, comprising:

[1632] "Example 2: Combining Emotion Engines"

[1633] (Claim 1)

[1634] means for receiving voice input from a user;

[1635] means for converting an audio input into a digital signal;

[1636] means for transmitting the digital signal to an external computer;

[1637] means for converting the digital signal into text data by an external computer;

[1638] means for analyzing the text data and generating response data based thereon;

[1639] means for converting the generated response data into voice data;

[1640] means for transmitting the converted voice data to a terminal;

[1641] means for playing audio data on the terminal;

[1642] means for analyzing a user's emotion and generating response data based on the emotion;

[1643] A system including:

[1644] (Claim 2)

[1645] means for packetizing data when transmitting the digital signal;

[1646] a means for an external computer to receive and decompress the packets;

[1647] 10. The system of claim 1, comprising:

[1648] (Claim 3)

[1649] 10. The system of claim 1, further comprising means for using a large-scale language model as a means for analyzing the text data and generating response data.

[1650] "Application example 2 when combining emotion engines"

[1651] (Claim 1)

[1652] means for receiving voice input from a user;

[1653] means for converting an audio input into a digital signal;

[1654] means for transmitting the digital signal to an external computer;

[1655] means for converting the digital signal into text data by an external computer;

[1656] means for analyzing the text data and generating response data based thereon;

[1657] means for converting the generated response data into voice data;

[1658] means for transmitting the converted voice data to a terminal;

[1659] means for playing audio data on the terminal;

[1660] means for recognizing a user's emotion in a customer interaction and generating an appropriate response based on the emotion;

[1661] A system including:

[1662] (Claim 2)

[1663] means for packetizing data when transmitting the digital signal;

[1664] a means for an external computer to receive and decompress the packets;

[1665] 10. The system of claim 1, comprising:

[1666] (Claim 3)

[1667] a means for analyzing the text data and generating response data using a large-scale language model;

[1668] 10. The system of claim 1, further comprising means for analyzing the customer's voice data with an emotion engine and providing an appropriate response. [Explanation of symbols]

[1669] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving voice input from a user; means for converting an audio input into a digital signal; means for transmitting the digital signal to an external computer; means for converting the digital signal into text data by an external computer; means for analyzing the text data and generating response data based thereon; means for converting the generated response data into voice data; means for transmitting the converted voice data to a terminal; means for playing audio data on the terminal; A system including:

2. means for packetizing data when transmitting the digital signal; a means for an external computer to receive and decompress the packets; The system of claim 1 , comprising:

3. 10. The system of claim 1, further comprising means for using a large-scale language model as a means for analyzing the text data and generating response data.

Images