System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing a voice of a user; means for converting the captured voice into digital data for transmission; means for converting the received voice data into text data; means for transmitting the text data to a generative model to generate a response; and means for receiving the generated response, converting the response into voice, and outputting the voice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention relates to a voice dialogue platform that utilizes generative models, and aims to provide a system that can be worn for long periods of time and enables natural and rapid dialogue with generative models. Conventional voice dialogue systems have had issues such as auditory strain during long periods of use and cost issues due to increased communication data volumes, limiting the user experience. [Means for solving the problem]

[0005] The present invention provides a system including means for capturing a user's voice, means for converting the captured voice into digital data and transmitting it, means for converting the received voice data into text data, means for transmitting the text data to a generative model to generate a response, and means for receiving the generated response, converting it into voice, and outputting it. Furthermore, the present invention includes means for outputting voice using bone conduction earphones, thereby reducing the strain on the ears' hearing even when worn for long periods of time. Furthermore, the present invention includes means for reducing the amount of communication data for received voice data, thereby reducing communication costs. This allows for natural and smooth voice interaction, making it possible to comfortably interact with a generative model in everyday life.

[0006] An "audio capturing means" is a device or function that collects audio signals in order to convert a user's speech into digital data.

[0007] "Means for converting into digital data and transmitting" refers to a device or function that converts the captured audio signal into a digital format and transmits the data to a designated server via a network.

[0008] "Means for converting into text data" refers to a device or function that converts digital voice data into text information using voice recognition technology.

[0009] The "means for sending to a generative model to generate a response" is a device or function that inputs text data into a generative model and generates an appropriate response based on the input.

[0010] The "means for converting into speech and outputting" refers to a device or function that converts the text response obtained from the generative model into speech data using speech synthesis technology and outputs it to the user.

[0011] Bone conduction earphones are earphones that transmit sound as vibrations to the inner ear through the skull. Because they do not block the ear, they are devices that put little strain on the hearing even when used for long periods of time.

[0012] A "data volume reduction means" is a device or technology that reduces communication costs and bandwidth by compressing or optimizing the amount of data sent and received.

[0013] "Natural and smooth voice dialogue" refers to dialogue in which the user can seamlessly and continuously exchange voice input and responses from the generative model. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention relates to a voice dialogue platform for capturing a user's voice and interacting with a generative model, which basically consists of a device worn by the user (including bone conduction earphones) and a server that hosts the generative model. This system converts the user's voice into a digital format, generates a response using a generative AI model, and finally transmits the response to the user as voice.

[0036] Program processing procedure

[0037] User operations

[0038] The user wears bone conduction earphones and asks the system a question, such as "What's the weather like today?" The voice is captured by a microphone built into the earphones.

[0039] Device audio capture and transmission

[0040] The device converts the captured audio into digital data, which is then transmitted to the server via Wi-Fi or a mobile network. At this time, the transmission module operates to send the audio data to the server in the appropriate data packet format.

[0041] Audio processing on the server

[0042] The server analyzes the received voice data and converts it into text data using a speech recognition engine. This text data is sent to a generative AI model, which generates an appropriate response based on the text. For example, in response to the question "What's the weather like today?", the generative AI model generates the response "It's sunny today."

[0043] Response transmission from the server to the terminal

[0044] The generated response text is sent back from the server to the user's terminal, where it is converted into data packets and sent again over the network to the terminal.

[0045] Device speech synthesis and output

[0046] The device converts the received text data into speech using a speech synthesis engine. This speech data is transmitted to the user using bone conduction earphones. Bone conduction earphones do not block the ears, so they put less strain on the ears even when worn for long periods of time, allowing the user to receive audio information naturally.

[0047] Specific examples

[0048] When a user asks "What's the weather like today?" the following actions occur:

[0049] 1. User asks a question: The user puts on the bone conduction earphones and asks, "What's the weather like today?"

[0050] 2. The device captures the audio: The device captures the user's audio, converts it into digital data, and sends it to the server.

[0051] 3. Server converts speech to text: The server converts the speech data into text and sends it to the generative AI model.

[0052] 4. The generative AI model generates a response: The generative AI model generates the response, "The weather is sunny today."

[0053] 5. The server sends the response to the terminal: The server sends the generated response text to the terminal.

[0054] 6. The device converts the response into speech: The device synthesizes speech and transmits it to the user through bone conduction earphones.

[0055] 7. User hears response: The user hears the audio response "The weather is sunny today."

[0056] In this way, the present invention makes it possible to realize natural voice dialogue using a generative model, allowing users to use the system comfortably for extended periods of time.

[0057] The processing flow will be explained below.

[0058] Program processing steps

[0059] Step 1: User asks a question

[0060] The user puts on the bone conduction earphones and asks, "What's the weather like today?" The sound is captured by a microphone built into the bone conduction earphones.

[0061] Step 2: Your device captures audio

[0062] The terminal captures the user's voice in real time, and the voice capture module operates to convert the voice signal into digital voice data.

[0063] Step 3: The device sends the audio data to the server

[0064] The device transmits the converted digital audio data to the server via Wi-Fi or mobile network, and the transmission module creates data packets and sends them to the server.

[0065] Step 4: The server receives the audio data

[0066] The server receives the voice data sent from the terminal, and the receiving module analyzes the complete data packet to obtain the voice data.

[0067] Step 5: The server converts the audio data into text data.

[0068] The server converts the received voice data into text data using a voice recognition engine, which analyzes the voice data and generates corresponding text.

[0069] Step 6: The server sends the text data to the generative model

[0070] The server sends the converted text data, "What's the weather like today?", to the generative model, which analyzes the input text data and generates an appropriate response.

[0071] Step 7: The generative model generates the response text

[0072] The generative model generates the response "The weather is sunny today" based on the text data sent. The generative model calculates and generates the response text.

[0073] Step 8: The server sends the generated response to the device

[0074] The server returns the generated response text to the user terminal, and the transmission module converts the response text into data packets and sends them to the terminal.

[0075] Step 9: Your device receives the response text

[0076] The terminal receives the response text sent from the server, and the receiving module analyzes the data packet to obtain the complete response text.

[0077] Step 10: The device performs speech synthesis

[0078] The device performs speech synthesis based on the text data "Today's weather is sunny" and converts it into voice data. The speech synthesis engine converts the text into synthetic speech.

[0079] Step 11: The device plays audio through the bone conduction earphones

[0080] The device plays the synthesized voice through bone conduction earphones, and the bone conduction speaker module outputs the synthesized voice in a format that is easy for the user to hear.

[0081] Step 12: User listens to response

[0082] The user hears the voice response "Today's weather is sunny" through the bone conduction earphones, allowing the user to obtain weather information.

[0083] Example 1

[0084] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0085] Voice dialogue systems are required to be comfortable and not cause fatigue even when used for long periods of time. However, conventional systems have limitations in speech recognition accuracy and real-time response generation, which reduces user convenience. Another issue is that earphones worn for long periods of time can easily strain the hearing. Furthermore, it has been pointed out that delays are likely to occur during the voice data conversion and transmission process, making it difficult to achieve smooth dialogue.

[0086] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0087] In this invention, the server includes digital signal processing means for converting voice data into digital format, means for converting voice data into text data using a voice recognition engine, and speech synthesis means for converting the generated response from text to voice. This enables highly accurate voice recognition and response generation in real time, resulting in a system that users can use comfortably for long periods of time. In particular, the use of bone conduction earphones reduces hearing strain and allows for long-term wearing. Furthermore, the use of high-performance digital signal processing minimizes delays in data conversion, enabling smooth dialogue.

[0088] A "user" is a person who uses this system to engage in voice interaction.

[0089] The term "means for capturing voice" refers to a device or mechanism that can capture the user's voice as a signal.

[0090] "Means for converting into digital data and transmitting" refers to a mechanism or method for converting the captured analog audio signal into digital data and transmitting it to another device via a network.

[0091] "Means for converting voice data to text data" refers to techniques or methods for analyzing received voice signals and converting them into corresponding text data.

[0092] "Means for sending to a generative model to generate a response" refers to technology or a system that inputs the converted text data into a generative artificial intelligence to generate an appropriate response.

[0093] "Means for receiving a generated response, converting it into speech, and outputting it" refers to a method or device for converting a text response returned from a generative model into speech and providing it to a user.

[0094] "Digital signal processing means" refers to a technique or device for converting an analog audio signal into a digital signal.

[0095] A "speech recognition engine" is software or a system that analyzes speech signals and converts them into text data.

[0096] "Speech synthesis means" refers to technology or devices for converting text data into speech signals.

[0097] Bone conduction earphones are earphones that transmit sound through bones, allowing you to hear audio without blocking your ears.

[0098] A "data packet" is a unit of information packed into a specific format for sending and receiving data over a network.

[0099] This invention relates to a voice dialogue platform for capturing a user's voice and interacting with a generative model. Its basic configuration consists of a device worn by the user (including bone conduction earphones) and a server hosting the generative model. The system converts the user's voice into a digital format, generates a response using a generative AI model, and finally transmits the response to the user as voice.

[0100] The device includes bone conduction earphones with a built-in, highly sensitive microphone. When the user asks a question such as "What's the weather today?", the microphone captures the audio signal. This captured audio is converted into digital data using Qualcomm's digital signal processing (DSP). The converted digital audio data is then sent to a server via Wi-Fi or a mobile network (e.g., LTE). This process uses a transmission module, which sends the audio data to the server in the form of UDP packets.

[0101] The server analyzes the received voice data and converts it into text data using the Google® Cloud Speech-to-Text API. The converted text data is sent as a prompt to a generative AI model such as GPT-3®. For example, in response to the question "What's the weather like today?", the generative AI model generates the response "It's sunny today."

[0102] The generated response text is sent from the server to the device in TCP / IP packet format. The device converts the received text data into voice data using the Google Cloud Text-to-Speech API. The generated voice data is transmitted to the user using bone conduction earphones. These earphones do not block the ears, so they put less strain on the hearing and can be used comfortably for long periods of time.

[0103] As a concrete example of this behavior, when a user asks "What's the weather like today?":

[0104] 1. User asks a question: The user puts on the bone conduction earphones and asks, "What's the weather like today?"

[0105] 2. The device captures the voice: The device captures the user's voice, converts it into digital data using a Qualcomm DSP, and sends it to the server.

[0106] 3. The server converts the speech to text: The server uses the Google Cloud Speech-to-Text API to convert the speech data to text and sends this text to a generative AI model (e.g., GPT-3).

[0107] 4. The generative AI model generates a response: The generative AI model generates the response "The weather is sunny today."

[0108] 5. The server sends the response to the terminal: The server sends the generated response text to the terminal as a TCP / IP packet.

[0109] 6. The device converts the response into speech and outputs it: The device uses the Google Cloud Text-to-Speech API to convert the text into speech and transmits it to the user through bone conduction earphones.

[0110] 7. User hears response: The user hears the audio response "The weather is sunny today."

[0111] In this way, the present invention makes it possible to realize natural voice dialogue using a generative model, allowing users to use the system comfortably for extended periods of time.

[0112] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0113] Step 1:

[0114] The user wears bone conduction earphones and asks the system a question such as "What's the weather like today?" This voice is captured by a microphone built into the earphones. The input is the user's voice, and the output is the captured analog voice signal. At this stage, the user's voice is acquired in real time.

[0115] Step 2:

[0116] The device converts the captured analog voice signal into digital data using digital signal processing (DSP). Specifically, the DSP built into the device samples the voice signal and converts the analog voice into digital data. The input is an analog voice signal, and the output is digital voice data. This digital data is sent to a server via Wi-Fi or a mobile network.

[0117] Step 3:

[0118] The server analyzes the received digital voice data and converts it into text data using a voice recognition engine. Specifically, the server converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is digital voice data, and the output is text data. For example, the voice saying "What's the weather like today?" is converted into text "What's the weather like today?"

[0119] Step 4:

[0120] The server sends the converted text data to a generative AI model to generate an appropriate response. A generative AI model (e.g., GPT-3) is used for this. The server inputs the text data as a prompt into the generative AI model and receives the generated response. The input is the text data "What's the weather like today?" and the output is the response text "Today's weather is sunny."

[0121] Step 5:

[0122] The server sends the generated response text to the terminal. The server converts the response text into data packets (using the TCP / IP protocol) and sends them again over the network to the terminal. The input is the generated response text, and the output is the data converted into network packets.

[0123] Step 6:

[0124] The device converts the received response text data into voice data using a speech synthesis engine. Specifically, the device converts text data into voice data using the Google Cloud Text-to-Speech API. The input is the response text data, and the output is the synthesized voice data.

[0125] Step 7:

[0126] The generated voice data is transmitted to the user via bone conduction earphones. The terminal sends the synthesized voice data to the earphones, which transmit the voice to the user using bone conduction technology. The input is the synthesized voice data, and the output is the voice that is actually transmitted to the user. The user can hear the voice response, "Today's weather is sunny."

[0127] (Application example 1)

[0128] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0129] In recent years, there has been a demand for providing customers with fast and accurate information in brick-and-mortar stores. However, it is difficult for store employees to constantly respond to customers, and maintaining customer satisfaction is especially challenging during busy times and peak hours. In addition, existing digital interfaces and guidance systems are complex to operate, making them difficult to use, especially for elderly people and those unfamiliar with technology. For this reason, a new voice dialogue system is needed to efficiently guide customers in brick-and-mortar stores and improve customer convenience.

[0130] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0131] In this invention, the server includes means for capturing the user's voice, means for converting the captured voice into digital data and transmitting it, means for converting the received voice data into text data, means for transmitting the text data to a generative model to generate a response, means for receiving the generated response, converting it into voice and outputting it, and means for providing information about products, services, and sales in a physical store. This allows customers to instantly find out the location of products in the store, special sale information, etc., simply by asking a question by voice.

[0132] "User voice" refers to voice data in natural language spoken by a system user.

[0133] A "capturing means" is a device such as a microphone or bone conduction earphone for acquiring an audio signal.

[0134] "Convert to digital data" is the process of converting analog audio signals into a digital format.

[0135] "Transmitting means" is a network interface for transmitting digital data to other devices or servers.

[0136] "Means for converting to text data" refers to the process of converting voice data into text format using voice recognition technology.

[0137] A "generative model" is an artificial intelligence model that generates appropriate responses based on input text data.

[0138] A "means for generating a response" is a process that uses a generative model to create a response based on text data.

[0139] "Means for converting into voice and outputting" refers to the process of converting text data into voice data using voice synthesis technology and making it available to the user.

[0140] A "brick and mortar store" is a physical store where customers visit in person to transact goods or services.

[0141] "Means for providing product, service and sale information" is a process for providing customers with product locations in stores, service details, special sale information, etc., through the generated response.

[0142] Bone conduction earphones are devices that transmit sound directly to the inner ear through the skull, bypassing the ear.

[0143] As a specific example of a distinctive application of the present invention, we will present a specific example of a system for implementing a "store navigation assistant." This system allows users to ask questions about products, services, and sales information by voice in a physical store and receive appropriate responses.

[0144] Configuration and Operation

[0145] Hardware and software used

[0146] Audio capture device: Bone conduction earphones have a built-in microphone that captures the user's voice.

[0147] Device: The device worn by the user converts the captured audio into digital data and sends it to the server.

[0148] Server: The server converts the voice data into text data using the Google Cloud Speech-to-Text API and sends this text data to the OpenAI (registered trademark) generative AI model. The generated response text is processed within the server and converted into voice data using the Google Cloud Text-to-Speech API. Finally, this voice data is sent to the device and transmitted to the user via bone conduction earphones.

[0149] Data processing and calculation

[0150] 1. Converting voice to digital data: The user's voice is converted into digital data by an encoder on the device, which is then sent to a server via Wi-Fi or a mobile network.

[0151] 2. Speech Recognition: The server converts the voice data into text data using the Google Cloud Speech-to-Text API. Here, the speech recognition engine analyzes the voice signal and converts it into text format.

[0152] 3. Response generation: A generative AI model (OpenAI's ChatGPT® API) generates an appropriate response based on the text data. The system uses the generative AI model to create a response to the user's question entered as a prompt.

[0153] 4. Speech synthesis: The generated text response is converted into natural-sounding speech using the Google Cloud Text-to-Speech API. This conversion is performed by a speech synthesis engine.

[0154] 5. Audio output: The device transmits the received audio data to the user through bone conduction earphones.

[0155] Specific use cases

[0156] As a concrete example, consider a situation where a user is in a physical store and asks, "Where are the sale items?" The processing steps in this case are as follows:

[0157] 1. A user asks, "Where are the sales?" while in a brick-and-mortar store.

[0158] 2. The device captures this audio, converts it into digital data, and sends it to the server.

[0159] 3. The server converts the voice data into text data and sends it to the "generative AI model" as a prompt.

[0160] 4. The generative AI model generates the response, "The sale items are located near the center of the store."

[0161] 5. The server converts this response into audio data and sends it back to the terminal.

[0162] 6. The device outputs a response through bone conduction earphones, telling the user, "The sale items are located near the center of the store."

[0163] Prompt Sentence Examples

[0164] User Question: "Where are the sale items?"

[0165] Prompt: "Where are the sale items? They are near the center of the store."

[0166] As described above, the system for implementing this invention is designed to provide users with information about physical stores quickly and appropriately, significantly improving the efficiency of customer service and increasing customer satisfaction.

[0167] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0168] Step 1:

[0169] The user puts on the bone conduction earphones and asks a question.

[0170] Input: User's natural language speech.

[0171] How it works: A user says, "Where are the sale items?" in a physical store. The microphone inside the bone conduction earphones captures this voice.

[0172] Output: Analog audio signal.

[0173] Step 2:

[0174] The device converts the captured audio into digital data and sends it to the server.

[0175] Input: Analog audio signal.

[0176] How it works: An encoder inside the device converts analog audio signals into digital audio data and sends it to a server via Wi-Fi or a mobile network.

[0177] Output: Digital audio data.

[0178] Step 3:

[0179] The server converts the received digital voice data into text data.

[0180] Input: Digital audio data.

[0181] How it works: The server uses the Google Cloud Speech-to-Text API to convert the voice data into text data. The speech recognition engine analyzes the voice signal and converts the data into text.

[0182] Output: Text data.

[0183] Step 4:

[0184] The server sends the text data to the generative AI model, which generates a response.

[0185] Input: Text data.

[0186] How it works: The generative AI model (OpenAI's ChatGPT API) generates an appropriate response based on the text data. The server sends this prompt to the generative AI model, which then receives a response such as "The sale items are located near the center of the store."

[0187] Output: The response text.

[0188] Step 5:

[0189] The server converts the generated response text into audio data.

[0190] Input: Response text.

[0191] How it works: The server uses the Google Cloud Text-to-Speech API to convert the generated text response into audio data. A speech synthesis engine analyzes this text and processes the data to convert it into natural-sounding speech.

[0192] Output: Audio data.

[0193] Step 6:

[0194] The audio data received by the device is transmitted to the user through bone conduction earphones.

[0195] Input: Audio data.

[0196] Specific operation: The terminal plays audio data and provides audio information to the user through bone conduction earphones. The user hears the voice guidance, "The sale items are located near the center of the store."

[0197] Output: Audio information to the user.

[0198] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0199] This invention combines a speech dialogue platform that utilizes a generative model with an emotion engine, and relates to a system that realizes more natural and adaptive dialogue by recognizing the user's emotions and generating responses accordingly. The system includes a user device (including bone conduction earphones), a server that hosts the generative model, and an emotion engine.

[0200] Program processing procedure

[0201] User operations

[0202] The user wears bone conduction earphones and asks the system a question, such as "What's the weather like today?" The voice is captured by a microphone built into the earphones.

[0203] Device audio capture and transmission

[0204] The device converts the captured audio into digital data, which is then transmitted to the server via Wi-Fi or a mobile network. At this time, the transmission module operates to send the audio data to the server in the appropriate data packet format.

[0205] Audio processing on the server

[0206] The server analyzes the received voice data and converts it into text data using a speech recognition engine. It also recognizes the user's emotions using an emotion engine. The emotion engine analyzes the tone, tempo, and pitch of the voice to identify the user's emotional state.

[0207] Emotion recognition by emotion engine

[0208] The emotion engine recognizes the user's emotional state, such as "angry" or "happy," based on features extracted from the user's voice data, allowing responses to be provided in a more appropriate emotional tone.

[0209] Server response generation

[0210] The server sends the text data and the recognized emotion information to the generative model, which then generates a response based on the input data. For example, in response to the question "What's the weather like today?", the model calculates and generates the response "The weather is sunny today."

[0211] Response transmission from the server to the terminal

[0212] The generated response text is sent back from the server to the user's terminal, where it is converted into data packets and sent again over the network to the terminal.

[0213] Device speech synthesis and output

[0214] The device converts the received text data into speech using a speech synthesis engine. This speech data is transmitted to the user using bone conduction earphones. Bone conduction earphones do not block the ears, so they put less strain on the ears even when worn for long periods of time, allowing the user to receive audio information naturally.

[0215] Specific examples

[0216] When a user asks "What's the weather like today?" the following actions occur:

[0217] 1. User asks a question: The user puts on the bone conduction earphones and asks, "What's the weather like today?"

[0218] 2. The device captures the audio: The device captures the user's audio, converts it into digital data, and sends it to the server.

[0219] 3. The server converts the voice data into text: The server converts the voice data into text and recognizes emotions using the emotion engine.

[0220] 4. The emotion engine recognizes emotions: If the emotion “angry” is recognized, that information is also sent to the generative model.

[0221] 5. The generative model generates a response: The generative model generates a response with the appropriate emotional tone, such as "The weather is sunny today."

[0222] 6. The server sends the response to the terminal: The server sends the generated response text to the terminal.

[0223] 7. The device converts the response into speech: The device synthesizes speech and transmits it to the user through bone conduction earphones.

[0224] 8. User hears the response: The user hears the emotionally sensitive voice response: "The weather is sunny today."

[0225] In this way, the present invention combines a generative model and an emotion engine to realize natural and adaptive voice dialogue that responds to the user's emotions. This not only allows users to use the system comfortably for extended periods of time, but also provides a more personalized dialogue experience.

[0226] The processing flow will be explained below.

[0227] Program processing steps

[0228] Step 1: User asks a question

[0229] The user puts on the bone conduction earphones and asks, "What's the weather like today?" The sound is captured by a microphone built into the bone conduction earphones.

[0230] Step 2: Your device captures audio

[0231] The terminal captures the user's voice in real time, and the voice capture module operates to convert the voice signal into digital voice data.

[0232] Step 3: The device sends the audio data to the server

[0233] The device transmits the converted digital audio data to the server via Wi-Fi or mobile network, and the transmission module creates data packets and sends them to the server.

[0234] Step 4: The server receives the audio data

[0235] The server receives the voice data sent from the terminal, and the receiving module analyzes the complete data packet to obtain the voice data.

[0236] Step 5: The server converts the audio data into text data.

[0237] The server converts the received voice data into text data using a voice recognition engine, which analyzes the voice data and generates corresponding text.

[0238] Step 6: The server recognizes the user's emotion using the emotion engine.

[0239] The server inputs the generated text data and voice data features into an emotion engine to recognize the user's emotions. The emotion engine analyzes the tone, tempo, and pitch of the voice to identify the user's emotional state.

[0240] Step 7: The emotion engine identifies the user's emotion

[0241] The emotion engine recognizes, for example, that the user is angry. The recognized emotion information is input into the generative model.

[0242] Step 8: The server sends the text data and emotion information to the generative model

[0243] The server sends the text data and the recognized emotions to the generative model, which generates a response based on the input data.

[0244] Step 9: The generative model generates the response text

[0245] Based on the text data and emotional information sent, the generative model generates a response with an appropriate emotional tone, such as "Today's weather is sunny."

[0246] Step 10: The server sends the generated response to the terminal

[0247] The server returns the generated response text to the user terminal, and the transmission module converts the response text into data packets and sends them to the terminal.

[0248] Step 11: The device receives the response text

[0249] The terminal receives the response text sent from the server, and the receiving module analyzes the data packet to obtain the complete response text.

[0250] Step 12: The device performs speech synthesis

[0251] The device synthesizes the text data and converts it into voice data. The speech synthesis engine converts the text into synthesized voice.

[0252] Step 13: The device plays audio through the bone conduction earphones

[0253] The device plays the synthesized voice through bone conduction earphones, and the bone conduction speaker module outputs the synthesized voice in a format that is easy for the user to hear.

[0254] Step 14: User listens to response

[0255] The user hears the emotionally sensitive voice response "Today's weather is sunny" through the bone conduction earphones, allowing the user to obtain weather information.

[0256] The above is a specific processing flow of the invention that combines an emotion engine, allowing users to enjoy natural and adaptive dialogue according to their emotions.

[0257] Example 2

[0258] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0259] Conventional voice dialogue systems have difficulty recognizing the user's emotions and generating appropriate responses. In particular, responses that do not take the user's emotional state into account are insufficient for realizing natural and adaptive dialogue and may impair the user experience. Furthermore, prolonged use can cause significant strain on the hearing and lead to a lack of comfort.

[0260] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0261] In this invention, the server includes means for receiving and analyzing voice data, means for converting the received voice data into text data, and means for analyzing features of the voice data to identify the user's emotional state, thereby enabling a natural and adaptive response according to the user's emotional state.

[0262] "Means for capturing user voice" refers to technology that captures user-generated voice using a microphone or other input device.

[0263] "Means for converting captured audio into digital data and transmitting it" refers to a technology for converting the acquired audio signal into a digital signal and transmitting it to a server via a communication network.

[0264] "Means for receiving and analyzing transmitted voice data" refers to a technology for receiving voice data transmitted via a network and analyzing that data.

[0265] The "means for converting received voice data into text data" refers to a technology that uses voice recognition technology to convert voice data into corresponding text format data.

[0266] "Means for identifying a user's emotional state by analyzing the features of audio data" refers to technology that analyzes features such as tone, tempo, and pitch extracted from audio data to estimate a user's emotional state.

[0267] "Means for sending the identified emotional state and text data to a generative model to generate an appropriate response" refers to a technology that inputs the analyzed emotional state and converted text data into a generative model to generate an appropriate response based thereon.

[0268] "Means for receiving the generated response, converting it into speech, and outputting it" refers to a technology for receiving the generated text response, converting the text into speech using a speech synthesis engine, and finally outputting it to the user.

[0269] "Bone conduction earphones" refer to acoustic devices that transmit sound to the user's inner ear through the bones without blocking the ear.

[0270] "Hearing reduction measures" refers to technologies or devices designed to reduce the user's hearing, even when used for extended periods of time.

[0271] This invention combines a speech dialogue platform that utilizes a generative model with an emotion engine, and relates to a system that realizes more natural and adaptive dialogue by recognizing the user's emotions and generating responses accordingly. The system includes a user device (including bone conduction earphones), a server that hosts the generative model, and an emotion engine.

[0272] Here, the user wears bone conduction earphones and asks the system a question, such as "What's the weather like today?" This voice is captured by a microphone built into the earphones. The device uses an analog-to-digital converter (ADC) to convert the captured voice into digital data. The converted digital voice data is then sent to the server via Wi-Fi or a mobile network. At this time, the transmission module operates and sends the voice data to the server in the appropriate data packet format.

[0273] The server analyzes the received voice data and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text). It then uses an emotion engine (e.g., IBM Watson® Tone Analyzer) to recognize the user's emotions. The emotion engine analyzes voice features such as tone, tempo, and pitch to identify the user's emotional state (e.g., "angry" or "happy").

[0274] The emotion information and text data recognized by the emotion engine are sent to a generative model (e.g., OpenAI GPT-4 (registered trademark)). The generative model generates an appropriate response based on the input data. For example, in response to the question, "What's the weather like today?", the response generated is, "It's sunny today." This response takes into account the user's emotions.

[0275] The generated response text is sent back from the server to the user's device. The server converts the response text into data packets and sends them again over the network to the device. The device converts the received text data into speech using a speech synthesis engine (e.g., Amazon Polly). This speech data is transmitted to the user using bone conduction earphones. Because bone conduction earphones do not block the ears, they put less strain on the ears even when worn for long periods of time, allowing the user to receive audio information naturally.

[0276] As a concrete example, if a user asks "What's the weather like today?" the following actions occur:

[0277] The user asks a question: "What's the weather like today?"

[0278] The device captures the audio, converts it into digital data, and sends it to a server.

[0279] The server converts the speech into text and uses an emotion engine to recognize emotions.

[0280] The emotion engine recognizes emotions: if the emotion "angry" is recognized, it also sends that information to the generative model.

[0281] The generative model generates a response: "The weather is sunny today," with the appropriate emotional tone.

[0282] The server sends a response to the terminal.

[0283] The device converts the response into audio and transmits it to the user through bone conduction earphones.

[0284] The user hears the response: "The weather is sunny today." The user hears the emotionally sensitive voice response.

[0285] By combining a generative model and an emotion engine, the present invention makes it possible to realize natural and adaptive voice dialogue that responds to the user's emotions. This not only allows users to use the device comfortably for extended periods of time, but also allows them to enjoy a more personalized dialogue experience.

[0286] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0287] Step 1:

[0288] A user puts on bone conduction earphones and asks a question such as "What's the weather like today?" The sound is captured by a microphone built into the bone conduction earphones. The input is the user's voice, and the output is an analog audio signal.

[0289] Step 2:

[0290] The terminal converts the captured analog voice signal into digital data using an ADC (Analog-to-Digital Converter). The input is an analog voice signal, and the output is digital voice data. This data conversion process preserves the waveform information of the voice signal.

[0291] Step 3:

[0292] The terminal transmits the converted digital audio data to the server via Wi-Fi or mobile network. The input is digital audio data, and the output is audio data in the form of data packets. The transmission module uses the TCP / IP stack to divide the audio data into small packets.

[0293] Step 4:

[0294] The server recombines the received voice data in data packets and converts it into text data using a speech recognition engine. The input is voice data in data packets, and the output is text data. A commonly used cloud-based API (e.g., Google Cloud Speech-to-Text) is used as the speech recognition engine.

[0295] Step 5:

[0296] The server passes the text data to the emotion engine for analysis. The input is text data, and the output is the user's emotional information. The emotion engine analyzes features such as tone, tempo, and pitch of the voice to identify emotions such as "angry" or "happy."

[0297] Step 6:

[0298] The server sends the text data and the identified emotion information to the generative AI model. The input is the text data and emotion information, and the output is the generated response text. The generative AI model (e.g., OpenAI GPT-4) generates an appropriate response based on this prompt.

[0299] Step 7:

[0300] The server converts the generated response text into data packets and sends them to the terminal again over the network. The input is the generated response text, and the output is the response text in data packet format.

[0301] Step 8:

[0302] The device converts the received response text into speech using a speech synthesis engine (e.g., Amazon Polly). The input is the response text, and the output is voice data. The speech synthesis engine converts the text data into natural-sounding speech, and the voice data is transmitted to the user via bone conduction earphones.

[0303] Step 9:

[0304] The user listens to the generated audio through bone conduction earphones. The input is audio data, and the output is the audio perceived by the user. Bone conduction earphones allow users to receive audio information naturally without blocking their ears, and put less strain on their hearing even when used for long periods of time.

[0305] (Application example 2)

[0306] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0307] Conventional voice dialogue systems have been limited to simple question-and-answer systems without considering the user's emotions, resulting in poor dialogue quality. Furthermore, when dealing with customers in physical stores, it has been difficult to understand the customer's emotions and respond accordingly.

[0308] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing the user's voice, means for converting the captured voice into digital data and transmitting it, means for converting the received voice data into text data, means for transmitting the text data to a generative model to generate a response, means for receiving the generated response, converting it into voice and outputting it, means for analyzing and recognizing the emotional state, and means for adjusting the generated response based on the emotional state. This enables natural and adaptive dialogue according to the customer's emotions, making it possible to improve the quality of customer service in physical stores.

[0309] "Means for capturing user voice" refers to a mechanism for acquiring the voice uttered by the user as digital data.

[0310] "Means for converting captured audio into digital data and transmitting it" refers to technology that processes audio signals, converts them into a digital format, and transmits them to a server via a communication means.

[0311] "Means for converting received voice data into text data" refers to the process by which the server uses voice recognition technology to analyze the voice data and convert it into text information.

[0312] "Means of sending to a generative model to generate a response" refers to a method of inputting text data into a generative AI model to automatically generate an appropriate response.

[0313] The "means for receiving the generated response, converting it into speech, and outputting it" is a system that converts the generated text response into speech using speech synthesis technology and transmits it to the user.

[0314] The "means for analyzing and recognizing emotional states" is a technology for analyzing emotional features from the user's voice and identifying the user's current emotional state.

[0315] The "means for adjusting the generated response based on the emotional state" is a method for adjusting the content and tone of the generated response depending on the emotional state of the user recognized by the emotion engine.

[0316] "Means for outputting sound using bone conduction earphones" refers to an acoustic device that uses bone conduction to transmit sound to the user, and is a technology that transmits sound without blocking the ears.

[0317] "Means for communicating with customers via smart glasses worn by store staff" refers to a system in which store staff wearing smart glasses respond to customers' questions or requests through voice or visual information.

[0318] The present invention is a system that captures a user's voice data, analyzes it using a generative AI model and an emotion engine, generates a response, and conveys it to the user.

[0319] In this system, a user puts on the smart glasses and speaks to ask or request a question. The smart glasses' built-in microphone captures the voice and converts it into digital data. The digital data is then sent to a server via Wi-Fi or a mobile network. A module then operates to transmit the voice data in the appropriate data packet format.

[0320] The server analyzes the received voice data and converts it into text data using a voice recognition engine. For voice recognition, technologies such as Google Speech Recognition can be used. Next, an emotion engine analyzes features such as tone, tempo, and pitch of the voice to recognize the user's emotion. This procedure identifies the user's emotional state, such as whether they are angry or happy. Emotion analysis technologies such as IBM Watson and Microsoft® Azure® Cognitive Services can be used for emotion recognition.

[0321] Based on the analysis results, the emotion information and text data obtained from the emotion engine are sent to a generative AI model, which can use advanced natural language processing models such as GPT-3 or T5 to generate a response with an appropriate emotional tone. The generated response is then sent back from the server to the smart glasses and displayed on their screens.

[0322] Here's a concrete example: If a customer asks, "Tell me about this product," the smart glasses will capture the voice and send the following prompt to the generative AI model:

[0323] "When the customer is calm, they say, 'Tell me about this product.' We create that response."

[0324] The response generated by the generative AI model might be something like, "This product is very popular, especially this feature." This allows store associates to respond to customers in the right emotional tone, improving customer satisfaction.

[0325] The present invention can provide a more natural and adaptive user experience through emotion-based dialogue, which can be applied not only to customer service in brick-and-mortar stores but also to other voice dialogue applications, and has a wide range of applications.

[0326] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0327] Step 1:

[0328] The user wears the smart glasses and inputs questions or requests by voice. The voice is captured by the microphone built into the smart glasses. The input data is a voice signal, and the captured voice signal is passed directly to the next step.

[0329] Step 2:

[0330] The device (smart glasses) converts the captured audio into digital data using a built-in audio-to-digital conversion module, which is then represented as audio samples and sent to a server via Wi-Fi or a mobile network.

[0331] Step 3:

[0332] The server analyzes the received digital voice data using a voice recognition engine and converts it into text data. The voice recognition engine used here is assumed to be "Google Speech Recognition." The input data is digital voice data, and the output data is recognized text data.

[0333] Step 4:

[0334] The server then uses an emotion engine to analyze the text data and its speech features to recognize the user's emotional state. Possible emotion recognition technologies used here include IBM Watson and Microsoft Azure Cognitive Services. The input data are the recognized text data and speech features, and the output data is metadata that represents the user's emotional state.

[0335] Step 5:

[0336] The server sends the emotional information and text data obtained from the emotion engine to the generative model. The generative AI model uses algorithms such as GPT-3 and T5 and receives text data and emotional states as input. The model generates an appropriate response based on this. The generated response is output in text format.

[0337] Step 6:

[0338] The server sends the generated text response back to the terminal (smart glasses). The output data is the text response, which is sent to the smart glasses via the network.

[0339] Step 7:

[0340] The device (smart glasses) converts the received text data into voice data using speech synthesis technology. A speech synthesis engine is used to create a voice response with an appropriate emotional tone. The input data is a text response, and the output data is voice data.

[0341] Step 8:

[0342] The device transmits the generated voice to the user through bone conduction earphones, allowing the user to hear the voice response without covering their ears. The output data is voice, and this is what reaches the user.

[0343] The above steps enable natural voice interaction that takes into consideration the customer's emotions. For example, if a customer asks, "Tell me about this product," the response can be provided in an appropriate tone, such as, "This product is very popular. This feature in particular is attracting attention." This process improves the quality of customer service and increases customer satisfaction.

[0344] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0345] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0346] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0347] [Second embodiment]

[0348] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0349] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0350] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0351] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0352] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0353] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0354] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0355] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0356] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0357] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0358] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0359] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0360] This invention relates to a voice dialogue platform for capturing a user's voice and interacting with a generative model, which basically consists of a device worn by the user (including bone conduction earphones) and a server that hosts the generative model. This system converts the user's voice into a digital format, generates a response using a generative AI model, and finally transmits the response to the user as voice.

[0361] Program processing procedure

[0362] User operations

[0363] The user wears bone conduction earphones and asks the system a question, such as "What's the weather like today?" The voice is captured by a microphone built into the earphones.

[0364] Device audio capture and transmission

[0365] The device converts the captured audio into digital data, which is then transmitted to the server via Wi-Fi or a mobile network. At this time, the transmission module operates to send the audio data to the server in the appropriate data packet format.

[0366] Audio processing on the server

[0367] The server analyzes the received voice data and converts it into text data using a speech recognition engine. This text data is sent to a generative AI model, which generates an appropriate response based on the text. For example, in response to the question "What's the weather like today?", the generative AI model generates the response "It's sunny today."

[0368] Response transmission from the server to the terminal

[0369] The generated response text is sent back from the server to the user's terminal, where it is converted into data packets and sent again over the network to the terminal.

[0370] Device speech synthesis and output

[0371] The device converts the received text data into speech using a speech synthesis engine. This speech data is transmitted to the user using bone conduction earphones. Bone conduction earphones do not block the ears, so they put less strain on the ears even when worn for long periods of time, allowing the user to receive audio information naturally.

[0372] Specific examples

[0373] When a user asks "What's the weather like today?" the following actions occur:

[0374] 1. User asks a question: The user puts on the bone conduction earphones and asks, "What's the weather like today?"

[0375] 2. The device captures the audio: The device captures the user's audio, converts it into digital data, and sends it to the server.

[0376] 3. Server converts speech to text: The server converts the speech data into text and sends it to the generative AI model.

[0377] 4. The generative AI model generates a response: The generative AI model generates the response, "The weather is sunny today."

[0378] 5. The server sends the response to the terminal: The server sends the generated response text to the terminal.

[0379] 6. The device converts the response into speech: The device synthesizes speech and transmits it to the user through bone conduction earphones.

[0380] 7. User hears response: The user hears the audio response "The weather is sunny today."

[0381] In this way, the present invention makes it possible to realize natural voice dialogue using a generative model, allowing users to use the system comfortably for extended periods of time.

[0382] The processing flow will be explained below.

[0383] Program processing steps

[0384] Step 1: User asks a question

[0385] The user puts on the bone conduction earphones and asks, "What's the weather like today?" The sound is captured by a microphone built into the bone conduction earphones.

[0386] Step 2: Your device captures audio

[0387] The terminal captures the user's voice in real time, and the voice capture module operates to convert the voice signal into digital voice data.

[0388] Step 3: The device sends the audio data to the server

[0389] The device transmits the converted digital audio data to the server via Wi-Fi or mobile network, and the transmission module creates data packets and sends them to the server.

[0390] Step 4: The server receives the audio data

[0391] The server receives the voice data sent from the terminal, and the receiving module analyzes the complete data packet to obtain the voice data.

[0392] Step 5: The server converts the audio data into text data.

[0393] The server converts the received voice data into text data using a voice recognition engine, which analyzes the voice data and generates corresponding text.

[0394] Step 6: The server sends the text data to the generative model

[0395] The server sends the converted text data, "What's the weather like today?", to the generative model, which analyzes the input text data and generates an appropriate response.

[0396] Step 7: The generative model generates the response text

[0397] The generative model generates the response "The weather is sunny today" based on the text data sent. The generative model calculates and generates the response text.

[0398] Step 8: The server sends the generated response to the device

[0399] The server returns the generated response text to the user terminal, and the transmission module converts the response text into data packets and sends them to the terminal.

[0400] Step 9: Your device receives the response text

[0401] The terminal receives the response text sent from the server, and the receiving module analyzes the data packet to obtain the complete response text.

[0402] Step 10: The device performs speech synthesis

[0403] The device performs speech synthesis based on the text data "Today's weather is sunny" and converts it into voice data. The speech synthesis engine converts the text into synthetic speech.

[0404] Step 11: The device plays audio through the bone conduction earphones

[0405] The device plays the synthesized voice through bone conduction earphones, and the bone conduction speaker module outputs the synthesized voice in a format that is easy for the user to hear.

[0406] Step 12: User listens to response

[0407] The user hears the voice response "Today's weather is sunny" through the bone conduction earphones, allowing the user to obtain weather information.

[0408] Example 1

[0409] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0410] Voice dialogue systems are required to be comfortable and not cause fatigue even when used for long periods of time. However, conventional systems have limitations in speech recognition accuracy and real-time response generation, which reduces user convenience. Another issue is that earphones worn for long periods of time can easily strain the hearing. Furthermore, it has been pointed out that delays are likely to occur during the voice data conversion and transmission process, making it difficult to achieve smooth dialogue.

[0411] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0412] In this invention, the server includes digital signal processing means for converting voice data into digital format, means for converting voice data into text data using a voice recognition engine, and speech synthesis means for converting the generated response from text to voice. This enables highly accurate voice recognition and response generation in real time, resulting in a system that users can use comfortably for long periods of time. In particular, the use of bone conduction earphones reduces hearing strain and allows for long-term wearing. Furthermore, the use of high-performance digital signal processing minimizes delays in data conversion, enabling smooth dialogue.

[0413] A "user" is a person who uses this system to engage in voice interaction.

[0414] The term "means for capturing voice" refers to a device or mechanism that can capture the user's voice as a signal.

[0415] "Means for converting into digital data and transmitting" refers to a mechanism or method for converting the captured analog audio signal into digital data and transmitting it to another device via a network.

[0416] "Means for converting voice data to text data" refers to techniques or methods for analyzing received voice signals and converting them into corresponding text data.

[0417] "Means for sending to a generative model to generate a response" refers to technology or a system that inputs the converted text data into a generative artificial intelligence to generate an appropriate response.

[0418] "Means for receiving a generated response, converting it into speech, and outputting it" refers to a method or device for converting a text response returned from a generative model into speech and providing it to a user.

[0419] "Digital signal processing means" refers to a technique or device for converting an analog audio signal into a digital signal.

[0420] A "speech recognition engine" is software or a system that analyzes speech signals and converts them into text data.

[0421] "Speech synthesis means" refers to technology or devices for converting text data into speech signals.

[0422] Bone conduction earphones are earphones that transmit sound through bones, allowing you to hear audio without blocking your ears.

[0423] A "data packet" is a unit of information packed into a specific format for sending and receiving data over a network.

[0424] This invention relates to a voice dialogue platform for capturing a user's voice and interacting with a generative model. Its basic configuration consists of a device worn by the user (including bone conduction earphones) and a server hosting the generative model. The system converts the user's voice into a digital format, generates a response using a generative AI model, and finally transmits the response to the user as voice.

[0425] The device includes bone conduction earphones with a built-in, highly sensitive microphone. When the user asks a question such as "What's the weather today?", the microphone captures the audio signal. This captured audio is converted into digital data using Qualcomm's digital signal processing (DSP). The converted digital audio data is then sent to a server via Wi-Fi or a mobile network (e.g., LTE). This process uses a transmission module, which sends the audio data to the server in the form of UDP packets.

[0426] The server analyzes the received voice data and converts it into text data using the Google Cloud Speech-to-Text API. The converted text data is sent as a prompt to a generative AI model such as GPT-3. For example, in response to the question "What's the weather like today?", the generative AI model generates the response "It's sunny today."

[0427] The generated response text is sent from the server to the device in TCP / IP packet format. The device converts the received text data into voice data using the Google Cloud Text-to-Speech API. The generated voice data is transmitted to the user using bone conduction earphones. These earphones do not block the ears, so they put less strain on the hearing and can be used comfortably for long periods of time.

[0428] As a concrete example of this behavior, when a user asks "What's the weather like today?":

[0429] 1. User asks a question: The user puts on the bone conduction earphones and asks, "What's the weather like today?"

[0430] 2. The device captures the voice: The device captures the user's voice, converts it into digital data using a Qualcomm DSP, and sends it to the server.

[0431] 3. The server converts the speech to text: The server uses the Google Cloud Speech-to-Text API to convert the speech data to text and sends this text to a generative AI model (e.g., GPT-3).

[0432] 4. The generative AI model generates a response: The generative AI model generates the response "The weather is sunny today."

[0433] 5. The server sends the response to the terminal: The server sends the generated response text to the terminal as a TCP / IP packet.

[0434] 6. The device converts the response into speech and outputs it: The device uses the Google Cloud Text-to-Speech API to convert the text into speech and transmits it to the user through bone conduction earphones.

[0435] 7. User hears response: The user hears the audio response "The weather is sunny today."

[0436] In this way, the present invention makes it possible to realize natural voice dialogue using a generative model, allowing users to use the system comfortably for extended periods of time.

[0437] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0438] Step 1:

[0439] The user wears bone conduction earphones and asks the system a question such as "What's the weather like today?" This voice is captured by a microphone built into the earphones. The input is the user's voice, and the output is the captured analog voice signal. At this stage, the user's voice is acquired in real time.

[0440] Step 2:

[0441] The device converts the captured analog voice signal into digital data using digital signal processing (DSP). Specifically, the DSP built into the device samples the voice signal and converts the analog voice into digital data. The input is an analog voice signal, and the output is digital voice data. This digital data is sent to a server via Wi-Fi or a mobile network.

[0442] Step 3:

[0443] The server analyzes the received digital voice data and converts it into text data using a voice recognition engine. Specifically, the server converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is digital voice data, and the output is text data. For example, the voice saying "What's the weather like today?" is converted into text "What's the weather like today?"

[0444] Step 4:

[0445] The server sends the converted text data to a generative AI model to generate an appropriate response. A generative AI model (e.g., GPT-3) is used for this. The server inputs the text data as a prompt into the generative AI model and receives the generated response. The input is the text data "What's the weather like today?" and the output is the response text "Today's weather is sunny."

[0446] Step 5:

[0447] The server sends the generated response text to the terminal. The server converts the response text into data packets (using the TCP / IP protocol) and sends them again over the network to the terminal. The input is the generated response text, and the output is the data converted into network packets.

[0448] Step 6:

[0449] The device converts the received response text data into voice data using a speech synthesis engine. Specifically, the device converts text data into voice data using the Google Cloud Text-to-Speech API. The input is the response text data, and the output is the synthesized voice data.

[0450] Step 7:

[0451] The generated voice data is transmitted to the user via bone conduction earphones. The terminal sends the synthesized voice data to the earphones, which transmit the voice to the user using bone conduction technology. The input is the synthesized voice data, and the output is the voice that is actually transmitted to the user. The user can hear the voice response, "Today's weather is sunny."

[0452] (Application example 1)

[0453] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0454] In recent years, there has been a demand for providing customers with fast and accurate information in brick-and-mortar stores. However, it is difficult for store employees to constantly respond to customers, and maintaining customer satisfaction is especially challenging during busy times and peak hours. In addition, existing digital interfaces and guidance systems are complex to operate, making them difficult to use, especially for elderly people and those unfamiliar with technology. For this reason, a new voice dialogue system is needed to efficiently guide customers in brick-and-mortar stores and improve customer convenience.

[0455] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0456] In this invention, the server includes means for capturing the user's voice, means for converting the captured voice into digital data and transmitting it, means for converting the received voice data into text data, means for transmitting the text data to a generative model to generate a response, means for receiving the generated response, converting it into voice and outputting it, and means for providing information about products, services, and sales in a physical store. This allows customers to instantly find out the location of products in the store, special sale information, etc., simply by asking a question by voice.

[0457] "User voice" refers to voice data in natural language spoken by a system user.

[0458] A "capturing means" is a device such as a microphone or bone conduction earphone for acquiring an audio signal.

[0459] "Convert to digital data" is the process of converting analog audio signals into a digital format.

[0460] "Transmitting means" is a network interface for transmitting digital data to other devices or servers.

[0461] "Means for converting to text data" refers to the process of converting voice data into text format using voice recognition technology.

[0462] A "generative model" is an artificial intelligence model that generates appropriate responses based on input text data.

[0463] A "means for generating a response" is a process that uses a generative model to create a response based on text data.

[0464] "Means for converting into voice and outputting" refers to the process of converting text data into voice data using voice synthesis technology and making it available to the user.

[0465] A "brick and mortar store" is a physical store where customers visit in person to transact goods or services.

[0466] "Means for providing product, service and sale information" is a process for providing customers with product locations in stores, service details, special sale information, etc., through the generated response.

[0467] Bone conduction earphones are devices that transmit sound directly to the inner ear through the skull, bypassing the ear.

[0468] As a specific example of a distinctive application of the present invention, we will present a specific example of a system for implementing a "store navigation assistant." This system allows users to ask questions about products, services, and sales information by voice in a physical store and receive appropriate responses.

[0469] Configuration and Operation

[0470] Hardware and software used

[0471] Audio capture device: Bone conduction earphones have a built-in microphone that captures the user's voice.

[0472] Device: The device worn by the user converts the captured audio into digital data and sends it to the server.

[0473] Server: The server converts the voice data into text using the Google Cloud Speech-to-Text API and sends this text to OpenAI's generative AI model. The generated response text is processed within the server and converted into voice data using the Google Cloud Text-to-Speech API. Finally, this voice data is sent to the device and transmitted to the user via bone conduction earphones.

[0474] Data processing and calculation

[0475] 1. Converting voice to digital data: The user's voice is converted into digital data by an encoder on the device, which is then sent to a server via Wi-Fi or a mobile network.

[0476] 2. Speech Recognition: The server converts the voice data into text data using the Google Cloud Speech-to-Text API. Here, the speech recognition engine analyzes the voice signal and converts it into text format.

[0477] 3. Response generation: A generative AI model (OpenAI's ChatGPT API) generates an appropriate response based on the text data. The system uses the generative AI model to create a response to the user's question entered as a prompt.

[0478] 4. Speech synthesis: The generated text response is converted into natural-sounding speech using the Google Cloud Text-to-Speech API. This conversion is performed by a speech synthesis engine.

[0479] 5. Audio output: The device transmits the received audio data to the user through bone conduction earphones.

[0480] Specific use cases

[0481] As a concrete example, consider a situation where a user is in a physical store and asks, "Where are the sale items?" The processing steps in this case are as follows:

[0482] 1. A user asks, "Where are the sales?" while in a brick-and-mortar store.

[0483] 2. The device captures this audio, converts it into digital data, and sends it to the server.

[0484] 3. The server converts the voice data into text data and sends it to the "generative AI model" as a prompt.

[0485] 4. The generative AI model generates the response, "The sale items are located near the center of the store."

[0486] 5. The server converts this response into audio data and sends it back to the terminal.

[0487] 6. The device outputs a response through bone conduction earphones, telling the user, "The sale items are located near the center of the store."

[0488] Prompt Sentence Examples

[0489] User Question: "Where are the sale items?"

[0490] Prompt: "Where are the sale items? They are near the center of the store."

[0491] As described above, the system for implementing this invention is designed to provide users with information about physical stores quickly and appropriately, significantly improving the efficiency of customer service and increasing customer satisfaction.

[0492] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0493] Step 1:

[0494] The user puts on the bone conduction earphones and asks a question.

[0495] Input: User's natural language speech.

[0496] How it works: A user says, "Where are the sale items?" in a physical store. The microphone inside the bone conduction earphones captures this voice.

[0497] Output: Analog audio signal.

[0498] Step 2:

[0499] The device converts the captured audio into digital data and sends it to the server.

[0500] Input: Analog audio signal.

[0501] How it works: An encoder inside the device converts analog audio signals into digital audio data and sends it to a server via Wi-Fi or a mobile network.

[0502] Output: Digital audio data.

[0503] Step 3:

[0504] The server converts the received digital voice data into text data.

[0505] Input: Digital audio data.

[0506] How it works: The server uses the Google Cloud Speech-to-Text API to convert the voice data into text data. The speech recognition engine analyzes the voice signal and converts the data into text.

[0507] Output: Text data.

[0508] Step 4:

[0509] The server sends the text data to the generative AI model, which generates a response.

[0510] Input: Text data.

[0511] How it works: The generative AI model (OpenAI's ChatGPT API) generates an appropriate response based on the text data. The server sends this prompt to the generative AI model, which then receives a response such as "The sale items are located near the center of the store."

[0512] Output: The response text.

[0513] Step 5:

[0514] The server converts the generated response text into audio data.

[0515] Input: Response text.

[0516] How it works: The server uses the Google Cloud Text-to-Speech API to convert the generated text response into audio data. A speech synthesis engine analyzes this text and processes the data to convert it into natural-sounding speech.

[0517] Output: Audio data.

[0518] Step 6:

[0519] The audio data received by the device is transmitted to the user through bone conduction earphones.

[0520] Input: Audio data.

[0521] Specific operation: The terminal plays audio data and provides audio information to the user through bone conduction earphones. The user hears the voice guidance, "The sale items are located near the center of the store."

[0522] Output: Audio information to the user.

[0523] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0524] This invention combines a speech dialogue platform that utilizes a generative model with an emotion engine, and relates to a system that realizes more natural and adaptive dialogue by recognizing the user's emotions and generating responses accordingly. The system includes a user device (including bone conduction earphones), a server that hosts the generative model, and an emotion engine.

[0525] Program processing procedure

[0526] User operations

[0527] The user wears bone conduction earphones and asks the system a question, such as "What's the weather like today?" The voice is captured by a microphone built into the earphones.

[0528] Device audio capture and transmission

[0529] The device converts the captured audio into digital data, which is then transmitted to the server via Wi-Fi or a mobile network. At this time, the transmission module operates to send the audio data to the server in the appropriate data packet format.

[0530] Audio processing on the server

[0531] The server analyzes the received voice data and converts it into text data using a speech recognition engine. It also recognizes the user's emotions using an emotion engine. The emotion engine analyzes the tone, tempo, and pitch of the voice to identify the user's emotional state.

[0532] Emotion recognition by emotion engine

[0533] The emotion engine recognizes the user's emotional state, such as "angry" or "happy," based on features extracted from the user's voice data, allowing responses to be provided in a more appropriate emotional tone.

[0534] Server response generation

[0535] The server sends the text data and the recognized emotion information to the generative model, which then generates a response based on the input data. For example, in response to the question "What's the weather like today?", the model calculates and generates the response "The weather is sunny today."

[0536] Response transmission from the server to the terminal

[0537] The generated response text is sent back from the server to the user's terminal, where it is converted into data packets and sent again over the network to the terminal.

[0538] Device speech synthesis and output

[0539] The device converts the received text data into speech using a speech synthesis engine. This speech data is transmitted to the user using bone conduction earphones. Bone conduction earphones do not block the ears, so they put less strain on the ears even when worn for long periods of time, allowing the user to receive audio information naturally.

[0540] Specific examples

[0541] When a user asks "What's the weather like today?" the following actions occur:

[0542] 1. User asks a question: The user puts on the bone conduction earphones and asks, "What's the weather like today?"

[0543] 2. The device captures the audio: The device captures the user's audio, converts it into digital data, and sends it to the server.

[0544] 3. The server converts the voice data into text: The server converts the voice data into text and recognizes emotions using the emotion engine.

[0545] 4. The emotion engine recognizes emotions: If the emotion “angry” is recognized, that information is also sent to the generative model.

[0546] 5. The generative model generates a response: The generative model generates a response with the appropriate emotional tone, such as "The weather is sunny today."

[0547] 6. The server sends the response to the terminal: The server sends the generated response text to the terminal.

[0548] 7. The device converts the response into speech: The device synthesizes speech and transmits it to the user through bone conduction earphones.

[0549] 8. User hears the response: The user hears the emotionally sensitive voice response: "The weather is sunny today."

[0550] In this way, the present invention combines a generative model and an emotion engine to realize natural and adaptive voice dialogue that responds to the user's emotions. This not only allows users to use the system comfortably for extended periods of time, but also provides a more personalized dialogue experience.

[0551] The processing flow will be explained below.

[0552] Program processing steps

[0553] Step 1: User asks a question

[0554] The user puts on the bone conduction earphones and asks, "What's the weather like today?" The sound is captured by a microphone built into the bone conduction earphones.

[0555] Step 2: Your device captures audio

[0556] The terminal captures the user's voice in real time, and the voice capture module operates to convert the voice signal into digital voice data.

[0557] Step 3: The device sends the audio data to the server

[0558] The device transmits the converted digital audio data to the server via Wi-Fi or mobile network, and the transmission module creates data packets and sends them to the server.

[0559] Step 4: The server receives the audio data

[0560] The server receives the voice data sent from the terminal, and the receiving module analyzes the complete data packet to obtain the voice data.

[0561] Step 5: The server converts the audio data into text data.

[0562] The server converts the received voice data into text data using a voice recognition engine, which analyzes the voice data and generates corresponding text.

[0563] Step 6: The server recognizes the user's emotion using the emotion engine.

[0564] The server inputs the generated text data and voice data features into an emotion engine to recognize the user's emotions. The emotion engine analyzes the tone, tempo, and pitch of the voice to identify the user's emotional state.

[0565] Step 7: The emotion engine identifies the user's emotion

[0566] The emotion engine recognizes, for example, that the user is angry. The recognized emotion information is input into the generative model.

[0567] Step 8: The server sends the text data and emotion information to the generative model

[0568] The server sends the text data and the recognized emotions to the generative model, which generates a response based on the input data.

[0569] Step 9: The generative model generates the response text

[0570] Based on the text data and emotional information sent, the generative model generates a response with an appropriate emotional tone, such as "Today's weather is sunny."

[0571] Step 10: The server sends the generated response to the terminal

[0572] The server returns the generated response text to the user terminal, and the transmission module converts the response text into data packets and sends them to the terminal.

[0573] Step 11: The device receives the response text

[0574] The terminal receives the response text sent from the server, and the receiving module analyzes the data packet to obtain the complete response text.

[0575] Step 12: The device performs speech synthesis

[0576] The device synthesizes the text data and converts it into voice data. The speech synthesis engine converts the text into synthesized voice.

[0577] Step 13: The device plays audio through the bone conduction earphones

[0578] The device plays the synthesized voice through bone conduction earphones, and the bone conduction speaker module outputs the synthesized voice in a format that is easy for the user to hear.

[0579] Step 14: User listens to response

[0580] The user hears the emotionally sensitive voice response "Today's weather is sunny" through the bone conduction earphones, allowing the user to obtain weather information.

[0581] The above is a specific processing flow of the invention that combines an emotion engine, allowing users to enjoy natural and adaptive dialogue according to their emotions.

[0582] Example 2

[0583] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] Conventional voice dialogue systems have difficulty recognizing the user's emotions and generating appropriate responses. In particular, responses that do not take the user's emotional state into account are insufficient for realizing natural and adaptive dialogue and may impair the user experience. Furthermore, prolonged use can cause significant strain on the hearing and lead to a lack of comfort.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0586] In this invention, the server includes means for receiving and analyzing voice data, means for converting the received voice data into text data, and means for analyzing features of the voice data to identify the user's emotional state, thereby enabling a natural and adaptive response according to the user's emotional state.

[0587] "Means for capturing user voice" refers to technology that captures user-generated voice using a microphone or other input device.

[0588] "Means for converting captured audio into digital data and transmitting it" refers to a technology for converting the acquired audio signal into a digital signal and transmitting it to a server via a communication network.

[0589] "Means for receiving and analyzing transmitted voice data" refers to a technology for receiving voice data transmitted via a network and analyzing that data.

[0590] The "means for converting received voice data into text data" refers to a technology that uses voice recognition technology to convert voice data into corresponding text format data.

[0591] "Means for identifying a user's emotional state by analyzing the features of audio data" refers to technology that analyzes features such as tone, tempo, and pitch extracted from audio data to estimate a user's emotional state.

[0592] "Means for sending the identified emotional state and text data to a generative model to generate an appropriate response" refers to a technology that inputs the analyzed emotional state and converted text data into a generative model to generate an appropriate response based thereon.

[0593] "Means for receiving the generated response, converting it into speech, and outputting it" refers to a technology for receiving the generated text response, converting the text into speech using a speech synthesis engine, and finally outputting it to the user.

[0594] "Bone conduction earphones" refer to acoustic devices that transmit sound to the user's inner ear through the bones without blocking the ear.

[0595] "Hearing reduction measures" refers to technologies or devices designed to reduce the user's hearing, even when used for extended periods of time.

[0596] This invention combines a speech dialogue platform that utilizes a generative model with an emotion engine, and relates to a system that realizes more natural and adaptive dialogue by recognizing the user's emotions and generating responses accordingly. The system includes a user device (including bone conduction earphones), a server that hosts the generative model, and an emotion engine.

[0597] Here, the user wears bone conduction earphones and asks the system a question, such as "What's the weather like today?" This voice is captured by a microphone built into the earphones. The device uses an analog-to-digital converter (ADC) to convert the captured voice into digital data. The converted digital voice data is then sent to the server via Wi-Fi or a mobile network. At this time, the transmission module operates and sends the voice data to the server in the appropriate data packet format.

[0598] The server analyzes the received voice data and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text). It then uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotions. The emotion engine analyzes voice features such as tone, tempo, and pitch to identify the user's emotional state (e.g., "angry" or "happy").

[0599] The emotional information and text data recognized by the emotion engine are sent to a generative model (e.g., OpenAI GPT-4). The generative model generates an appropriate response based on the input data. For example, in response to the question, "What's the weather like today?", the response generated is, "It's sunny today." This response takes into account the user's emotions.

[0600] The generated response text is sent back from the server to the user's device. The server converts the response text into data packets and sends them again over the network to the device. The device converts the received text data into speech using a speech synthesis engine (e.g., Amazon Polly). This speech data is transmitted to the user using bone conduction earphones. Because bone conduction earphones do not block the ears, they put less strain on the ears even when worn for long periods of time, allowing the user to receive audio information naturally.

[0601] As a concrete example, if a user asks "What's the weather like today?" the following actions occur:

[0602] The user asks a question: "What's the weather like today?"

[0603] The device captures the audio, converts it into digital data, and sends it to a server.

[0604] The server converts the speech into text and uses an emotion engine to recognize emotions.

[0605] The emotion engine recognizes emotions: if the emotion "angry" is recognized, it also sends that information to the generative model.

[0606] The generative model generates a response: "The weather is sunny today," with the appropriate emotional tone.

[0607] The server sends a response to the terminal.

[0608] The device converts the response into audio and transmits it to the user through bone conduction earphones.

[0609] The user hears the response: "The weather is sunny today." The user hears the emotionally sensitive voice response.

[0610] By combining a generative model and an emotion engine, the present invention makes it possible to realize natural and adaptive voice dialogue that responds to the user's emotions. This not only allows users to use the device comfortably for extended periods of time, but also allows them to enjoy a more personalized dialogue experience.

[0611] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0612] Step 1:

[0613] A user puts on bone conduction earphones and asks a question such as "What's the weather like today?" The sound is captured by a microphone built into the bone conduction earphones. The input is the user's voice, and the output is an analog audio signal.

[0614] Step 2:

[0615] The terminal converts the captured analog voice signal into digital data using an ADC (Analog-to-Digital Converter). The input is an analog voice signal, and the output is digital voice data. This data conversion process preserves the waveform information of the voice signal.

[0616] Step 3:

[0617] The terminal transmits the converted digital audio data to the server via Wi-Fi or mobile network. The input is digital audio data, and the output is audio data in the form of data packets. The transmission module uses the TCP / IP stack to divide the audio data into small packets.

[0618] Step 4:

[0619] The server recombines the received voice data in data packets and converts it into text data using a speech recognition engine. The input is voice data in data packets, and the output is text data. A commonly used cloud-based API (e.g., Google Cloud Speech-to-Text) is used as the speech recognition engine.

[0620] Step 5:

[0621] The server passes the text data to the emotion engine for analysis. The input is text data, and the output is the user's emotional information. The emotion engine analyzes features such as tone, tempo, and pitch of the voice to identify emotions such as "angry" or "happy."

[0622] Step 6:

[0623] The server sends the text data and the identified emotion information to the generative AI model. The input is the text data and emotion information, and the output is the generated response text. The generative AI model (e.g., OpenAI GPT-4) generates an appropriate response based on this prompt.

[0624] Step 7:

[0625] The server converts the generated response text into data packets and sends them to the terminal again over the network. The input is the generated response text, and the output is the response text in data packet format.

[0626] Step 8:

[0627] The device converts the received response text into speech using a speech synthesis engine (e.g., Amazon Polly). The input is the response text, and the output is voice data. The speech synthesis engine converts the text data into natural-sounding speech, and the voice data is transmitted to the user via bone conduction earphones.

[0628] Step 9:

[0629] The user listens to the generated audio through bone conduction earphones. The input is audio data, and the output is the audio perceived by the user. Bone conduction earphones allow users to receive audio information naturally without blocking their ears, and put less strain on their hearing even when used for long periods of time.

[0630] (Application example 2)

[0631] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0632] Conventional voice dialogue systems have been limited to simple question-and-answer systems without considering the user's emotions, resulting in poor dialogue quality. Furthermore, when dealing with customers in physical stores, it has been difficult to understand the customer's emotions and respond accordingly.

[0633] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing the user's voice, means for converting the captured voice into digital data and transmitting it, means for converting the received voice data into text data, means for transmitting the text data to a generative model to generate a response, means for receiving the generated response, converting it into voice and outputting it, means for analyzing and recognizing the emotional state, and means for adjusting the generated response based on the emotional state. This enables natural and adaptive dialogue according to the customer's emotions, making it possible to improve the quality of customer service in physical stores.

[0634] "Means for capturing user voice" refers to a mechanism for acquiring the voice uttered by the user as digital data.

[0635] "Means for converting captured audio into digital data and transmitting it" refers to technology that processes audio signals, converts them into a digital format, and transmits them to a server via a communication means.

[0636] "Means for converting received voice data into text data" refers to the process by which the server uses voice recognition technology to analyze the voice data and convert it into text information.

[0637] "Means of sending to a generative model to generate a response" refers to a method of inputting text data into a generative AI model to automatically generate an appropriate response.

[0638] The "means for receiving the generated response, converting it into speech, and outputting it" is a system that converts the generated text response into speech using speech synthesis technology and transmits it to the user.

[0639] The "means for analyzing and recognizing emotional states" is a technology for analyzing emotional features from the user's voice and identifying the user's current emotional state.

[0640] The "means for adjusting the generated response based on the emotional state" is a method for adjusting the content and tone of the generated response depending on the emotional state of the user recognized by the emotion engine.

[0641] "Means for outputting sound using bone conduction earphones" refers to an acoustic device that uses bone conduction to transmit sound to the user, and is a technology that transmits sound without blocking the ears.

[0642] "Means for communicating with customers via smart glasses worn by store staff" refers to a system in which store staff wearing smart glasses respond to customers' questions or requests through voice or visual information.

[0643] The present invention is a system that captures a user's voice data, analyzes it using a generative AI model and an emotion engine, generates a response, and conveys it to the user.

[0644] In this system, a user puts on the smart glasses and speaks to ask or request a question. The smart glasses' built-in microphone captures the voice and converts it into digital data. The digital data is then sent to a server via Wi-Fi or a mobile network. A module then operates to transmit the voice data in the appropriate data packet format.

[0645] The server analyzes the received voice data and converts it into text data using a voice recognition engine. For voice recognition, technologies such as "Google Speech Recognition" can be used. Next, an emotion engine analyzes features such as tone, tempo, and pitch of the voice to recognize the user's emotion. This procedure identifies the user's emotional state, such as "angry" or "happy." Emotion analysis technologies such as "IBM Watson" and "Microsoft Azure Cognitive Services" can be used for emotion recognition.

[0646] Based on the analysis results, the emotion information and text data obtained from the emotion engine are sent to a generative AI model, which can use advanced natural language processing models such as GPT-3 or T5 to generate a response with an appropriate emotional tone. The generated response is then sent back from the server to the smart glasses and displayed on their screens.

[0647] Here's a concrete example: If a customer asks, "Tell me about this product," the smart glasses will capture the voice and send the following prompt to the generative AI model:

[0648] "When the customer is calm, they say, 'Tell me about this product.' We create that response."

[0649] The response generated by the generative AI model might be something like, "This product is very popular, especially this feature." This allows store associates to respond to customers in the right emotional tone, improving customer satisfaction.

[0650] The present invention can provide a more natural and adaptive user experience through emotion-based dialogue, which can be applied not only to customer service in brick-and-mortar stores but also to other voice dialogue applications, and has a wide range of applications.

[0651] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0652] Step 1:

[0653] The user wears the smart glasses and inputs questions or requests by voice. The voice is captured by the microphone built into the smart glasses. The input data is a voice signal, and the captured voice signal is passed directly to the next step.

[0654] Step 2:

[0655] The device (smart glasses) converts the captured audio into digital data using a built-in audio-to-digital conversion module, which is then represented as audio samples and sent to a server via Wi-Fi or a mobile network.

[0656] Step 3:

[0657] The server analyzes the received digital voice data using a voice recognition engine and converts it into text data. The voice recognition engine used here is assumed to be "Google Speech Recognition." The input data is digital voice data, and the output data is recognized text data.

[0658] Step 4:

[0659] The server then uses an emotion engine to analyze the text data and its speech features to recognize the user's emotional state. Possible emotion recognition technologies used here include IBM Watson and Microsoft Azure Cognitive Services. The input data are the recognized text data and speech features, and the output data is metadata that represents the user's emotional state.

[0660] Step 5:

[0661] The server sends the emotional information and text data obtained from the emotion engine to the generative model. The generative AI model uses algorithms such as GPT-3 and T5 and receives text data and emotional states as input. The model generates an appropriate response based on this. The generated response is output in text format.

[0662] Step 6:

[0663] The server sends the generated text response back to the terminal (smart glasses). The output data is the text response, which is sent to the smart glasses via the network.

[0664] Step 7:

[0665] The device (smart glasses) converts the received text data into voice data using speech synthesis technology. A speech synthesis engine is used to create a voice response with an appropriate emotional tone. The input data is a text response, and the output data is voice data.

[0666] Step 8:

[0667] The device transmits the generated voice to the user through bone conduction earphones, allowing the user to hear the voice response without covering their ears. The output data is voice, and this is what reaches the user.

[0668] The above steps enable natural voice interaction that takes into consideration the customer's emotions. For example, if a customer asks, "Tell me about this product," the response can be provided in an appropriate tone, such as, "This product is very popular. This feature in particular is attracting attention." This process improves the quality of customer service and increases customer satisfaction.

[0669] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0670] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0671] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0672] [Third embodiment]

[0673] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0674] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0675] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0676] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0677] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0678] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0679] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0680] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0681] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0682] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0683] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0684] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0685] This invention relates to a voice dialogue platform for capturing a user's voice and interacting with a generative model, which basically consists of a device worn by the user (including bone conduction earphones) and a server that hosts the generative model. This system converts the user's voice into a digital format, generates a response using a generative AI model, and finally transmits the response to the user as voice.

[0686] Program processing procedure

[0687] User operations

[0688] The user wears bone conduction earphones and asks the system a question, such as "What's the weather like today?" The voice is captured by a microphone built into the earphones.

[0689] Device audio capture and transmission

[0690] The device converts the captured audio into digital data, which is then transmitted to the server via Wi-Fi or a mobile network. At this time, the transmission module operates to send the audio data to the server in the appropriate data packet format.

[0691] Audio processing on the server

[0692] The server analyzes the received voice data and converts it into text data using a speech recognition engine. This text data is sent to a generative AI model, which generates an appropriate response based on the text. For example, in response to the question "What's the weather like today?", the generative AI model generates the response "It's sunny today."

[0693] Response transmission from the server to the terminal

[0694] The generated response text is sent back from the server to the user's terminal, where it is converted into data packets and sent again over the network to the terminal.

[0695] Device speech synthesis and output

[0696] The device converts the received text data into speech using a speech synthesis engine. This speech data is transmitted to the user using bone conduction earphones. Bone conduction earphones do not block the ears, so they put less strain on the ears even when worn for long periods of time, allowing the user to receive audio information naturally.

[0697] Specific examples

[0698] When a user asks "What's the weather like today?" the following actions occur:

[0699] 1. User asks a question: The user puts on the bone conduction earphones and asks, "What's the weather like today?"

[0700] 2. The device captures the audio: The device captures the user's audio, converts it into digital data, and sends it to the server.

[0701] 3. Server converts speech to text: The server converts the speech data into text and sends it to the generative AI model.

[0702] 4. The generative AI model generates a response: The generative AI model generates the response, "The weather is sunny today."

[0703] 5. The server sends the response to the terminal: The server sends the generated response text to the terminal.

[0704] 6. The device converts the response into speech: The device synthesizes speech and transmits it to the user through bone conduction earphones.

[0705] 7. User hears response: The user hears the audio response "The weather is sunny today."

[0706] In this way, the present invention makes it possible to realize natural voice dialogue using a generative model, allowing users to use the system comfortably for extended periods of time.

[0707] The processing flow will be explained below.

[0708] Program processing steps

[0709] Step 1: User asks a question

[0710] The user puts on the bone conduction earphones and asks, "What's the weather like today?" The sound is captured by a microphone built into the bone conduction earphones.

[0711] Step 2: Your device captures audio

[0712] The terminal captures the user's voice in real time, and the voice capture module operates to convert the voice signal into digital voice data.

[0713] Step 3: The device sends the audio data to the server

[0714] The device transmits the converted digital audio data to the server via Wi-Fi or mobile network, and the transmission module creates data packets and sends them to the server.

[0715] Step 4: The server receives the audio data

[0716] The server receives the voice data sent from the terminal, and the receiving module analyzes the complete data packet to obtain the voice data.

[0717] Step 5: The server converts the audio data into text data.

[0718] The server converts the received voice data into text data using a voice recognition engine, which analyzes the voice data and generates corresponding text.

[0719] Step 6: The server sends the text data to the generative model

[0720] The server sends the converted text data, "What's the weather like today?", to the generative model, which analyzes the input text data and generates an appropriate response.

[0721] Step 7: The generative model generates the response text

[0722] The generative model generates the response "The weather is sunny today" based on the text data sent. The generative model calculates and generates the response text.

[0723] Step 8: The server sends the generated response to the device

[0724] The server returns the generated response text to the user terminal, and the transmission module converts the response text into data packets and sends them to the terminal.

[0725] Step 9: Your device receives the response text

[0726] The terminal receives the response text sent from the server, and the receiving module analyzes the data packet to obtain the complete response text.

[0727] Step 10: The device performs speech synthesis

[0728] The device performs speech synthesis based on the text data "Today's weather is sunny" and converts it into voice data. The speech synthesis engine converts the text into synthetic speech.

[0729] Step 11: The device plays audio through the bone conduction earphones

[0730] The device plays the synthesized voice through bone conduction earphones, and the bone conduction speaker module outputs the synthesized voice in a format that is easy for the user to hear.

[0731] Step 12: User listens to response

[0732] The user hears the voice response "Today's weather is sunny" through the bone conduction earphones, allowing the user to obtain weather information.

[0733] Example 1

[0734] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0735] Voice dialogue systems are required to be comfortable and not cause fatigue even when used for long periods of time. However, conventional systems have limitations in speech recognition accuracy and real-time response generation, which reduces user convenience. Another issue is that earphones worn for long periods of time can easily strain the hearing. Furthermore, it has been pointed out that delays are likely to occur during the voice data conversion and transmission process, making it difficult to achieve smooth dialogue.

[0736] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0737] In this invention, the server includes digital signal processing means for converting voice data into digital format, means for converting voice data into text data using a voice recognition engine, and speech synthesis means for converting the generated response from text to voice. This enables highly accurate voice recognition and response generation in real time, resulting in a system that users can use comfortably for long periods of time. In particular, the use of bone conduction earphones reduces hearing strain and allows for long-term wearing. Furthermore, the use of high-performance digital signal processing minimizes delays in data conversion, enabling smooth dialogue.

[0738] A "user" is a person who uses this system to engage in voice interaction.

[0739] The term "means for capturing voice" refers to a device or mechanism that can capture the user's voice as a signal.

[0740] "Means for converting into digital data and transmitting" refers to a mechanism or method for converting the captured analog audio signal into digital data and transmitting it to another device via a network.

[0741] "Means for converting voice data to text data" refers to techniques or methods for analyzing received voice signals and converting them into corresponding text data.

[0742] "Means for sending to a generative model to generate a response" refers to technology or a system that inputs the converted text data into a generative artificial intelligence to generate an appropriate response.

[0743] "Means for receiving a generated response, converting it into speech, and outputting it" refers to a method or device for converting a text response returned from a generative model into speech and providing it to a user.

[0744] "Digital signal processing means" refers to a technique or device for converting an analog audio signal into a digital signal.

[0745] A "speech recognition engine" is software or a system that analyzes speech signals and converts them into text data.

[0746] "Speech synthesis means" refers to technology or devices for converting text data into speech signals.

[0747] Bone conduction earphones are earphones that transmit sound through bones, allowing you to hear audio without blocking your ears.

[0748] A "data packet" is a unit of information packed into a specific format for sending and receiving data over a network.

[0749] This invention relates to a voice dialogue platform for capturing a user's voice and interacting with a generative model. Its basic configuration consists of a device worn by the user (including bone conduction earphones) and a server hosting the generative model. The system converts the user's voice into a digital format, generates a response using a generative AI model, and finally transmits the response to the user as voice.

[0750] The device includes bone conduction earphones with a built-in, highly sensitive microphone. When the user asks a question such as "What's the weather today?", the microphone captures the audio signal. This captured audio is converted into digital data using Qualcomm's digital signal processing (DSP). The converted digital audio data is then sent to a server via Wi-Fi or a mobile network (e.g., LTE). This process uses a transmission module, which sends the audio data to the server in the form of UDP packets.

[0751] The server analyzes the received voice data and converts it into text data using the Google Cloud Speech-to-Text API. The converted text data is sent as a prompt to a generative AI model such as GPT-3. For example, in response to the question "What's the weather like today?", the generative AI model generates the response "It's sunny today."

[0752] The generated response text is sent from the server to the device in TCP / IP packet format. The device converts the received text data into voice data using the Google Cloud Text-to-Speech API. The generated voice data is transmitted to the user using bone conduction earphones. These earphones do not block the ears, so they put less strain on the hearing and can be used comfortably for long periods of time.

[0753] As a concrete example of this behavior, when a user asks "What's the weather like today?":

[0754] 1. User asks a question: The user puts on the bone conduction earphones and asks, "What's the weather like today?"

[0755] 2. The device captures the voice: The device captures the user's voice, converts it into digital data using a Qualcomm DSP, and sends it to the server.

[0756] 3. The server converts the speech to text: The server uses the Google Cloud Speech-to-Text API to convert the speech data to text and sends this text to a generative AI model (e.g., GPT-3).

[0757] 4. The generative AI model generates a response: The generative AI model generates the response "The weather is sunny today."

[0758] 5. The server sends the response to the terminal: The server sends the generated response text to the terminal as a TCP / IP packet.

[0759] 6. The device converts the response into speech and outputs it: The device uses the Google Cloud Text-to-Speech API to convert the text into speech and transmits it to the user through bone conduction earphones.

[0760] 7. User hears response: The user hears the audio response "The weather is sunny today."

[0761] In this way, the present invention makes it possible to realize natural voice dialogue using a generative model, allowing users to use the system comfortably for extended periods of time.

[0762] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0763] Step 1:

[0764] The user wears bone conduction earphones and asks the system a question such as "What's the weather like today?" This voice is captured by a microphone built into the earphones. The input is the user's voice, and the output is the captured analog voice signal. At this stage, the user's voice is acquired in real time.

[0765] Step 2:

[0766] The device converts the captured analog voice signal into digital data using digital signal processing (DSP). Specifically, the DSP built into the device samples the voice signal and converts the analog voice into digital data. The input is an analog voice signal, and the output is digital voice data. This digital data is sent to a server via Wi-Fi or a mobile network.

[0767] Step 3:

[0768] The server analyzes the received digital voice data and converts it into text data using a voice recognition engine. Specifically, the server converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is digital voice data, and the output is text data. For example, the voice saying "What's the weather like today?" is converted into text "What's the weather like today?"

[0769] Step 4:

[0770] The server sends the converted text data to a generative AI model to generate an appropriate response. A generative AI model (e.g., GPT-3) is used for this. The server inputs the text data as a prompt into the generative AI model and receives the generated response. The input is the text data "What's the weather like today?" and the output is the response text "Today's weather is sunny."

[0771] Step 5:

[0772] The server sends the generated response text to the terminal. The server converts the response text into data packets (using the TCP / IP protocol) and sends them again over the network to the terminal. The input is the generated response text, and the output is the data converted into network packets.

[0773] Step 6:

[0774] The device converts the received response text data into voice data using a speech synthesis engine. Specifically, the device converts text data into voice data using the Google Cloud Text-to-Speech API. The input is the response text data, and the output is the synthesized voice data.

[0775] Step 7:

[0776] The generated voice data is transmitted to the user via bone conduction earphones. The terminal sends the synthesized voice data to the earphones, which transmit the voice to the user using bone conduction technology. The input is the synthesized voice data, and the output is the voice that is actually transmitted to the user. The user can hear the voice response, "Today's weather is sunny."

[0777] (Application example 1)

[0778] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0779] In recent years, there has been a demand for providing customers with fast and accurate information in brick-and-mortar stores. However, it is difficult for store employees to constantly respond to customers, and maintaining customer satisfaction is especially challenging during busy times and peak hours. In addition, existing digital interfaces and guidance systems are complex to operate, making them difficult to use, especially for elderly people and those unfamiliar with technology. For this reason, a new voice dialogue system is needed to efficiently guide customers in brick-and-mortar stores and improve customer convenience.

[0780] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0781] In this invention, the server includes means for capturing the user's voice, means for converting the captured voice into digital data and transmitting it, means for converting the received voice data into text data, means for transmitting the text data to a generative model to generate a response, means for receiving the generated response, converting it into voice and outputting it, and means for providing information about products, services, and sales in a physical store. This allows customers to instantly find out the location of products in the store, special sale information, etc., simply by asking a question by voice.

[0782] "User voice" refers to voice data in natural language spoken by a system user.

[0783] A "capturing means" is a device such as a microphone or bone conduction earphone for acquiring an audio signal.

[0784] "Convert to digital data" is the process of converting analog audio signals into a digital format.

[0785] "Transmitting means" is a network interface for transmitting digital data to other devices or servers.

[0786] "Means for converting to text data" refers to the process of converting voice data into text format using voice recognition technology.

[0787] A "generative model" is an artificial intelligence model that generates appropriate responses based on input text data.

[0788] A "means for generating a response" is a process that uses a generative model to create a response based on text data.

[0789] "Means for converting into voice and outputting" refers to the process of converting text data into voice data using voice synthesis technology and making it available to the user.

[0790] A "brick and mortar store" is a physical store where customers visit in person to transact goods or services.

[0791] "Means for providing product, service and sale information" is a process for providing customers with product locations in stores, service details, special sale information, etc., through the generated response.

[0792] Bone conduction earphones are devices that transmit sound directly to the inner ear through the skull, bypassing the ear.

[0793] As a specific example of a distinctive application of the present invention, we will present a specific example of a system for implementing a "store navigation assistant." This system allows users to ask questions about products, services, and sales information by voice in a physical store and receive appropriate responses.

[0794] Configuration and Operation

[0795] Hardware and software used

[0796] Audio capture device: Bone conduction earphones have a built-in microphone that captures the user's voice.

[0797] Device: The device worn by the user converts the captured audio into digital data and sends it to the server.

[0798] Server: The server converts the voice data into text using the Google Cloud Speech-to-Text API and sends this text to OpenAI's generative AI model. The generated response text is processed within the server and converted into voice data using the Google Cloud Text-to-Speech API. Finally, this voice data is sent to the device and transmitted to the user via bone conduction earphones.

[0799] Data processing and calculation

[0800] 1. Converting voice to digital data: The user's voice is converted into digital data by an encoder on the device, which is then sent to a server via Wi-Fi or a mobile network.

[0801] 2. Speech Recognition: The server converts the voice data into text data using the Google Cloud Speech-to-Text API. Here, the speech recognition engine analyzes the voice signal and converts it into text format.

[0802] 3. Response generation: A generative AI model (OpenAI's ChatGPT API) generates an appropriate response based on the text data. The system uses the generative AI model to create a response to the user's question entered as a prompt.

[0803] 4. Speech synthesis: The generated text response is converted into natural-sounding speech using the Google Cloud Text-to-Speech API. This conversion is performed by a speech synthesis engine.

[0804] 5. Audio output: The device transmits the received audio data to the user through bone conduction earphones.

[0805] Specific use cases

[0806] As a concrete example, consider a situation where a user is in a physical store and asks, "Where are the sale items?" The processing steps in this case are as follows:

[0807] 1. A user asks, "Where are the sales?" while in a brick-and-mortar store.

[0808] 2. The device captures this audio, converts it into digital data, and sends it to the server.

[0809] 3. The server converts the voice data into text data and sends it to the "generative AI model" as a prompt.

[0810] 4. The generative AI model generates the response, "The sale items are located near the center of the store."

[0811] 5. The server converts this response into audio data and sends it back to the terminal.

[0812] 6. The device outputs a response through bone conduction earphones, telling the user, "The sale items are located near the center of the store."

[0813] Prompt Sentence Examples

[0814] User Question: "Where are the sale items?"

[0815] Prompt: "Where are the sale items? They are near the center of the store."

[0816] As described above, the system for implementing this invention is designed to provide users with information about physical stores quickly and appropriately, significantly improving the efficiency of customer service and increasing customer satisfaction.

[0817] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0818] Step 1:

[0819] The user puts on the bone conduction earphones and asks a question.

[0820] Input: User's natural language speech.

[0821] How it works: A user says, "Where are the sale items?" in a physical store. The microphone inside the bone conduction earphones captures this voice.

[0822] Output: Analog audio signal.

[0823] Step 2:

[0824] The device converts the captured audio into digital data and sends it to the server.

[0825] Input: Analog audio signal.

[0826] How it works: An encoder inside the device converts analog audio signals into digital audio data and sends it to a server via Wi-Fi or a mobile network.

[0827] Output: Digital audio data.

[0828] Step 3:

[0829] The server converts the received digital voice data into text data.

[0830] Input: Digital audio data.

[0831] How it works: The server uses the Google Cloud Speech-to-Text API to convert the voice data into text data. The speech recognition engine analyzes the voice signal and converts the data into text.

[0832] Output: Text data.

[0833] Step 4:

[0834] The server sends the text data to the generative AI model, which generates a response.

[0835] Input: Text data.

[0836] How it works: The generative AI model (OpenAI's ChatGPT API) generates an appropriate response based on the text data. The server sends this prompt to the generative AI model, which then receives a response such as "The sale items are located near the center of the store."

[0837] Output: The response text.

[0838] Step 5:

[0839] The server converts the generated response text into audio data.

[0840] Input: Response text.

[0841] How it works: The server uses the Google Cloud Text-to-Speech API to convert the generated text response into audio data. A speech synthesis engine analyzes this text and processes the data to convert it into natural-sounding speech.

[0842] Output: Audio data.

[0843] Step 6:

[0844] The audio data received by the device is transmitted to the user through bone conduction earphones.

[0845] Input: Audio data.

[0846] Specific operation: The terminal plays audio data and provides audio information to the user through bone conduction earphones. The user hears the voice guidance, "The sale items are located near the center of the store."

[0847] Output: Audio information to the user.

[0848] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0849] This invention combines a speech dialogue platform that utilizes a generative model with an emotion engine, and relates to a system that realizes more natural and adaptive dialogue by recognizing the user's emotions and generating responses accordingly. The system includes a user device (including bone conduction earphones), a server that hosts the generative model, and an emotion engine.

[0850] Program processing procedure

[0851] User operations

[0852] The user wears bone conduction earphones and asks the system a question, such as "What's the weather like today?" The voice is captured by a microphone built into the earphones.

[0853] Device audio capture and transmission

[0854] The device converts the captured audio into digital data, which is then transmitted to the server via Wi-Fi or a mobile network. At this time, the transmission module operates to send the audio data to the server in the appropriate data packet format.

[0855] Audio processing on the server

[0856] The server analyzes the received voice data and converts it into text data using a speech recognition engine. It also recognizes the user's emotions using an emotion engine. The emotion engine analyzes the tone, tempo, and pitch of the voice to identify the user's emotional state.

[0857] Emotion recognition by emotion engine

[0858] The emotion engine recognizes the user's emotional state, such as "angry" or "happy," based on features extracted from the user's voice data, allowing responses to be provided in a more appropriate emotional tone.

[0859] Server response generation

[0860] The server sends the text data and the recognized emotion information to the generative model, which then generates a response based on the input data. For example, in response to the question "What's the weather like today?", the model calculates and generates the response "The weather is sunny today."

[0861] Response transmission from the server to the terminal

[0862] The generated response text is sent back from the server to the user's terminal, where it is converted into data packets and sent again over the network to the terminal.

[0863] Device speech synthesis and output

[0864] The device converts the received text data into speech using a speech synthesis engine. This speech data is transmitted to the user using bone conduction earphones. Bone conduction earphones do not block the ears, so they put less strain on the ears even when worn for long periods of time, allowing the user to receive audio information naturally.

[0865] Specific examples

[0866] When a user asks "What's the weather like today?" the following actions occur:

[0867] 1. User asks a question: The user puts on the bone conduction earphones and asks, "What's the weather like today?"

[0868] 2. The device captures the audio: The device captures the user's audio, converts it into digital data, and sends it to the server.

[0869] 3. The server converts the voice data into text: The server converts the voice data into text and recognizes emotions using the emotion engine.

[0870] 4. The emotion engine recognizes emotions: If the emotion “angry” is recognized, that information is also sent to the generative model.

[0871] 5. The generative model generates a response: The generative model generates a response with the appropriate emotional tone, such as "The weather is sunny today."

[0872] 6. The server sends the response to the terminal: The server sends the generated response text to the terminal.

[0873] 7. The device converts the response into speech: The device synthesizes speech and transmits it to the user through bone conduction earphones.

[0874] 8. User hears the response: The user hears the emotionally sensitive voice response: "The weather is sunny today."

[0875] In this way, the present invention combines a generative model and an emotion engine to realize natural and adaptive voice dialogue that responds to the user's emotions. This not only allows users to use the system comfortably for extended periods of time, but also provides a more personalized dialogue experience.

[0876] The processing flow will be explained below.

[0877] Program processing steps

[0878] Step 1: User asks a question

[0879] The user puts on the bone conduction earphones and asks, "What's the weather like today?" The sound is captured by a microphone built into the bone conduction earphones.

[0880] Step 2: Your device captures audio

[0881] The terminal captures the user's voice in real time, and the voice capture module operates to convert the voice signal into digital voice data.

[0882] Step 3: The device sends the audio data to the server

[0883] The device transmits the converted digital audio data to the server via Wi-Fi or mobile network, and the transmission module creates data packets and sends them to the server.

[0884] Step 4: The server receives the audio data

[0885] The server receives the voice data sent from the terminal, and the receiving module analyzes the complete data packet to obtain the voice data.

[0886] Step 5: The server converts the audio data into text data.

[0887] The server converts the received voice data into text data using a voice recognition engine, which analyzes the voice data and generates corresponding text.

[0888] Step 6: The server recognizes the user's emotion using the emotion engine.

[0889] The server inputs the generated text data and voice data features into an emotion engine to recognize the user's emotions. The emotion engine analyzes the tone, tempo, and pitch of the voice to identify the user's emotional state.

[0890] Step 7: The emotion engine identifies the user's emotion

[0891] The emotion engine recognizes, for example, that the user is angry. The recognized emotion information is input into the generative model.

[0892] Step 8: The server sends the text data and emotion information to the generative model

[0893] The server sends the text data and the recognized emotions to the generative model, which generates a response based on the input data.

[0894] Step 9: The generative model generates the response text

[0895] Based on the text data and emotional information sent, the generative model generates a response with an appropriate emotional tone, such as "Today's weather is sunny."

[0896] Step 10: The server sends the generated response to the terminal

[0897] The server returns the generated response text to the user terminal, and the transmission module converts the response text into data packets and sends them to the terminal.

[0898] Step 11: The device receives the response text

[0899] The terminal receives the response text sent from the server, and the receiving module analyzes the data packet to obtain the complete response text.

[0900] Step 12: The device performs speech synthesis

[0901] The device synthesizes the text data and converts it into voice data. The speech synthesis engine converts the text into synthesized voice.

[0902] Step 13: The device plays audio through the bone conduction earphones

[0903] The device plays the synthesized voice through bone conduction earphones, and the bone conduction speaker module outputs the synthesized voice in a format that is easy for the user to hear.

[0904] Step 14: User listens to response

[0905] The user hears the emotionally sensitive voice response "Today's weather is sunny" through the bone conduction earphones, allowing the user to obtain weather information.

[0906] The above is a specific processing flow of the invention that combines an emotion engine, allowing users to enjoy natural and adaptive dialogue according to their emotions.

[0907] Example 2

[0908] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0909] Conventional voice dialogue systems have difficulty recognizing the user's emotions and generating appropriate responses. In particular, responses that do not take the user's emotional state into account are insufficient for realizing natural and adaptive dialogue and may impair the user experience. Furthermore, prolonged use can cause significant strain on the hearing and lead to a lack of comfort.

[0910] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0911] In this invention, the server includes means for receiving and analyzing voice data, means for converting the received voice data into text data, and means for analyzing features of the voice data to identify the user's emotional state, thereby enabling a natural and adaptive response according to the user's emotional state.

[0912] "Means for capturing user voice" refers to technology that captures user-generated voice using a microphone or other input device.

[0913] "Means for converting captured audio into digital data and transmitting it" refers to a technology for converting the acquired audio signal into a digital signal and transmitting it to a server via a communication network.

[0914] "Means for receiving and analyzing transmitted voice data" refers to a technology for receiving voice data transmitted via a network and analyzing that data.

[0915] The "means for converting received voice data into text data" refers to a technology that uses voice recognition technology to convert voice data into corresponding text format data.

[0916] "Means for identifying a user's emotional state by analyzing the features of audio data" refers to technology that analyzes features such as tone, tempo, and pitch extracted from audio data to estimate a user's emotional state.

[0917] "Means for sending the identified emotional state and text data to a generative model to generate an appropriate response" refers to a technology that inputs the analyzed emotional state and converted text data into a generative model to generate an appropriate response based thereon.

[0918] "Means for receiving the generated response, converting it into speech, and outputting it" refers to a technology for receiving the generated text response, converting the text into speech using a speech synthesis engine, and finally outputting it to the user.

[0919] "Bone conduction earphones" refer to acoustic devices that transmit sound to the user's inner ear through the bones without blocking the ear.

[0920] "Hearing reduction measures" refers to technologies or devices designed to reduce the user's hearing, even when used for extended periods of time.

[0921] This invention combines a speech dialogue platform that utilizes a generative model with an emotion engine, and relates to a system that realizes more natural and adaptive dialogue by recognizing the user's emotions and generating responses accordingly. The system includes a user device (including bone conduction earphones), a server that hosts the generative model, and an emotion engine.

[0922] Here, the user wears bone conduction earphones and asks the system a question, such as "What's the weather like today?" This voice is captured by a microphone built into the earphones. The device uses an analog-to-digital converter (ADC) to convert the captured voice into digital data. The converted digital voice data is then sent to the server via Wi-Fi or a mobile network. At this time, the transmission module operates and sends the voice data to the server in the appropriate data packet format.

[0923] The server analyzes the received voice data and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text). It then uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotions. The emotion engine analyzes voice features such as tone, tempo, and pitch to identify the user's emotional state (e.g., "angry" or "happy").

[0924] The emotional information and text data recognized by the emotion engine are sent to a generative model (e.g., OpenAI GPT-4). The generative model generates an appropriate response based on the input data. For example, in response to the question, "What's the weather like today?", the response generated is, "It's sunny today." This response takes into account the user's emotions.

[0925] The generated response text is sent back from the server to the user's device. The server converts the response text into data packets and sends them again over the network to the device. The device converts the received text data into speech using a speech synthesis engine (e.g., Amazon Polly). This speech data is transmitted to the user using bone conduction earphones. Because bone conduction earphones do not block the ears, they put less strain on the ears even when worn for long periods of time, allowing the user to receive audio information naturally.

[0926] As a concrete example, if a user asks "What's the weather like today?" the following actions occur:

[0927] The user asks a question: "What's the weather like today?"

[0928] The device captures the audio, converts it into digital data, and sends it to a server.

[0929] The server converts the speech into text and uses an emotion engine to recognize emotions.

[0930] The emotion engine recognizes emotions: if the emotion "angry" is recognized, it also sends that information to the generative model.

[0931] The generative model generates a response: "The weather is sunny today," with the appropriate emotional tone.

[0932] The server sends a response to the terminal.

[0933] The device converts the response into audio and transmits it to the user through bone conduction earphones.

[0934] The user hears the response: "The weather is sunny today." The user hears the emotionally sensitive voice response.

[0935] By combining a generative model and an emotion engine, the present invention makes it possible to realize natural and adaptive voice dialogue that responds to the user's emotions. This not only allows users to use the device comfortably for extended periods of time, but also allows them to enjoy a more personalized dialogue experience.

[0936] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0937] Step 1:

[0938] A user puts on bone conduction earphones and asks a question such as "What's the weather like today?" The sound is captured by a microphone built into the bone conduction earphones. The input is the user's voice, and the output is an analog audio signal.

[0939] Step 2:

[0940] The terminal converts the captured analog voice signal into digital data using an ADC (Analog-to-Digital Converter). The input is an analog voice signal, and the output is digital voice data. This data conversion process preserves the waveform information of the voice signal.

[0941] Step 3:

[0942] The terminal transmits the converted digital audio data to the server via Wi-Fi or mobile network. The input is digital audio data, and the output is audio data in the form of data packets. The transmission module uses the TCP / IP stack to divide the audio data into small packets.

[0943] Step 4:

[0944] The server recombines the received voice data in data packets and converts it into text data using a speech recognition engine. The input is voice data in data packets, and the output is text data. A commonly used cloud-based API (e.g., Google Cloud Speech-to-Text) is used as the speech recognition engine.

[0945] Step 5:

[0946] The server passes the text data to the emotion engine for analysis. The input is text data, and the output is the user's emotional information. The emotion engine analyzes features such as tone, tempo, and pitch of the voice to identify emotions such as "angry" or "happy."

[0947] Step 6:

[0948] The server sends the text data and the identified emotion information to the generative AI model. The input is the text data and emotion information, and the output is the generated response text. The generative AI model (e.g., OpenAI GPT-4) generates an appropriate response based on this prompt.

[0949] Step 7:

[0950] The server converts the generated response text into data packets and sends them to the terminal again over the network. The input is the generated response text, and the output is the response text in data packet format.

[0951] Step 8:

[0952] The device converts the received response text into speech using a speech synthesis engine (e.g., Amazon Polly). The input is the response text, and the output is voice data. The speech synthesis engine converts the text data into natural-sounding speech, and the voice data is transmitted to the user via bone conduction earphones.

[0953] Step 9:

[0954] The user listens to the generated audio through bone conduction earphones. The input is audio data, and the output is the audio perceived by the user. Bone conduction earphones allow users to receive audio information naturally without blocking their ears, and put less strain on their hearing even when used for long periods of time.

[0955] (Application example 2)

[0956] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0957] Conventional voice dialogue systems have been limited to simple question-and-answer systems without considering the user's emotions, resulting in poor dialogue quality. Furthermore, when dealing with customers in physical stores, it has been difficult to understand the customer's emotions and respond accordingly.

[0958] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing the user's voice, means for converting the captured voice into digital data and transmitting it, means for converting the received voice data into text data, means for transmitting the text data to a generative model to generate a response, means for receiving the generated response, converting it into voice and outputting it, means for analyzing and recognizing the emotional state, and means for adjusting the generated response based on the emotional state. This enables natural and adaptive dialogue according to the customer's emotions, making it possible to improve the quality of customer service in physical stores.

[0959] "Means for capturing user voice" refers to a mechanism for acquiring the voice uttered by the user as digital data.

[0960] "Means for converting captured audio into digital data and transmitting it" refers to technology that processes audio signals, converts them into a digital format, and transmits them to a server via a communication means.

[0961] "Means for converting received voice data into text data" refers to the process by which the server uses voice recognition technology to analyze the voice data and convert it into text information.

[0962] "Means of sending to a generative model to generate a response" refers to a method of inputting text data into a generative AI model to automatically generate an appropriate response.

[0963] The "means for receiving the generated response, converting it into speech, and outputting it" is a system that converts the generated text response into speech using speech synthesis technology and transmits it to the user.

[0964] The "means for analyzing and recognizing emotional states" is a technology for analyzing emotional features from the user's voice and identifying the user's current emotional state.

[0965] The "means for adjusting the generated response based on the emotional state" is a method for adjusting the content and tone of the generated response depending on the emotional state of the user recognized by the emotion engine.

[0966] "Means for outputting sound using bone conduction earphones" refers to an acoustic device that uses bone conduction to transmit sound to the user, and is a technology that transmits sound without blocking the ears.

[0967] "Means for communicating with customers via smart glasses worn by store staff" refers to a system in which store staff wearing smart glasses respond to customers' questions or requests through voice or visual information.

[0968] The present invention is a system that captures a user's voice data, analyzes it using a generative AI model and an emotion engine, generates a response, and conveys it to the user.

[0969] In this system, a user puts on the smart glasses and speaks to ask or request a question. The smart glasses' built-in microphone captures the voice and converts it into digital data. The digital data is then sent to a server via Wi-Fi or a mobile network. A module then operates to transmit the voice data in the appropriate data packet format.

[0970] The server analyzes the received voice data and converts it into text data using a voice recognition engine. For voice recognition, technologies such as "Google Speech Recognition" can be used. Next, an emotion engine analyzes features such as tone, tempo, and pitch of the voice to recognize the user's emotion. This procedure identifies the user's emotional state, such as "angry" or "happy." Emotion analysis technologies such as "IBM Watson" and "Microsoft Azure Cognitive Services" can be used for emotion recognition.

[0971] Based on the analysis results, the emotion information and text data obtained from the emotion engine are sent to a generative AI model, which can use advanced natural language processing models such as GPT-3 or T5 to generate a response with an appropriate emotional tone. The generated response is then sent back from the server to the smart glasses and displayed on their screens.

[0972] Here's a concrete example: If a customer asks, "Tell me about this product," the smart glasses will capture the voice and send the following prompt to the generative AI model:

[0973] "When the customer is calm, they say, 'Tell me about this product.' We create that response."

[0974] The response generated by the generative AI model might be something like, "This product is very popular, especially this feature." This allows store associates to respond to customers in the right emotional tone, improving customer satisfaction.

[0975] The present invention can provide a more natural and adaptive user experience through emotion-based dialogue, which can be applied not only to customer service in brick-and-mortar stores but also to other voice dialogue applications, and has a wide range of applications.

[0976] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0977] Step 1:

[0978] The user wears the smart glasses and inputs questions or requests by voice. The voice is captured by the microphone built into the smart glasses. The input data is a voice signal, and the captured voice signal is passed directly to the next step.

[0979] Step 2:

[0980] The device (smart glasses) converts the captured audio into digital data using a built-in audio-to-digital conversion module, which is then represented as audio samples and sent to a server via Wi-Fi or a mobile network.

[0981] Step 3:

[0982] The server analyzes the received digital voice data using a voice recognition engine and converts it into text data. The voice recognition engine used here is assumed to be "Google Speech Recognition." The input data is digital voice data, and the output data is recognized text data.

[0983] Step 4:

[0984] The server then uses an emotion engine to analyze the text data and its speech features to recognize the user's emotional state. Possible emotion recognition technologies used here include IBM Watson and Microsoft Azure Cognitive Services. The input data are the recognized text data and speech features, and the output data is metadata that represents the user's emotional state.

[0985] Step 5:

[0986] The server sends the emotional information and text data obtained from the emotion engine to the generative model. The generative AI model uses algorithms such as GPT-3 and T5 and receives text data and emotional states as input. The model generates an appropriate response based on this. The generated response is output in text format.

[0987] Step 6:

[0988] The server sends the generated text response back to the terminal (smart glasses). The output data is the text response, which is sent to the smart glasses via the network.

[0989] Step 7:

[0990] The device (smart glasses) converts the received text data into voice data using speech synthesis technology. A speech synthesis engine is used to create a voice response with an appropriate emotional tone. The input data is a text response, and the output data is voice data.

[0991] Step 8:

[0992] The device transmits the generated voice to the user through bone conduction earphones, allowing the user to hear the voice response without covering their ears. The output data is voice, and this is what reaches the user.

[0993] The above steps enable natural voice interaction that takes into consideration the customer's emotions. For example, if a customer asks, "Tell me about this product," the response can be provided in an appropriate tone, such as, "This product is very popular. This feature in particular is attracting attention." This process improves the quality of customer service and increases customer satisfaction.

[0994] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0995] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0996] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0997] [Fourth embodiment]

[0998] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0999] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1000] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1001] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1002] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1003] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1004] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1005] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1006] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1007] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1008] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1009] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1010] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1011] This invention relates to a voice dialogue platform for capturing a user's voice and interacting with a generative model, which basically consists of a device worn by the user (including bone conduction earphones) and a server that hosts the generative model. This system converts the user's voice into a digital format, generates a response using a generative AI model, and finally transmits the response to the user as voice.

[1012] Program processing procedure

[1013] User operations

[1014] The user wears bone conduction earphones and asks the system a question, such as "What's the weather like today?" The voice is captured by a microphone built into the earphones.

[1015] Device audio capture and transmission

[1016] The device converts the captured audio into digital data, which is then transmitted to the server via Wi-Fi or a mobile network. At this time, the transmission module operates to send the audio data to the server in the appropriate data packet format.

[1017] Audio processing on the server

[1018] The server analyzes the received voice data and converts it into text data using a speech recognition engine. This text data is sent to a generative AI model, which generates an appropriate response based on the text. For example, in response to the question "What's the weather like today?", the generative AI model generates the response "It's sunny today."

[1019] Response transmission from the server to the terminal

[1020] The generated response text is sent back from the server to the user's terminal, where it is converted into data packets and sent again over the network to the terminal.

[1021] Device speech synthesis and output

[1022] The device converts the received text data into speech using a speech synthesis engine. This speech data is transmitted to the user using bone conduction earphones. Bone conduction earphones do not block the ears, so they put less strain on the ears even when worn for long periods of time, allowing the user to receive audio information naturally.

[1023] Specific examples

[1024] When a user asks "What's the weather like today?" the following actions occur:

[1025] 1. User asks a question: The user puts on the bone conduction earphones and asks, "What's the weather like today?"

[1026] 2. The device captures the audio: The device captures the user's audio, converts it into digital data, and sends it to the server.

[1027] 3. Server converts speech to text: The server converts the speech data into text and sends it to the generative AI model.

[1028] 4. The generative AI model generates a response: The generative AI model generates the response, "The weather is sunny today."

[1029] 5. The server sends the response to the terminal: The server sends the generated response text to the terminal.

[1030] 6. The device converts the response into speech: The device synthesizes speech and transmits it to the user through bone conduction earphones.

[1031] 7. User hears response: The user hears the audio response "The weather is sunny today."

[1032] In this way, the present invention makes it possible to realize natural voice dialogue using a generative model, allowing users to use the system comfortably for extended periods of time.

[1033] The processing flow will be explained below.

[1034] Program processing steps

[1035] Step 1: User asks a question

[1036] The user puts on the bone conduction earphones and asks, "What's the weather like today?" The sound is captured by a microphone built into the bone conduction earphones.

[1037] Step 2: Your device captures audio

[1038] The terminal captures the user's voice in real time, and the voice capture module operates to convert the voice signal into digital voice data.

[1039] Step 3: The device sends the audio data to the server

[1040] The device transmits the converted digital audio data to the server via Wi-Fi or mobile network, and the transmission module creates data packets and sends them to the server.

[1041] Step 4: The server receives the audio data

[1042] The server receives the voice data sent from the terminal, and the receiving module analyzes the complete data packet to obtain the voice data.

[1043] Step 5: The server converts the audio data into text data.

[1044] The server converts the received voice data into text data using a voice recognition engine, which analyzes the voice data and generates corresponding text.

[1045] Step 6: The server sends the text data to the generative model

[1046] The server sends the converted text data, "What's the weather like today?", to the generative model, which analyzes the input text data and generates an appropriate response.

[1047] Step 7: The generative model generates the response text

[1048] The generative model generates the response "The weather is sunny today" based on the text data sent. The generative model calculates and generates the response text.

[1049] Step 8: The server sends the generated response to the device

[1050] The server returns the generated response text to the user terminal, and the transmission module converts the response text into data packets and sends them to the terminal.

[1051] Step 9: Your device receives the response text

[1052] The terminal receives the response text sent from the server, and the receiving module analyzes the data packet to obtain the complete response text.

[1053] Step 10: The device performs speech synthesis

[1054] The device performs speech synthesis based on the text data "Today's weather is sunny" and converts it into voice data. The speech synthesis engine converts the text into synthetic speech.

[1055] Step 11: The device plays audio through the bone conduction earphones

[1056] The device plays the synthesized voice through bone conduction earphones, and the bone conduction speaker module outputs the synthesized voice in a format that is easy for the user to hear.

[1057] Step 12: User listens to response

[1058] The user hears the voice response "Today's weather is sunny" through the bone conduction earphones, allowing the user to obtain weather information.

[1059] Example 1

[1060] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1061] Voice dialogue systems are required to be comfortable and not cause fatigue even when used for long periods of time. However, conventional systems have limitations in speech recognition accuracy and real-time response generation, which reduces user convenience. Another issue is that earphones worn for long periods of time can easily strain the hearing. Furthermore, it has been pointed out that delays are likely to occur during the voice data conversion and transmission process, making it difficult to achieve smooth dialogue.

[1062] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1063] In this invention, the server includes digital signal processing means for converting voice data into digital format, means for converting voice data into text data using a voice recognition engine, and speech synthesis means for converting the generated response from text to voice. This enables highly accurate voice recognition and response generation in real time, resulting in a system that users can use comfortably for long periods of time. In particular, the use of bone conduction earphones reduces hearing strain and allows for long-term wearing. Furthermore, the use of high-performance digital signal processing minimizes delays in data conversion, enabling smooth dialogue.

[1064] A "user" is a person who uses this system to engage in voice interaction.

[1065] The term "means for capturing voice" refers to a device or mechanism that can capture the user's voice as a signal.

[1066] "Means for converting into digital data and transmitting" refers to a mechanism or method for converting the captured analog audio signal into digital data and transmitting it to another device via a network.

[1067] "Means for converting voice data to text data" refers to techniques or methods for analyzing received voice signals and converting them into corresponding text data.

[1068] "Means for sending to a generative model to generate a response" refers to technology or a system that inputs the converted text data into a generative artificial intelligence to generate an appropriate response.

[1069] "Means for receiving a generated response, converting it into speech, and outputting it" refers to a method or device for converting a text response returned from a generative model into speech and providing it to a user.

[1070] "Digital signal processing means" refers to a technique or device for converting an analog audio signal into a digital signal.

[1071] A "speech recognition engine" is software or a system that analyzes speech signals and converts them into text data.

[1072] "Speech synthesis means" refers to technology or devices for converting text data into speech signals.

[1073] Bone conduction earphones are earphones that transmit sound through bones, allowing you to hear audio without blocking your ears.

[1074] A "data packet" is a unit of information packed into a specific format for sending and receiving data over a network.

[1075] This invention relates to a voice dialogue platform for capturing a user's voice and interacting with a generative model. Its basic configuration consists of a device worn by the user (including bone conduction earphones) and a server hosting the generative model. The system converts the user's voice into a digital format, generates a response using a generative AI model, and finally transmits the response to the user as voice.

[1076] The device includes bone conduction earphones with a built-in, highly sensitive microphone. When the user asks a question such as "What's the weather today?", the microphone captures the audio signal. This captured audio is converted into digital data using Qualcomm's digital signal processing (DSP). The converted digital audio data is then sent to a server via Wi-Fi or a mobile network (e.g., LTE). This process uses a transmission module, which sends the audio data to the server in the form of UDP packets.

[1077] The server analyzes the received voice data and converts it into text data using the Google Cloud Speech-to-Text API. The converted text data is sent as a prompt to a generative AI model such as GPT-3. For example, in response to the question "What's the weather like today?", the generative AI model generates the response "It's sunny today."

[1078] The generated response text is sent from the server to the device in TCP / IP packet format. The device converts the received text data into voice data using the Google Cloud Text-to-Speech API. The generated voice data is transmitted to the user using bone conduction earphones. These earphones do not block the ears, so they put less strain on the hearing and can be used comfortably for long periods of time.

[1079] As a concrete example of this behavior, when a user asks "What's the weather like today?":

[1080] 1. User asks a question: The user puts on the bone conduction earphones and asks, "What's the weather like today?"

[1081] 2. The device captures the voice: The device captures the user's voice, converts it into digital data using a Qualcomm DSP, and sends it to the server.

[1082] 3. The server converts the speech to text: The server uses the Google Cloud Speech-to-Text API to convert the speech data to text and sends this text to a generative AI model (e.g., GPT-3).

[1083] 4. The generative AI model generates a response: The generative AI model generates the response "The weather is sunny today."

[1084] 5. The server sends the response to the terminal: The server sends the generated response text to the terminal as a TCP / IP packet.

[1085] 6. The device converts the response into speech and outputs it: The device uses the Google Cloud Text-to-Speech API to convert the text into speech and transmits it to the user through bone conduction earphones.

[1086] 7. User hears response: The user hears the audio response "The weather is sunny today."

[1087] In this way, the present invention makes it possible to realize natural voice dialogue using a generative model, allowing users to use the system comfortably for extended periods of time.

[1088] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1089] Step 1:

[1090] The user wears bone conduction earphones and asks the system a question such as "What's the weather like today?" This voice is captured by a microphone built into the earphones. The input is the user's voice, and the output is the captured analog voice signal. At this stage, the user's voice is acquired in real time.

[1091] Step 2:

[1092] The device converts the captured analog voice signal into digital data using digital signal processing (DSP). Specifically, the DSP built into the device samples the voice signal and converts the analog voice into digital data. The input is an analog voice signal, and the output is digital voice data. This digital data is sent to a server via Wi-Fi or a mobile network.

[1093] Step 3:

[1094] The server analyzes the received digital voice data and converts it into text data using a voice recognition engine. Specifically, the server converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is digital voice data, and the output is text data. For example, the voice saying "What's the weather like today?" is converted into text "What's the weather like today?"

[1095] Step 4:

[1096] The server sends the converted text data to a generative AI model to generate an appropriate response. A generative AI model (e.g., GPT-3) is used for this. The server inputs the text data as a prompt into the generative AI model and receives the generated response. The input is the text data "What's the weather like today?" and the output is the response text "Today's weather is sunny."

[1097] Step 5:

[1098] The server sends the generated response text to the terminal. The server converts the response text into data packets (using the TCP / IP protocol) and sends them again over the network to the terminal. The input is the generated response text, and the output is the data converted into network packets.

[1099] Step 6:

[1100] The device converts the received response text data into voice data using a speech synthesis engine. Specifically, the device converts text data into voice data using the Google Cloud Text-to-Speech API. The input is the response text data, and the output is the synthesized voice data.

[1101] Step 7:

[1102] The generated voice data is transmitted to the user via bone conduction earphones. The terminal sends the synthesized voice data to the earphones, which transmit the voice to the user using bone conduction technology. The input is the synthesized voice data, and the output is the voice that is actually transmitted to the user. The user can hear the voice response, "Today's weather is sunny."

[1103] (Application example 1)

[1104] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1105] In recent years, there has been a demand for providing customers with fast and accurate information in brick-and-mortar stores. However, it is difficult for store employees to constantly respond to customers, and maintaining customer satisfaction is especially challenging during busy times and peak hours. In addition, existing digital interfaces and guidance systems are complex to operate, making them difficult to use, especially for elderly people and those unfamiliar with technology. For this reason, a new voice dialogue system is needed to efficiently guide customers in brick-and-mortar stores and improve customer convenience.

[1106] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1107] In this invention, the server includes means for capturing the user's voice, means for converting the captured voice into digital data and transmitting it, means for converting the received voice data into text data, means for transmitting the text data to a generative model to generate a response, means for receiving the generated response, converting it into voice and outputting it, and means for providing information about products, services, and sales in a physical store. This allows customers to instantly find out the location of products in the store, special sale information, etc., simply by asking a question by voice.

[1108] "User voice" refers to voice data in natural language spoken by a system user.

[1109] A "capturing means" is a device such as a microphone or bone conduction earphone for acquiring an audio signal.

[1110] "Convert to digital data" is the process of converting analog audio signals into a digital format.

[1111] "Transmitting means" is a network interface for transmitting digital data to other devices or servers.

[1112] "Means for converting to text data" refers to the process of converting voice data into text format using voice recognition technology.

[1113] A "generative model" is an artificial intelligence model that generates appropriate responses based on input text data.

[1114] A "means for generating a response" is a process that uses a generative model to create a response based on text data.

[1115] "Means for converting into voice and outputting" refers to the process of converting text data into voice data using voice synthesis technology and making it available to the user.

[1116] A "brick and mortar store" is a physical store where customers visit in person to transact goods or services.

[1117] "Means for providing product, service and sale information" is a process for providing customers with product locations in stores, service details, special sale information, etc., through the generated response.

[1118] Bone conduction earphones are devices that transmit sound directly to the inner ear through the skull, bypassing the ear.

[1119] As a specific example of a distinctive application of the present invention, we will present a specific example of a system for implementing a "store navigation assistant." This system allows users to ask questions about products, services, and sales information by voice in a physical store and receive appropriate responses.

[1120] Configuration and Operation

[1121] Hardware and software used

[1122] Audio capture device: Bone conduction earphones have a built-in microphone that captures the user's voice.

[1123] Device: The device worn by the user converts the captured audio into digital data and sends it to the server.

[1124] Server: The server converts the voice data into text using the Google Cloud Speech-to-Text API and sends this text to OpenAI's generative AI model. The generated response text is processed within the server and converted into voice data using the Google Cloud Text-to-Speech API. Finally, this voice data is sent to the device and transmitted to the user via bone conduction earphones.

[1125] Data processing and calculation

[1126] 1. Converting voice to digital data: The user's voice is converted into digital data by an encoder on the device, which is then sent to a server via Wi-Fi or a mobile network.

[1127] 2. Speech Recognition: The server converts the voice data into text data using the Google Cloud Speech-to-Text API. Here, the speech recognition engine analyzes the voice signal and converts it into text format.

[1128] 3. Response generation: A generative AI model (OpenAI's ChatGPT API) generates an appropriate response based on the text data. The system uses the generative AI model to create a response to the user's question entered as a prompt.

[1129] 4. Speech synthesis: The generated text response is converted into natural-sounding speech using the Google Cloud Text-to-Speech API. This conversion is performed by a speech synthesis engine.

[1130] 5. Audio output: The device transmits the received audio data to the user through bone conduction earphones.

[1131] Specific use cases

[1132] As a concrete example, consider a situation where a user is in a physical store and asks, "Where are the sale items?" The processing steps in this case are as follows:

[1133] 1. A user asks, "Where are the sales?" while in a brick-and-mortar store.

[1134] 2. The device captures this audio, converts it into digital data, and sends it to the server.

[1135] 3. The server converts the voice data into text data and sends it to the "generative AI model" as a prompt.

[1136] 4. The generative AI model generates the response, "The sale items are located near the center of the store."

[1137] 5. The server converts this response into audio data and sends it back to the terminal.

[1138] 6. The device outputs a response through bone conduction earphones, telling the user, "The sale items are located near the center of the store."

[1139] Prompt Sentence Examples

[1140] User Question: "Where are the sale items?"

[1141] Prompt: "Where are the sale items? They are near the center of the store."

[1142] As described above, the system for implementing this invention is designed to provide users with information about physical stores quickly and appropriately, significantly improving the efficiency of customer service and increasing customer satisfaction.

[1143] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1144] Step 1:

[1145] The user puts on the bone conduction earphones and asks a question.

[1146] Input: User's natural language speech.

[1147] How it works: A user says, "Where are the sale items?" in a physical store. The microphone inside the bone conduction earphones captures this voice.

[1148] Output: Analog audio signal.

[1149] Step 2:

[1150] The device converts the captured audio into digital data and sends it to the server.

[1151] Input: Analog audio signal.

[1152] How it works: An encoder inside the device converts analog audio signals into digital audio data and sends it to a server via Wi-Fi or a mobile network.

[1153] Output: Digital audio data.

[1154] Step 3:

[1155] The server converts the received digital voice data into text data.

[1156] Input: Digital audio data.

[1157] How it works: The server uses the Google Cloud Speech-to-Text API to convert the voice data into text data. The speech recognition engine analyzes the voice signal and converts the data into text.

[1158] Output: Text data.

[1159] Step 4:

[1160] The server sends the text data to the generative AI model, which generates a response.

[1161] Input: Text data.

[1162] How it works: The generative AI model (OpenAI's ChatGPT API) generates an appropriate response based on the text data. The server sends this prompt to the generative AI model, which then receives a response such as "The sale items are located near the center of the store."

[1163] Output: The response text.

[1164] Step 5:

[1165] The server converts the generated response text into audio data.

[1166] Input: Response text.

[1167] How it works: The server uses the Google Cloud Text-to-Speech API to convert the generated text response into audio data. A speech synthesis engine analyzes this text and processes the data to convert it into natural-sounding speech.

[1168] Output: Audio data.

[1169] Step 6:

[1170] The audio data received by the device is transmitted to the user through bone conduction earphones.

[1171] Input: Audio data.

[1172] Specific operation: The terminal plays audio data and provides audio information to the user through bone conduction earphones. The user hears the voice guidance, "The sale items are located near the center of the store."

[1173] Output: Audio information to the user.

[1174] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1175] This invention combines a speech dialogue platform that utilizes a generative model with an emotion engine, and relates to a system that realizes more natural and adaptive dialogue by recognizing the user's emotions and generating responses accordingly. The system includes a user device (including bone conduction earphones), a server that hosts the generative model, and an emotion engine.

[1176] Program processing procedure

[1177] User operations

[1178] The user wears bone conduction earphones and asks the system a question, such as "What's the weather like today?" The voice is captured by a microphone built into the earphones.

[1179] Device audio capture and transmission

[1180] The device converts the captured audio into digital data, which is then transmitted to the server via Wi-Fi or a mobile network. At this time, the transmission module operates to send the audio data to the server in the appropriate data packet format.

[1181] Audio processing on the server

[1182] The server analyzes the received voice data and converts it into text data using a speech recognition engine. It also recognizes the user's emotions using an emotion engine. The emotion engine analyzes the tone, tempo, and pitch of the voice to identify the user's emotional state.

[1183] Emotion recognition by emotion engine

[1184] The emotion engine recognizes the user's emotional state, such as "angry" or "happy," based on features extracted from the user's voice data, allowing responses to be provided in a more appropriate emotional tone.

[1185] Server response generation

[1186] The server sends the text data and the recognized emotion information to the generative model, which then generates a response based on the input data. For example, in response to the question "What's the weather like today?", the model calculates and generates the response "The weather is sunny today."

[1187] Response transmission from the server to the terminal

[1188] The generated response text is sent back from the server to the user's terminal, where it is converted into data packets and sent again over the network to the terminal.

[1189] Device speech synthesis and output

[1190] The device converts the received text data into speech using a speech synthesis engine. This speech data is transmitted to the user using bone conduction earphones. Bone conduction earphones do not block the ears, so they put less strain on the ears even when worn for long periods of time, allowing the user to receive audio information naturally.

[1191] Specific examples

[1192] When a user asks "What's the weather like today?" the following actions occur:

[1193] 1. User asks a question: The user puts on the bone conduction earphones and asks, "What's the weather like today?"

[1194] 2. The device captures the audio: The device captures the user's audio, converts it into digital data, and sends it to the server.

[1195] 3. The server converts the voice data into text: The server converts the voice data into text and recognizes emotions using the emotion engine.

[1196] 4. The emotion engine recognizes emotions: If the emotion “angry” is recognized, that information is also sent to the generative model.

[1197] 5. The generative model generates a response: The generative model generates a response with the appropriate emotional tone, such as "The weather is sunny today."

[1198] 6. The server sends the response to the terminal: The server sends the generated response text to the terminal.

[1199] 7. The device converts the response into speech: The device synthesizes speech and transmits it to the user through bone conduction earphones.

[1200] 8. User hears the response: The user hears the emotionally sensitive voice response: "The weather is sunny today."

[1201] In this way, the present invention combines a generative model and an emotion engine to realize natural and adaptive voice dialogue that responds to the user's emotions. This not only allows users to use the system comfortably for extended periods of time, but also provides a more personalized dialogue experience.

[1202] The processing flow will be explained below.

[1203] Program processing steps

[1204] Step 1: User asks a question

[1205] The user puts on the bone conduction earphones and asks, "What's the weather like today?" The sound is captured by a microphone built into the bone conduction earphones.

[1206] Step 2: Your device captures audio

[1207] The terminal captures the user's voice in real time, and the voice capture module operates to convert the voice signal into digital voice data.

[1208] Step 3: The device sends the audio data to the server

[1209] The device transmits the converted digital audio data to the server via Wi-Fi or mobile network, and the transmission module creates data packets and sends them to the server.

[1210] Step 4: The server receives the audio data

[1211] The server receives the voice data sent from the terminal, and the receiving module analyzes the complete data packet to obtain the voice data.

[1212] Step 5: The server converts the audio data into text data.

[1213] The server converts the received voice data into text data using a voice recognition engine, which analyzes the voice data and generates corresponding text.

[1214] Step 6: The server recognizes the user's emotion using the emotion engine.

[1215] The server inputs the generated text data and voice data features into an emotion engine to recognize the user's emotions. The emotion engine analyzes the tone, tempo, and pitch of the voice to identify the user's emotional state.

[1216] Step 7: The emotion engine identifies the user's emotion

[1217] The emotion engine recognizes, for example, that the user is angry. The recognized emotion information is input into the generative model.

[1218] Step 8: The server sends the text data and emotion information to the generative model

[1219] The server sends the text data and the recognized emotions to the generative model, which generates a response based on the input data.

[1220] Step 9: The generative model generates the response text

[1221] Based on the text data and emotional information sent, the generative model generates a response with an appropriate emotional tone, such as "Today's weather is sunny."

[1222] Step 10: The server sends the generated response to the terminal

[1223] The server returns the generated response text to the user terminal, and the transmission module converts the response text into data packets and sends them to the terminal.

[1224] Step 11: The device receives the response text

[1225] The terminal receives the response text sent from the server, and the receiving module analyzes the data packet to obtain the complete response text.

[1226] Step 12: The device performs speech synthesis

[1227] The device synthesizes the text data and converts it into voice data. The speech synthesis engine converts the text into synthesized voice.

[1228] Step 13: The device plays audio through the bone conduction earphones

[1229] The device plays the synthesized voice through bone conduction earphones, and the bone conduction speaker module outputs the synthesized voice in a format that is easy for the user to hear.

[1230] Step 14: User listens to response

[1231] The user hears the emotionally sensitive voice response "Today's weather is sunny" through the bone conduction earphones, allowing the user to obtain weather information.

[1232] The above is a specific processing flow of the invention that combines an emotion engine, allowing users to enjoy natural and adaptive dialogue according to their emotions.

[1233] Example 2

[1234] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1235] Conventional voice dialogue systems have difficulty recognizing the user's emotions and generating appropriate responses. In particular, responses that do not take the user's emotional state into account are insufficient for realizing natural and adaptive dialogue and may impair the user experience. Furthermore, prolonged use can cause significant strain on the hearing and lead to a lack of comfort.

[1236] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1237] In this invention, the server includes means for receiving and analyzing voice data, means for converting the received voice data into text data, and means for analyzing features of the voice data to identify the user's emotional state, thereby enabling a natural and adaptive response according to the user's emotional state.

[1238] "Means for capturing user voice" refers to technology that captures user-generated voice using a microphone or other input device.

[1239] "Means for converting captured audio into digital data and transmitting it" refers to a technology for converting the acquired audio signal into a digital signal and transmitting it to a server via a communication network.

[1240] "Means for receiving and analyzing transmitted voice data" refers to a technology for receiving voice data transmitted via a network and analyzing that data.

[1241] The "means for converting received voice data into text data" refers to a technology that uses voice recognition technology to convert voice data into corresponding text format data.

[1242] "Means for identifying a user's emotional state by analyzing the features of audio data" refers to technology that analyzes features such as tone, tempo, and pitch extracted from audio data to estimate a user's emotional state.

[1243] "Means for sending the identified emotional state and text data to a generative model to generate an appropriate response" refers to a technology that inputs the analyzed emotional state and converted text data into a generative model to generate an appropriate response based thereon.

[1244] "Means for receiving the generated response, converting it into speech, and outputting it" refers to a technology for receiving the generated text response, converting the text into speech using a speech synthesis engine, and finally outputting it to the user.

[1245] "Bone conduction earphones" refer to acoustic devices that transmit sound to the user's inner ear through the bones without blocking the ear.

[1246] "Hearing reduction measures" refers to technologies or devices designed to reduce the user's hearing, even when used for extended periods of time.

[1247] This invention combines a speech dialogue platform that utilizes a generative model with an emotion engine, and relates to a system that realizes more natural and adaptive dialogue by recognizing the user's emotions and generating responses accordingly. The system includes a user device (including bone conduction earphones), a server that hosts the generative model, and an emotion engine.

[1248] Here, the user wears bone conduction earphones and asks the system a question, such as "What's the weather like today?" This voice is captured by a microphone built into the earphones. The device uses an analog-to-digital converter (ADC) to convert the captured voice into digital data. The converted digital voice data is then sent to the server via Wi-Fi or a mobile network. At this time, the transmission module operates and sends the voice data to the server in the appropriate data packet format.

[1249] The server analyzes the received voice data and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text). It then uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotions. The emotion engine analyzes voice features such as tone, tempo, and pitch to identify the user's emotional state (e.g., "angry" or "happy").

[1250] The emotional information and text data recognized by the emotion engine are sent to a generative model (e.g., OpenAI GPT-4). The generative model generates an appropriate response based on the input data. For example, in response to the question, "What's the weather like today?", the response generated is, "It's sunny today." This response takes into account the user's emotions.

[1251] The generated response text is sent back from the server to the user's device. The server converts the response text into data packets and sends them again over the network to the device. The device converts the received text data into speech using a speech synthesis engine (e.g., Amazon Polly). This speech data is transmitted to the user using bone conduction earphones. Because bone conduction earphones do not block the ears, they put less strain on the ears even when worn for long periods of time, allowing the user to receive audio information naturally.

[1252] As a concrete example, if a user asks "What's the weather like today?" the following actions occur:

[1253] The user asks a question: "What's the weather like today?"

[1254] The device captures the audio, converts it into digital data, and sends it to a server.

[1255] The server converts the speech into text and uses an emotion engine to recognize emotions.

[1256] The emotion engine recognizes emotions: if the emotion "angry" is recognized, it also sends that information to the generative model.

[1257] The generative model generates a response: "The weather is sunny today," with the appropriate emotional tone.

[1258] The server sends a response to the terminal.

[1259] The device converts the response into audio and transmits it to the user through bone conduction earphones.

[1260] The user hears the response: "The weather is sunny today." The user hears the emotionally sensitive voice response.

[1261] By combining a generative model and an emotion engine, the present invention makes it possible to realize natural and adaptive voice dialogue that responds to the user's emotions. This not only allows users to use the device comfortably for extended periods of time, but also allows them to enjoy a more personalized dialogue experience.

[1262] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1263] Step 1:

[1264] A user puts on bone conduction earphones and asks a question such as "What's the weather like today?" The sound is captured by a microphone built into the bone conduction earphones. The input is the user's voice, and the output is an analog audio signal.

[1265] Step 2:

[1266] The terminal converts the captured analog voice signal into digital data using an ADC (Analog-to-Digital Converter). The input is an analog voice signal, and the output is digital voice data. This data conversion process preserves the waveform information of the voice signal.

[1267] Step 3:

[1268] The terminal transmits the converted digital audio data to the server via Wi-Fi or mobile network. The input is digital audio data, and the output is audio data in the form of data packets. The transmission module uses the TCP / IP stack to divide the audio data into small packets.

[1269] Step 4:

[1270] The server recombines the received voice data in data packets and converts it into text data using a speech recognition engine. The input is voice data in data packets, and the output is text data. A commonly used cloud-based API (e.g., Google Cloud Speech-to-Text) is used as the speech recognition engine.

[1271] Step 5:

[1272] The server passes the text data to the emotion engine for analysis. The input is text data, and the output is the user's emotional information. The emotion engine analyzes features such as tone, tempo, and pitch of the voice to identify emotions such as "angry" or "happy."

[1273] Step 6:

[1274] The server sends the text data and the identified emotion information to the generative AI model. The input is the text data and emotion information, and the output is the generated response text. The generative AI model (e.g., OpenAI GPT-4) generates an appropriate response based on this prompt.

[1275] Step 7:

[1276] The server converts the generated response text into data packets and sends them to the terminal again over the network. The input is the generated response text, and the output is the response text in data packet format.

[1277] Step 8:

[1278] The device converts the received response text into speech using a speech synthesis engine (e.g., Amazon Polly). The input is the response text, and the output is voice data. The speech synthesis engine converts the text data into natural-sounding speech, and the voice data is transmitted to the user via bone conduction earphones.

[1279] Step 9:

[1280] The user listens to the generated audio through bone conduction earphones. The input is audio data, and the output is the audio perceived by the user. Bone conduction earphones allow users to receive audio information naturally without blocking their ears, and put less strain on their hearing even when used for long periods of time.

[1281] (Application example 2)

[1282] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1283] Conventional voice dialogue systems have been limited to simple question-and-answer systems without considering the user's emotions, resulting in poor dialogue quality. Furthermore, when dealing with customers in physical stores, it has been difficult to understand the customer's emotions and respond accordingly.

[1284] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing the user's voice, means for converting the captured voice into digital data and transmitting it, means for converting the received voice data into text data, means for transmitting the text data to a generative model to generate a response, means for receiving the generated response, converting it into voice and outputting it, means for analyzing and recognizing the emotional state, and means for adjusting the generated response based on the emotional state. This enables natural and adaptive dialogue according to the customer's emotions, making it possible to improve the quality of customer service in physical stores.

[1285] "Means for capturing user voice" refers to a mechanism for acquiring the voice uttered by the user as digital data.

[1286] "Means for converting captured audio into digital data and transmitting it" refers to technology that processes audio signals, converts them into a digital format, and transmits them to a server via a communication means.

[1287] "Means for converting received voice data into text data" refers to the process by which the server uses voice recognition technology to analyze the voice data and convert it into text information.

[1288] "Means of sending to a generative model to generate a response" refers to a method of inputting text data into a generative AI model to automatically generate an appropriate response.

[1289] The "means for receiving the generated response, converting it into speech, and outputting it" is a system that converts the generated text response into speech using speech synthesis technology and transmits it to the user.

[1290] The "means for analyzing and recognizing emotional states" is a technology for analyzing emotional features from the user's voice and identifying the user's current emotional state.

[1291] The "means for adjusting the generated response based on the emotional state" is a method for adjusting the content and tone of the generated response depending on the emotional state of the user recognized by the emotion engine.

[1292] "Means for outputting sound using bone conduction earphones" refers to an acoustic device that uses bone conduction to transmit sound to the user, and is a technology that transmits sound without blocking the ears.

[1293] "Means for communicating with customers via smart glasses worn by store staff" refers to a system in which store staff wearing smart glasses respond to customers' questions or requests through voice or visual information.

[1294] The present invention is a system that captures a user's voice data, analyzes it using a generative AI model and an emotion engine, generates a response, and conveys it to the user.

[1295] In this system, a user puts on the smart glasses and speaks to ask or request a question. The smart glasses' built-in microphone captures the voice and converts it into digital data. The digital data is then sent to a server via Wi-Fi or a mobile network. A module then operates to transmit the voice data in the appropriate data packet format.

[1296] The server analyzes the received voice data and converts it into text data using a voice recognition engine. For voice recognition, technologies such as "Google Speech Recognition" can be used. Next, an emotion engine analyzes features such as tone, tempo, and pitch of the voice to recognize the user's emotion. This procedure identifies the user's emotional state, such as "angry" or "happy." Emotion analysis technologies such as "IBM Watson" and "Microsoft Azure Cognitive Services" can be used for emotion recognition.

[1297] Based on the analysis results, the emotion information and text data obtained from the emotion engine are sent to a generative AI model, which can use advanced natural language processing models such as GPT-3 or T5 to generate a response with an appropriate emotional tone. The generated response is then sent back from the server to the smart glasses and displayed on their screens.

[1298] Here's a concrete example: If a customer asks, "Tell me about this product," the smart glasses will capture the voice and send the following prompt to the generative AI model:

[1299] "When the customer is calm, they say, 'Tell me about this product.' We create that response."

[1300] The response generated by the generative AI model might be something like, "This product is very popular, especially this feature." This allows store associates to respond to customers in the right emotional tone, improving customer satisfaction.

[1301] The present invention can provide a more natural and adaptive user experience through emotion-based dialogue, which can be applied not only to customer service in brick-and-mortar stores but also to other voice dialogue applications, and has a wide range of applications.

[1302] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1303] Step 1:

[1304] The user wears the smart glasses and inputs questions or requests by voice. The voice is captured by the microphone built into the smart glasses. The input data is a voice signal, and the captured voice signal is passed directly to the next step.

[1305] Step 2:

[1306] The device (smart glasses) converts the captured audio into digital data using a built-in audio-to-digital conversion module, which is then represented as audio samples and sent to a server via Wi-Fi or a mobile network.

[1307] Step 3:

[1308] The server analyzes the received digital voice data using a voice recognition engine and converts it into text data. The voice recognition engine used here is assumed to be "Google Speech Recognition." The input data is digital voice data, and the output data is recognized text data.

[1309] Step 4:

[1310] The server then uses an emotion engine to analyze the text data and its speech features to recognize the user's emotional state. Possible emotion recognition technologies used here include IBM Watson and Microsoft Azure Cognitive Services. The input data are the recognized text data and speech features, and the output data is metadata that represents the user's emotional state.

[1311] Step 5:

[1312] The server sends the emotional information and text data obtained from the emotion engine to the generative model. The generative AI model uses algorithms such as GPT-3 and T5 and receives text data and emotional states as input. The model generates an appropriate response based on this. The generated response is output in text format.

[1313] Step 6:

[1314] The server sends the generated text response back to the terminal (smart glasses). The output data is the text response, which is sent to the smart glasses via the network.

[1315] Step 7:

[1316] The device (smart glasses) converts the received text data into voice data using speech synthesis technology. A speech synthesis engine is used to create a voice response with an appropriate emotional tone. The input data is a text response, and the output data is voice data.

[1317] Step 8:

[1318] The device transmits the generated voice to the user through bone conduction earphones, allowing the user to hear the voice response without covering their ears. The output data is voice, and this is what reaches the user.

[1319] The above steps enable natural voice interaction that takes into consideration the customer's emotions. For example, if a customer asks, "Tell me about this product," the response can be provided in an appropriate tone, such as, "This product is very popular. This feature in particular is attracting attention." This process improves the quality of customer service and increases customer satisfaction.

[1320] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1321] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1322] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1323] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1324] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1325] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1326] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1327] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1328] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1329] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1330] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1331] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1332] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1333] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1334] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1335] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1336] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1337] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1338] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1339] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1340] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1341] The following is further disclosed regarding the above embodiment.

[1342] (Claim 1)

[1343] means for capturing the user's voice;

[1344] means for converting the captured audio into digital data for transmission;

[1345] means for converting received voice data into text data;

[1346] means for transmitting the text data to a generative model to generate a response;

[1347] means for receiving the generated response, converting it into speech, and outputting it;

[1348] A system including:

[1349] (Claim 2)

[1350] 10. The system of claim 1, further comprising means for outputting audio using bone conduction earphones.

[1351] (Claim 3)

[1352] 3. The system according to claim 2, further comprising means for reducing the burden on the user's hearing when the generated audio data is worn for a long period of time.

[1353] (Claim 4)

[1354] 2. The system according to claim 1, further comprising means for reducing the communication data amount of the received voice data.

[1355] "Example 1"

[1356] (Claim 1)

[1357] means for capturing the user's voice;

[1358] means for converting the captured audio into digital data for transmission;

[1359] means for converting received voice data into text data;

[1360] means for transmitting the text data to a generative model to generate a response;

[1361] means for receiving the generated response, converting it into speech, and outputting it;

[1362] digital signal processing means for converting the audio data into digital form;

[1363] means for converting voice data into text data using a voice recognition engine;

[1364] a speech synthesis means for converting the generated response from text to speech;

[1365] A system including:

[1366] (Claim 2)

[1367] 10. The system of claim 1, further comprising means for outputting audio using bone conduction earphones.

[1368] (Claim 3)

[1369] 3. The system according to claim 2, further comprising means for reducing the burden on the user's hearing when the generated audio data is worn for a long period of time.

[1370] "Application Example 1"

[1371] (Claim 1)

[1372] means for capturing the user's voice;

[1373] means for converting the captured audio into digital data for transmission;

[1374] means for converting received voice data into text data;

[1375] means for transmitting the text data to a generative model to generate a response;

[1376] means for receiving the generated response, converting it into speech, and outputting it;

[1377] A means of providing product, service and sale information in physical stores,

[1378] A system including:

[1379] (Claim 2)

[1380] 10. The system of claim 1, further comprising means for outputting audio using bone conduction earphones.

[1381] (Claim 3)

[1382] 2. The system according to claim 1, further comprising means for reducing the strain on the user's hearing when the generated audio data is worn for a long period of time.

[1383] "Example 2: Combining Emotion Engines"

[1384] (Claim 1)

[1385] means for capturing the user's voice;

[1386] means for converting the captured audio into digital data for transmission;

[1387] means for receiving and analyzing the transmitted voice data;

[1388] means for converting received voice data into text data;

[1389] means for analyzing features of the voice data to identify the emotional state of the user;

[1390] means for transmitting the identified emotional state and text data to a generative model to generate an appropriate response;

[1391] means for receiving the generated response, converting it into speech, and outputting it;

[1392] A system including:

[1393] (Claim 2)

[1394] 10. The system of claim 1, further comprising means for outputting audio using bone conduction earphones.

[1395] (Claim 3)

[1396] 3. The system according to claim 2, further comprising means for reducing the burden on the user's hearing when the generated audio data is worn for a long period of time.

[1397] "Application example 2 when combining emotion engines"

[1398] (Claim 1)

[1399] means for capturing the user's voice;

[1400] means for converting the captured audio into digital data for transmission;

[1401] means for converting received voice data into text data;

[1402] means for transmitting the text data to a generative model to generate a response;

[1403] means for receiving the generated response, converting it into speech, and outputting it;

[1404] a means for analyzing and recognizing an emotional state;

[1405] means for adjusting the generated response based on said emotional state;

[1406] A system including:

[1407] (Claim 2)

[1408] 10. The system of claim 1, further comprising means for outputting audio using bone conduction earphones.

[1409] (Claim 3)

[1410] 10. The system of claim 1, further comprising means for interacting with customers via smart glasses worn by store staff. [Explanation of symbols]

[1411] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for capturing the user's voice; means for converting the captured audio into digital data for transmission; means for converting received voice data into text data; means for transmitting the text data to a generative model to generate a response; means for receiving the generated response, converting it into speech, and outputting it; A system including:

2. The system of claim 1 further comprising means for outputting audio using bone conduction earphones.

3. 3. The system according to claim 2, further comprising means for reducing the burden on the user's hearing when the generated audio data is worn for a long period of time.

4. 2. The system according to claim 1, further comprising means for reducing the communication data volume of the received voice data.

Images