System

Abstract

A system is provided.SOLUTION: A system comprising: voice analysis means for capturing voice data of a user and performing voice recognition; generation means for generating an appropriate reaction content based on a voice recognition result; timing calculation means for calculating a response timing of the generated reaction; voice generation means for generating voice data based on the reaction content and the response timing; and reproduction means for reproducing the generated voice data to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] While the use of voice assistants has expanded in recent years, the following issues remain with current voice assistants. First, the conversation feels one-sided, preventing natural dialogue between the user and the system. Second, inaccurate voice recognition can easily lead to frustration among users. As a result, users may find communication with the voice assistant stressful, and some may find it difficult to continue using the system. There is a need for a system that can resolve these issues and enable users to enjoy natural and satisfying dialogue. [Means for solving the problem]

[0005] The present invention provides a system including a voice analysis means for capturing a user's voice data and performing voice recognition, a generation means for generating appropriate reaction content based on the voice recognition results, a timing calculation means for calculating the response timing of the generated reaction, a voice generation means for generating voice data based on the reaction content and the response timing, and a playback means for playing the generated voice data to the user.

[0006] This system analyzes the content of a user's speech in real time and generates reactions that include appropriate emotional expressions. It also listens back to what the user is saying in the middle of it, ensuring accurate speech recognition and adjusting the timing of responses to make them feel natural. This allows for natural communication with the user, providing a less stressful dialogue environment.

[0007] The "voice analysis means" is a means having a function of capturing voice data input by a user in real time and converting the voice data into text.

[0008] The "generation means" is a means having a function of generating appropriate reaction content based on the text data converted by the voice analysis means.

[0009] The "timing calculation means" is a means for calculating the appropriate response timing for the generated reaction and for responding at that timing.

[0010] The "voice generating means" is a means for generating voice data based on the reaction content generated by the generating means and the response timing calculated by the timing calculating means.

[0011] The "playback means" is a means for playing back to the user the voice data generated by the voice generation means. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] This embodiment relates to a voice assistant system that uses a generation AI to automatically generate reaction content, response timing, and tone of voice in order to realize natural dialogue with a user. This system mainly includes the following components:

[0034] 1. Audio analysis methods:

[0035] This is a function that allows the device to capture the user's voice data and perform speech recognition in real time. This allows the user's voice to be converted into text data. For example, if the user says, "The weather is nice today," the device will capture this utterance and generate the text data, "The weather is nice today."

[0036] 2. Generation means:

[0037] This function uses a generative AI to generate appropriate reactions based on the text data received by the server. The generative AI analyzes the user's speech and generates emotional expressions according to the context (e.g., a positive reaction such as "That's right! The weather is really nice today!").

[0038] 3. Timing calculation method:

[0039] This function calculates the appropriate response timing based on the reaction content generated by the server. For example, it calculates the timing to start the response approximately 0.5 seconds after the user finishes speaking.

[0040] 4. Sound Generation Means:

[0041] This function uses speech synthesis technology to generate reaction content as voice based on information from the generation method and timing calculation method. By adding tone of voice and emotion according to the reaction, natural dialogue voice can be generated. For example, if the reaction is "That's right! It's really nice weather today!", a tone of joy will be added to the generated voice.

[0042] 5. Regeneration means:

[0043] This function allows the terminal to play back the voice data sent from the server to the user. This playback function allows the user to experience natural conversation.

[0044] Program processing

[0045] The specific program processing is performed as follows: Below is a specific example of how the system responds to user utterances.

[0046] 1. The user says, "The weather is nice today."

[0047] 2. The device captures the user's voice and uses a speech recognition engine to generate text data such as "The weather is nice today."

[0048] 3. The device sends the generated text data to the server.

[0049] 4. The server receives the text data and uses generative AI to generate the most appropriate reaction for the context: "Yes! It's really nice weather today!"

[0050] 5. The server uses the response timing calculation module to calculate an appropriate interval for the response (e.g., 0.5 seconds).

[0051] 6. The server uses the speech generation module to generate a reaction, "Yes! It's really nice weather today!", with a joyful tone added.

[0052] 7. The device receives the generated voice data and plays it back to the user based on the calculated response timing.

[0053] This allows the user to experience a natural and smooth conversation with the voice assistant. The system of this embodiment automatically generates reactions, calculates response timing, and generates voices, so it can provide a natural dialogue environment without requiring user operation. The system can also achieve more accurate dialogue by using call center recording data and other call data as training data.

[0054] The processing flow will be explained below.

[0055] Step 1:

[0056] The user speaks to the voice assistant. Specifically, the user says, "The weather is nice today."

[0057] Step 2:

[0058] The device captures the user's voice with a microphone and temporarily stores it as voice data. At the same time, the device's voice recognition engine activates and converts the voice data into text data in real time. For example, the text generated might be, "The weather is nice today."

[0059] Step 3:

[0060] The terminal sends the generated text data to the back-end server, where it arrives via the network.

[0061] Step 4:

[0062] The server receives the text data. The generative AI module in the server analyzes the text data and understands the user's intent. Specifically, it extracts the positive context of "the weather is nice."

[0063] Step 5:

[0064] The server uses the generative AI module to generate an appropriate reaction, for example, "Yes! It's really nice weather today!" This reaction is intended to be a positive response to the user's comment.

[0065] Step 6:

[0066] The server uses a response timing calculation module to calculate the optimal timing for the reaction. Based on the speech analysis, it determines the timing to start the response, for example, 0.5 seconds later.

[0067] Step 7:

[0068] The server converts the generated text reaction into speech data using a speech generation module, where the generated speech has an appropriate emotional tone (in this example, a tone of joy).

[0069] Step 8:

[0070] The server sends the generated voice data to the terminal, which then receives the voice data via the network.

[0071] Step 9:

[0072] The device stores the received voice data in an internal buffer and plays it back through the speaker after the 0.5 second interval calculated based on the response timing has elapsed.

[0073] Step 10:

[0074] The device will play a voice message over the speaker saying, "Yes! It's really nice weather today!" This allows the user to experience a natural conversation.

[0075] Step 11:

[0076] The user receives a response from the voice assistant and can continue the conversation. The system waits for a new utterance.

[0077] In this way, the system automatically captures and analyzes the user's speech, generating natural reactions at the appropriate times and responding as voice, allowing users to enjoy natural, stress-free conversations.

[0078] Example 1

[0079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0080] Conventional voice assistant systems often lack natural dialogue with users, particularly insufficient response timing and emotional expression. This leads to lower user satisfaction and a lower rate of continued use of the system. The goal of this invention is to achieve more natural, human-like dialogue and increase user satisfaction.

[0081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0082] In this invention, the server includes a generation means for generating appropriate reaction content using a generation AI model based on the speech recognition result, a timing calculation means for calculating the response timing of the generated reaction, and a voice generation means for generating voice data based on the reaction content and response timing. This enables natural and smooth dialogue by analyzing and responding to the user's speech in real time and expressing emotions appropriately.

[0083] "User" refers to the individual or end user who uses the voice assistant system.

[0084] "Voice data" refers to data that records what a user says as a digital signal.

[0085] "Voice analysis means" refers to equipment or software for capturing a user's voice data and performing voice recognition.

[0086] A "generative AI model" refers to an artificial intelligence model that uses machine learning and neural network technology to generate optimal output for input data.

[0087] "Generation means" refers to a device or program that has the function of generating appropriate reaction content using a generative AI model based on the voice recognition results.

[0088] "Response timing" refers to the time interval between a user's utterance and the system's response.

[0089] "Timing calculation means" refers to a device or program that has the function of calculating the response timing of the generated reaction.

[0090] "Speech generation means" refers to a device or program that has the function of generating voice data based on the generated reaction content and response timing.

[0091] "Playback means" refers to equipment or software for playing back the generated audio data to the user.

[0092] "System" refers to the entire voice assistant, including the above means, that enables natural dialogue with the user.

[0093] The present invention relates to a voice assistant system that uses generation AI to automatically generate reaction content, response timing, and tone of voice in order to realize natural dialogue with users. This system mainly includes the following components:

[0094] 1. Audio analysis methods:

[0095] The device has the ability to capture the user's voice data and perform voice recognition in real time. This typically uses the device's built-in microphone and a voice recognition engine (such as a general voice recognition API). For example, if a user says, "The weather is nice today," the device captures this speech and generates "The weather is nice today" as text data.

[0096] 2. Generation means:

[0097] The system has the ability to generate appropriate reactions using a generative AI model based on the text data received by the server. This uses a generative AI model (e.g., a widely used generative AI technology). The server analyzes the user's speech and generates emotional expressions according to the context. For example, it generates a positive reaction such as, "Yes! It's really nice weather today!"

[0098] 3. Timing calculation method:

[0099] The server has a function to calculate the appropriate response timing based on the reaction content generated by the generation means. For example, it calculates the timing so that the response starts about 0.5 seconds after the user finishes speaking. For this, a timing calculation module for timing control within the program is used.

[0100] 4. Sound Generation Means:

[0101] The server has the function of generating reaction content as voice using voice synthesis technology based on information from the generation means and timing calculation means. In particular, it is possible to generate natural dialogue voice by adding emotions and tone of voice according to the reaction. This implementation uses a voice synthesis API (e.g., a widely used voice synthesis technology). Specifically, voice is generated by adding a tone of joy.

[0102] 5. Regeneration means:

[0103] The device has a function to play back the voice data sent from the server to the user. This playback means allows the user to experience a natural conversation. The voice is output using the device's built-in speaker.

[0104] Specific examples

[0105] For example, the following prompt sentence is input to a generative AI model:

[0106] "When a user says, 'The weather is nice today,' generate an appropriate response."

[0107] Given this input, the generator AI will output the following:

[0108] "Yes! It's really beautiful weather today!"

[0109] Based on the reactions generated in this way, the server calculates the response timing and generates voice data.Finally, the device receives the generated voice data and plays it back to the user at the appropriate time.

[0110] This allows users to have a natural and smooth conversation with the voice assistant. The system of the present invention provides a natural conversation environment by automatically processing each step, without requiring user operation.

[0111] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0112] Step 1:

[0113] The user says, "The weather is nice today."

[0114] Specific operation: The device's built-in microphone captures the user's speech.

[0115] Input: User's voice

[0116] Output: Audio data captured within the device

[0117] Step 2:

[0118] The voice data captured by the terminal is converted into text data using a voice recognition engine.

[0119] Specific operation: The device calls the Google Speech-to-Text API or similar and converts the voice data into text data such as "The weather is nice today."

[0120] Input: Audio data

[0121] Output: Text data "The weather is nice today."

[0122] Step 3:

[0123] The terminal transmits the generated text data to the server.

[0124] Specific operation: The terminal generates an HTTP request and sends text data to the server.

[0125] Input: Text data "The weather is nice today"

[0126] Output: Text data received by the server

[0127] Step 4:

[0128] The server uses a generative AI model to generate appropriate reaction content based on the text data received.

[0129] How it works: The server uses a generative AI model such as OpenAI GPT-4 to input text data as a prompt and generate a response text such as "Yes! It's really nice weather today!"

[0130] Input: Text data "The weather is nice today"

[0131] Output: Response text "Yes! It's really nice weather today!"

[0132] Step 5:

[0133] The server calculates the appropriate response timing based on the generated response text.

[0134] Specific operation: The server uses the timing calculation module to calculate the timing to respond 0.5 seconds after the user finishes speaking.

[0135] Input: Response text "Yes! It's really beautiful weather today!"

[0136] Output: Response timing (e.g., after 0.5 seconds)

[0137] Step 6:

[0138] The server generates voice data using a voice synthesis technique based on the response text and response timing.

[0139] Specific operation: The server calls the Google Cloud Text-to-Speech API or similar and generates the response text as voice data with a joyful tone added.

[0140] Input: Response text "Yes! It's really nice weather today!", Response timing

[0141] Output: Audio data

[0142] Step 7:

[0143] The terminal receives the generated voice data and plays it back to the user based on the calculated response timing.

[0144] Specific operation: The device stores the voice data in a buffer, checks the response timing, and then plays the voice through the built-in speaker.

[0145] Input: Voice data, response timing

[0146] Output: Audio output to the user

[0147] (Application example 1)

[0148] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0149] The goal is to provide an assistant system that makes interactions with customers in physical stores more natural and effective. In particular, when customers ask questions about products, it is required to provide smooth and easy-to-understand information. Conventional systems have difficulty increasing customer satisfaction and purchasing motivation because their responses to questions are unnatural and the information they provide is inaccurate.

[0150] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0151] In this invention, the server includes a voice analysis means for capturing user voice data and performing voice recognition, a generation means for generating appropriate reaction content based on the voice recognition result, a timing calculation means for calculating the response timing of the generated reaction, a voice generation means for generating voice data based on the reaction content and the response timing, a playback means for playing back the generated voice data to the user, an information provision means for generating appropriate product information in response to the user's question, and a voice information playback means for playing back the information generated by the information provision means as voice, thereby enabling natural and quick responses to customer questions and the provision of accurate product information.

[0152] The "voice analysis means" is a means for capturing the user's voice data and performing voice recognition.

[0153] The "generation means" is a means for generating appropriate reaction content based on the speech recognition result.

[0154] The "timing calculation means" is a means for calculating the response timing of the generated reaction.

[0155] The "voice generating means" is a means for generating voice data based on the reaction content and response timing.

[0156] The "playback means" is a means for playing back the generated audio data to the user.

[0157] The "information providing means" is a means for generating appropriate product information in response to a user's question.

[0158] The "audio information reproducing means" is a means for reproducing the information generated by the information providing means as audio.

[0159] This embodiment will be described as a specific example of a voice assistant system that realizes natural conversations in a brick-and-mortar store.

[0160] The system uses smart glasses to support natural interactions between customers and store staff. Its main functions include voice analysis, reaction generation, timing calculation, voice generation, product information provision, and voice playback.

[0161] The server first uses a voice analysis means to capture the voice data sent from the smart glasses and performs voice recognition in real time. This voice recognition engine uses Google Cloud Speech-to-Text. Next, a generation means generates an appropriate reaction content based on the voice recognition results. This generation means uses OpenAI's GPT-4. The generated text data is passed to a timing calculation means to calculate the response timing of the reaction. Then, based on the calculated appropriate response timing, a voice generation means converts the reaction content into voice data. Google Cloud Text-to-Speech is used to generate this voice data. Finally, the generated voice data is played back to the user through the playback means of the smart glasses.

[0162] Furthermore, when a user asks a question about a product, the information providing means generates appropriate product information and plays it back as voice. Through this series of processes, customers can receive a quick and accurate response to their question, improving their shopping experience in physical stores.

[0163] As a concrete example, consider a case where a customer asks, "What is this item made of?" in a store. The voice assistant on the smart glasses captures this question, and Google Cloud Speech-to-Text converts it into text: "What is this item made of?" OpenAI GPT-4 then generates the appropriate response: "This item is made of 100% cotton." Google Cloud Text-to-Speech then converts this response into audio, which is played back to the customer through the smart glasses.

[0164] The following text is used as an example of a prompt sentence:

[0165] "A customer asks about the material of an item. Generate a natural response."

[0166] In this way, this system combines the functions of voice recognition, generative AI, timing calculation, voice generation, and information provision to enable natural conversations in physical stores.

[0167] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0168] Step 1:

[0169] The device (smart glasses) captures the user's voice. The input is the user's voice data, and the output is the captured voice data. Specifically, the microphone in the smart glasses records the user's speech in real time.

[0170] Step 2:

[0171] The device sends the captured voice data to the server, where speech recognition is performed using a speech analysis tool. The input is voice data, and the output is the speech recognition result (text data). Specifically, Google Cloud Speech-to-Text analyzes the voice data and generates text data such as "What is the material of this product?"

[0172] Step 3:

[0173] The server's generation means receives the speech recognition results (text data) and uses generation AI to generate an appropriate reaction. The input is text data, and the output is the generated reaction (text). Specifically, OpenAI's GPT-4 generates a response such as "This product is made of 100% cotton."

[0174] Step 4:

[0175] The server's timing calculation means calculates the response timing of the generated reaction. The input is the generated reaction content (text), and the output is the response timing (time interval). Specifically, it calculates the appropriate response timing (e.g., 0.5 seconds) after the user's utterance ends.

[0176] Step 5:

[0177] The server's voice generation means generates voice data based on the reaction content and response timing. The input is the generated reaction content (text) and response timing, and the output is voice data. Specifically, Google Cloud Text-to-Speech generates voice data saying, "This product is made of 100% cotton."

[0178] Step 6:

[0179] The server sends the generated voice data to the terminal. The input is the voice data, and the output is the completion of sending the voice data to the terminal. Specifically, the generated voice data is sent to the smart glasses via the network.

[0180] Step 7:

[0181] The device plays the generated voice data to the user based on the calculated response timing. The input is the voice data and the response timing, and the output is the voice playback. Specifically, the speaker on the smart glasses plays the voice, "This product is made of 100% cotton."

[0182] This series of steps allows users to get answers to their questions through natural dialogue.

[0183] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0184] This embodiment relates to a voice assistant system that automatically generates reaction content, response timing, and tone of voice using a generation AI and an emotion engine to realize natural dialogue with a user. This system mainly includes the following components:

[0185] 1. Audio analysis methods:

[0186] This is a function that allows the device to capture the user's voice data and perform speech recognition in real time. This allows the user's voice to be converted into text data. For example, if the user says, "The weather is nice today," the device will capture this utterance and generate the text data, "The weather is nice today."

[0187] 2. Emotion Engine:

[0188] This function recognizes the user's emotions by analyzing the text and voice data obtained by the server using voice analysis. This emotion engine determines the user's emotional state by taking into account not only the content of the speech but also the tone and pace of the voice.

[0189] 3. Generation means:

[0190] This function uses a generative AI to generate appropriate reactions based on the text data received by the server and the emotional data recognized by the emotion engine. The generative AI compares the content of the user's utterance with their emotions and generates an emotional expression that matches the context (e.g., a positive reaction such as "That's right! The weather is really nice today!").

[0191] 4. Timing calculation method:

[0192] This function calculates the appropriate response timing based on the reaction content generated by the server. For example, it calculates the timing to start the response approximately 0.5 seconds after the user finishes speaking.

[0193] 5. Sound Generation Means:

[0194] This function uses speech synthesis technology to generate the reaction content as voice based on the information from the generation means and timing calculation means. The emotions recognized by the emotion engine are also reflected. Therefore, the generated voice contains an appropriate emotional tone (in this example, a tone of joy).

[0195] 6. Regeneration means:

[0196] This function allows the terminal to play back the voice data sent from the server to the user. This playback function allows the user to experience natural conversation.

[0197] Program processing

[0198] The specific program processing is performed as follows: Below is a specific example of how the system responds to user utterances.

[0199] 1. The user says, "The weather is nice today."

[0200] 2. The device captures the user's voice and uses a speech recognition engine to generate text data such as "The weather is nice today."

[0201] 3. The device sends the generated text data to the back-end server, where it arrives via the network.

[0202] 4. The server receives the text data and analyzes the user's emotions using an emotion engine that works in conjunction with the generative AI. For example, the user's emotions are recognized as positive.

[0203] 5. The server uses the generative AI to generate an appropriate reaction. Based on the analysis results of the emotion engine, a response such as "Yes! It's really nice weather today!" is generated. This reaction corresponds to the user's emotions.

[0204] 6. The server uses the response timing calculation module to calculate an appropriate interval for the response (e.g., 0.5 seconds).

[0205] 7. The server uses the speech generation module to generate a reaction, "Yes! It's really nice weather today!", with a joyful tone added.

[0206] 8. The device receives the generated voice data and plays it back to the user based on the calculated response timing.

[0207] 9. The device will play "Yes! It's really nice weather today!" over the speaker, allowing the user to experience a natural conversation.

[0208] In this way, the system automatically performs a series of processes, from capturing and analyzing the user's speech to generating natural reactions at the appropriate time and responding as voice. This allows users to enjoy natural, stress-free conversations. The system can also achieve more accurate conversations by using call center recordings and other call data as training data.

[0209] The processing flow will be explained below.

[0210] Step 1:

[0211] The user speaks to the voice assistant. Specifically, the user says, "The weather is nice today."

[0212] Step 2:

[0213] The device captures the user's voice with a microphone and temporarily stores it as voice data. At the same time, the device's voice recognition engine activates and converts the voice data into text data in real time. For example, the text generated might be, "The weather is nice today."

[0214] Step 3:

[0215] The terminal sends the generated text data to the back-end server, where it arrives via the network.

[0216] Step 4:

[0217] The server receives the text data and begins analyzing it. As a preprocessing step for the generative AI, the text data is analyzed based on the context.

[0218] Step 5:

[0219] The server uses an emotion engine to analyze the user's emotions from the voice and text data, for example, recognizing that the user is in a positive emotion from the tone of voice and pace of speech.

[0220] Step 6:

[0221] The server uses generative AI to generate an appropriate reaction, taking into account the emotional data obtained from the emotion engine. Specifically, a positive reaction such as "Yes! It's really nice weather today!" is generated.

[0222] Step 7:

[0223] The server uses a response timing calculation module to calculate the optimal response timing for the generated reaction. It detects the end of the user's utterance and determines the timing to start the response, for example, 0.5 seconds later.

[0224] Step 8:

[0225] The server uses a voice generation module to convert the generated reaction text into voice data, which reflects emotional expressions such as a tone of joy.

[0226] Step 9:

[0227] The server sends the generated voice data to the terminal, which then receives the voice data via the network.

[0228] Step 10:

[0229] The device stores the received audio data in an internal buffer and waits until the calculated response timing. After a 0.5 second delay, the device prepares to play the audio data.

[0230] Step 11:

[0231] The device will play the voice "Yes! It's really nice weather today!" through the speaker, allowing the user to experience a natural and emotional interaction.

[0232] Step 12:

[0233] The user receives the voice assistant's response and returns to a system state where they can continue the conversation. The system waits for a new utterance.

[0234] In this way, the system automatically captures and analyzes the user's speech, recognizes their emotions, generates natural reactions at the appropriate time, and responds in voice, allowing users to enjoy natural, stress-free conversations.

[0235] Example 2

[0236] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0237] Conventional voice assistant systems often simply process user speech as text data and then react to it. This makes it difficult to understand and respond to the user's emotions and context, making it difficult to achieve natural dialogue with the user. Furthermore, the tone and timing of the generated voice can be mechanical, making the responses to the user seem unnatural.

[0238] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an analysis means for capturing user voice data and performing voice recognition, an analysis means for recognizing the user's emotion based on the voice recognition result and the voice data, a generation means for generating appropriate reaction content based on the analyzed emotion data and the voice recognition result, a calculation means for calculating the response timing of the generated reaction, a generation means for generating voice data based on the reaction content and response timing, and a playback means for playing the generated voice data to the user. This makes it possible to automatically generate natural reactions that take into account the user's utterance content and emotion, and play them as voice at appropriate timing.

[0239] "User" refers to an individual who uses this system to provide voice input.

[0240] "Voice Data" means information that is a digital recording of a user's speech.

[0241] "Analysis Means" refers to technical equipment or software for capturing audio data and performing voice recognition and emotion analysis.

[0242] "Text data" refers to character information converted from voice data by voice recognition.

[0243] "Emotion Engine" refers to a technical device or software for analyzing text and voice data to determine a user's emotional state.

[0244] "Generation means" refers to a technical device or software for automatically creating appropriate reaction content based on the analysis results of the emotion engine and the voice recognition results.

[0245] "Calculation means" refers to a technical device or software for calculating response timing based on reaction content.

[0246] "Generative AI" refers to artificial intelligence technology that generates appropriate reactions based on the user's speech and emotions.

[0247] A "prompt sentence" is an instruction sentence input to the generation AI that provides context for reaction generation.

[0248] "Speech generation means" refers to technical devices or software for generating reactions as synthetic voices based on text data and emotion data.

[0249] "Playback means" refers to a technical device for playing back the generated audio data in a form audible to the user.

[0250] This invention relates to a voice assistant system that analyzes a user's voice data, recognizes emotions, generates appropriate reactions, and provides the reactions to the user as voice. This system uses a generative AI model and an emotion engine to realize natural dialogue with the user.

[0251] 1. Audio analysis methods:

[0252] The device captures the user's voice data and performs real-time speech recognition. Specific speech recognition software used includes Google Speech-to-Text API and Amazon Transcribe. This speech analysis method converts the user's speech into text data. For example, if a user says, "The weather is nice today," the device captures this speech and generates the text data, "The weather is nice today."

[0253] 2. Emotion Engine:

[0254] The server analyzes the text and voice data obtained through the voice analysis tool to recognize the user's emotions. Specifically, emotion analysis software such as IBM Watson Tone Analyzer is used. This emotion engine determines the user's emotional state by taking into account not only the content of the speech but also the tone and pace of the voice.

[0255] 3. Generation means:

[0256] Based on the text data received by the server and the emotional data recognized by the emotion engine, an appropriate reaction is generated using a generative AI. The specific generative AI used includes OpenAI GPT-3. This generator compares the user's speech content with their emotions to generate emotional expressions that fit the context. For example, if the prompt text is input as "The user is in a positive mood and talking about the weather. Please generate an appropriate response," the response generated will be "Yes! The weather is really nice today!"

[0257] 4. Timing calculation method:

[0258] The server calculates an appropriate response timing based on the reaction content generated by the generation means. For example, a software module is provided to calculate the timing so that the response starts with a time interval of about 0.5 seconds after the user finishes speaking.

[0259] 5. Sound Generation Means:

[0260] The server uses speech synthesis technology based on the information from the generation method and timing calculation method to generate the reaction content as voice. Specific speech synthesis technologies used include Google Text-to-Speech API and Amazon Polly. The emotional tone recognized by the emotion engine is reflected in the voice. For example, the generated voice may include a tone of joy.

[0261] 6. Regeneration means:

[0262] The terminal reproduces the voice data sent from the server to the user, allowing the user to experience natural conversation.

[0263] By combining the above methods, the system can capture and analyze user utterances, and automatically generate natural reactions and respond as voice at the appropriate time. This allows users to enjoy natural, stress-free conversations. The system can also achieve higher accuracy by using call center recordings and other call data as training data.

[0264] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0265] Step 1:

[0266] The user inputs voice data. For example, the user might say, "The weather is nice today." This voice data becomes the input.

[0267] Step 2:

[0268] The device captures the user's voice data and converts it into text data using a speech recognition engine (e.g., Google Speech-to-Text API). The input is the user's voice data, and the output is the text data generated by speech recognition: "The weather is nice today."

[0269] Step 3:

[0270] The terminal sends the generated text data to the back-end server. The input is text data, and the output reaches the server via the network.

[0271] Step 4:

[0272] Based on the text data received by the server, an emotion engine (e.g., IBM Watson Tone Analyzer) is used to analyze the user's emotions. The input is text data, and "positive emotions" are recognized through data analysis. The output is the analyzed emotion data.

[0273] Step 5:

[0274] The server uses a generative AI (e.g., OpenAI GPT-3) to generate an appropriate reaction based on emotional and text data. The input is emotional and text data, and the prompt is, "The user is talking about the weather in a positive mood. Please generate an appropriate response." The output is the response generated by the generative AI: "Yes! The weather is really nice today!"

[0275] Step 6:

[0276] The server uses a response timing calculation module to calculate the appropriate interval (e.g., 0.5 seconds) for the response based on the reaction content. The input is the reaction content, and the output is the calculated response timing.

[0277] Step 7:

[0278] The server uses a speech generation module (e.g., Google Text-to-Speech API) to generate the reaction as speech data. The input is the text data "Yes! It's really nice weather today!" and an emotional tone (a tone of joy), and the output is the generated speech data.

[0279] Step 8:

[0280] The terminal receives the voice data sent from the server and plays it back to the user based on the calculated response timing. The input is the voice data and the response timing, and the output is audible voice for the user.

[0281] Step 9:

[0282] The device will play a voice message over the speaker saying, "Yes! It's really nice weather today!" This action allows the user to experience a natural conversation.

[0283] (Application example 2)

[0284] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0285] There is a demand for voice assistant systems that can provide more natural and effective customer service in brick-and-mortar stores. Conventional voice assistant systems have issues in that they are unable to fully achieve functions such as natural conversation that reflects the customer's emotions, or providing store guidance, product explanations, and promotional information.

[0286] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0287] In this invention, the server includes a voice analysis means for capturing user voice data and performing voice recognition, a generation means for generating appropriate reaction content based on the voice recognition result and the analyzed emotional state, a timing calculation means for calculating the response timing of the generated reaction, a voice generation means for generating voice data based on the reaction content and the response timing, a playback means for playing back the generated voice data to the user, and an information provision means for providing product information, store guidance, and promotion information. This allows for natural dialogue with customers and makes it possible to effectively provide in-store guidance, product explanations, and promotion information.

[0288] The "voice analysis means" is a function for capturing the user's voice data and performing voice recognition in real time.

[0289] The "generation means" is a function that generates appropriate reaction content based on the voice recognition results and the analyzed emotional state.

[0290] The "timing calculation means" is a function that calculates the response timing of the generated reaction.

[0291] The "voice generating means" is a function that generates voice data based on the reaction content and response timing.

[0292] The "playback means" is a function that plays back the generated audio data to the user.

[0293] "Information providing means" is a function that provides information about products, store guides, and promotional information.

[0294] To implement this invention, a specific system configuration and program are required. The hardware and software for realizing this system, as well as the methods for processing and calculating data, will be described below.

[0295] System Overview

[0296] The system has the following main components:

[0297] 1. Voice analysis means: Captures the user's voice data and performs voice recognition.

[0298] 2. Generation method: Generate appropriate reaction content based on the speech recognition results and the analyzed emotional state.

[0299] 3. Timing calculation means: Calculate the response timing of the generated reaction.

[0300] 4. Voice generation means: Generates voice data based on the reaction content and response timing.

[0301] 5. Playback means: Plays the generated audio data to the user.

[0302] 6. Information provision means: Providing product information, store guides, and promotional information.

[0303] Hardware and software used

[0304] Microphones and speakers: These are built into smartphones, smart glasses, and head-mounted displays.

[0305] Cloud servers: Used to run data analysis and generative AI models.

[0306] Generative AI models: Uses open-source GPT-3 and Azure Cognitive Services.

[0307] Emotion engine: Uses IBM Watson Tone Analyzer.

[0308] Speech recognition and synthesis: Uses Google Speech-to-Text and Amazon Polly.

[0309] Program processing flow

[0310] The program of this system operates as follows.

[0311] First, the user speaks to the voice assistant, and the voice data is captured through a microphone and converted into text data using a speech recognition method. Specifically, Google Speech-to-Text is used to convert voice to text.

[0312] This text data is then sent to a cloud server where an emotion engine analyzes the user's emotions, for example, using IBM Watson Tone Analyzer to determine whether the user is expressing positive emotions.

[0313] Based on the analyzed emotional state, the generator uses GPT-3 to generate an appropriate reaction, such as "This product uses new technology and is very durable."

[0314] Next, the timing calculation means calculates the response timing of approximately 0.5 seconds, and the voice generation means converts the generated reaction into voice data using Amazon Polly.

[0315] Finally, the playback means plays back the audio data through a speaker, allowing the user to experience natural dialogue.

[0316] Usage example

[0317] For example, consider a situation where a customer asks a voice assistant, "Tell me about the latest promotions." Here's a sample interaction:

[0318] Specific examples

[0319] 1. User: "Tell me about the latest promotions."

[0320] 2. Voice Capture: "What are the latest promotions?"

[0321] 3. Speech recognition: Generate text data for "Tell me the latest promotion information"

[0322] 4. Data Transmission

[0323] 5. Sentiment Analysis: Recognizing User Interest

[0324] 6. Reaction Generation: "These new shoes are currently 20% off!"

[0325] 7. Response timing calculation: approx. 0.5 seconds

[0326] 8. Speech synthesis: Convert "These new shoes are currently 20% off!" into speech data

[0327] 9. Audio playback

[0328] Prompt Sentence Examples

[0329] Customer: "Tell me about this product."

[0330] Analysis results:

[0331] Emotion: Interested

[0332] Product information: New product, durable, priced at $50

[0333] Prompt the generative AI model:

[0334] Input text: Please explain what happens when a customer expresses interest in a product that uses new technology.

[0335] Output: "This product uses the latest technology and is extremely durable. It costs $50."

[0336] In this way, the system can realize natural dialogue with customers and effectively provide in-store guidance, product explanations, and promotional information.

[0337] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0338] Step 1:

[0339] The user speaks to the voice assistant. The input is the user's voice data, and the specific action is the user saying, "Tell me about this product."

[0340] Step 2:

[0341] The device captures the user's voice data and performs speech recognition. The input is the user's voice data and the output is text data. The speech "Tell me about this product" is converted into text "Tell me about this product."

[0342] Step 3:

[0343] The device sends the generated text data to the cloud server. The input is text data, and the output is data transmission to the server.

[0344] Step 4:

[0345] The server receives the text data and uses the emotion engine to analyze the user's emotions. The input is text data, and the output is analyzed emotion data. Specifically, the text "Tell me about this product" is sent, and the emotion engine analyzes it as "Interested."

[0346] Step 5:

[0347] The server uses a generative AI model to generate appropriate reaction content based on the analyzed emotional state and text data. The input is text data and emotional data, and the output is the generated reaction content. Specifically, in response to the question "Tell me about this product," the server generates the response "This product uses the latest technology and is extremely durable."

[0348] Step 6:

[0349] The server uses the response timing calculation means to calculate the response timing of the generated reaction. The input is the generated reaction content, and the output is response timing information. Specifically, the server calculates the response timing of approximately 0.5 seconds.

[0350] Step 7:

[0351] The server uses a voice generation means to convert the generated reaction into voice data. The input is the generated reaction content and response timing information, and the output is voice data. Specifically, the text "This product uses the latest technology and is very durable" is converted into voice data.

[0352] Step 8:

[0353] The device plays back the audio data sent from the server. The input is audio data, and the output is audio playback to the user. Specifically, the device plays back through the speaker the audio, "This product uses the latest technology and is extremely durable."

[0354] This series of processing steps enables the system to have natural dialogue with the user, and effectively provide in-store guidance, product explanations, and promotional information.

[0355] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0356] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0357] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0358] [Second embodiment]

[0359] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0360] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0361] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0362] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0363] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0364] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0365] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0366] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0367] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0368] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0369] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0370] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0371] This embodiment relates to a voice assistant system that uses a generation AI to automatically generate reaction content, response timing, and tone of voice in order to realize natural dialogue with a user. This system mainly includes the following components:

[0372] 1. Audio analysis methods:

[0373] This is a function that allows the device to capture the user's voice data and perform speech recognition in real time. This allows the user's voice to be converted into text data. For example, if the user says, "The weather is nice today," the device will capture this utterance and generate the text data, "The weather is nice today."

[0374] 2. Generation means:

[0375] This function uses a generative AI to generate appropriate reactions based on the text data received by the server. The generative AI analyzes the user's speech and generates emotional expressions according to the context (e.g., a positive reaction such as "That's right! The weather is really nice today!").

[0376] 3. Timing calculation method:

[0377] This function calculates the appropriate response timing based on the reaction content generated by the server. For example, it calculates the timing to start the response approximately 0.5 seconds after the user finishes speaking.

[0378] 4. Sound Generation Means:

[0379] This function uses speech synthesis technology to generate reaction content as voice based on information from the generation method and timing calculation method. By adding tone of voice and emotion according to the reaction, natural dialogue voice can be generated. For example, if the reaction is "That's right! It's really nice weather today!", a tone of joy will be added to the generated voice.

[0380] 5. Regeneration means:

[0381] This function allows the terminal to play back the voice data sent from the server to the user. This playback function allows the user to experience natural conversation.

[0382] Program processing

[0383] The specific program processing is performed as follows: Below is a specific example of how the system responds to user utterances.

[0384] 1. The user says, "The weather is nice today."

[0385] 2. The device captures the user's voice and uses a speech recognition engine to generate text data such as "The weather is nice today."

[0386] 3. The device sends the generated text data to the server.

[0387] 4. The server receives the text data and uses generative AI to generate the most appropriate reaction for the context: "Yes! It's really nice weather today!"

[0388] 5. The server uses the response timing calculation module to calculate an appropriate interval for the response (e.g., 0.5 seconds).

[0389] 6. The server uses the speech generation module to generate a reaction, "Yes! It's really nice weather today!", with a joyful tone added.

[0390] 7. The device receives the generated voice data and plays it back to the user based on the calculated response timing.

[0391] This allows the user to experience a natural and smooth conversation with the voice assistant. The system of this embodiment automatically generates reactions, calculates response timing, and generates voices, so it can provide a natural dialogue environment without requiring user operation. The system can also achieve more accurate dialogue by using call center recording data and other call data as training data.

[0392] The processing flow will be explained below.

[0393] Step 1:

[0394] The user speaks to the voice assistant. Specifically, the user says, "The weather is nice today."

[0395] Step 2:

[0396] The device captures the user's voice with a microphone and temporarily stores it as voice data. At the same time, the device's voice recognition engine activates and converts the voice data into text data in real time. For example, the text generated might be, "The weather is nice today."

[0397] Step 3:

[0398] The terminal sends the generated text data to the back-end server, where it arrives via the network.

[0399] Step 4:

[0400] The server receives the text data. The generative AI module in the server analyzes the text data and understands the user's intent. Specifically, it extracts the positive context of "the weather is nice."

[0401] Step 5:

[0402] The server uses the generative AI module to generate an appropriate reaction, for example, "Yes! It's really nice weather today!" This reaction is intended to be a positive response to the user's comment.

[0403] Step 6:

[0404] The server uses a response timing calculation module to calculate the optimal timing for the reaction. Based on the speech analysis, it determines the timing to start the response, for example, 0.5 seconds later.

[0405] Step 7:

[0406] The server converts the generated text reaction into speech data using a speech generation module, where the generated speech has an appropriate emotional tone (in this example, a tone of joy).

[0407] Step 8:

[0408] The server sends the generated voice data to the terminal, which then receives the voice data via the network.

[0409] Step 9:

[0410] The device stores the received voice data in an internal buffer and plays it back through the speaker after the 0.5 second interval calculated based on the response timing has elapsed.

[0411] Step 10:

[0412] The device will play a voice message over the speaker saying, "Yes! It's really nice weather today!" This allows the user to experience a natural conversation.

[0413] Step 11:

[0414] The user receives a response from the voice assistant and can continue the conversation. The system waits for a new utterance.

[0415] In this way, the system automatically captures and analyzes the user's speech, generating natural reactions at the appropriate times and responding as voice, allowing users to enjoy natural, stress-free conversations.

[0416] Example 1

[0417] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0418] Conventional voice assistant systems often lack natural dialogue with users, particularly insufficient response timing and emotional expression. This leads to lower user satisfaction and a lower rate of continued use of the system. The goal of this invention is to achieve more natural, human-like dialogue and increase user satisfaction.

[0419] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0420] In this invention, the server includes a generation means for generating appropriate reaction content using a generation AI model based on the speech recognition result, a timing calculation means for calculating the response timing of the generated reaction, and a voice generation means for generating voice data based on the reaction content and response timing. This enables natural and smooth dialogue by analyzing and responding to the user's speech in real time and expressing emotions appropriately.

[0421] "User" refers to the individual or end user who uses the voice assistant system.

[0422] "Voice data" refers to data that records what a user says as a digital signal.

[0423] "Voice analysis means" refers to equipment or software for capturing a user's voice data and performing voice recognition.

[0424] A "generative AI model" refers to an artificial intelligence model that uses machine learning and neural network technology to generate optimal output for input data.

[0425] "Generation means" refers to a device or program that has the function of generating appropriate reaction content using a generative AI model based on the voice recognition results.

[0426] "Response timing" refers to the time interval between a user's utterance and the system's response.

[0427] "Timing calculation means" refers to a device or program that has the function of calculating the response timing of the generated reaction.

[0428] "Speech generation means" refers to a device or program that has the function of generating voice data based on the generated reaction content and response timing.

[0429] "Playback means" refers to equipment or software for playing back the generated audio data to the user.

[0430] "System" refers to the entire voice assistant, including the above means, that enables natural dialogue with the user.

[0431] The present invention relates to a voice assistant system that uses generation AI to automatically generate reaction content, response timing, and tone of voice in order to realize natural dialogue with users. This system mainly includes the following components:

[0432] 1. Audio analysis methods:

[0433] The device has the ability to capture the user's voice data and perform voice recognition in real time. This typically uses the device's built-in microphone and a voice recognition engine (such as a general voice recognition API). For example, if a user says, "The weather is nice today," the device captures this speech and generates "The weather is nice today" as text data.

[0434] 2. Generation means:

[0435] The system has the ability to generate appropriate reactions using a generative AI model based on the text data received by the server. This uses a generative AI model (e.g., a widely used generative AI technology). The server analyzes the user's speech and generates emotional expressions according to the context. For example, it generates a positive reaction such as, "Yes! It's really nice weather today!"

[0436] 3. Timing calculation method:

[0437] The server has a function to calculate the appropriate response timing based on the reaction content generated by the generation means. For example, it calculates the timing so that the response starts about 0.5 seconds after the user finishes speaking. For this, a timing calculation module for timing control within the program is used.

[0438] 4. Sound Generation Means:

[0439] The server has the function of generating reaction content as voice using voice synthesis technology based on information from the generation means and timing calculation means. In particular, it is possible to generate natural dialogue voice by adding emotions and tone of voice according to the reaction. This implementation uses a voice synthesis API (e.g., a widely used voice synthesis technology). Specifically, voice is generated by adding a tone of joy.

[0440] 5. Regeneration means:

[0441] The device has a function to play back the voice data sent from the server to the user. This playback means allows the user to experience a natural conversation. The voice is output using the device's built-in speaker.

[0442] Specific examples

[0443] For example, the following prompt sentence is input to a generative AI model:

[0444] "When a user says, 'The weather is nice today,' generate an appropriate response."

[0445] Given this input, the generator AI will output the following:

[0446] "Yes! It's really beautiful weather today!"

[0447] Based on the reactions generated in this way, the server calculates the response timing and generates voice data.Finally, the device receives the generated voice data and plays it back to the user at the appropriate time.

[0448] This allows users to have a natural and smooth conversation with the voice assistant. The system of the present invention provides a natural conversation environment by automatically processing each step, without requiring user operation.

[0449] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0450] Step 1:

[0451] The user says, "The weather is nice today."

[0452] Specific operation: The device's built-in microphone captures the user's speech.

[0453] Input: User's voice

[0454] Output: Audio data captured within the device

[0455] Step 2:

[0456] The voice data captured by the terminal is converted into text data using a voice recognition engine.

[0457] Specific operation: The device calls the Google Speech-to-Text API or similar and converts the voice data into text data such as "The weather is nice today."

[0458] Input: Audio data

[0459] Output: Text data "The weather is nice today."

[0460] Step 3:

[0461] The terminal transmits the generated text data to the server.

[0462] Specific operation: The terminal generates an HTTP request and sends text data to the server.

[0463] Input: Text data "The weather is nice today"

[0464] Output: Text data received by the server

[0465] Step 4:

[0466] The server uses a generative AI model to generate appropriate reaction content based on the text data received.

[0467] How it works: The server uses a generative AI model such as OpenAI GPT-4 to input text data as a prompt and generate a response text such as "Yes! It's really nice weather today!"

[0468] Input: Text data "The weather is nice today"

[0469] Output: Response text "Yes! It's really nice weather today!"

[0470] Step 5:

[0471] The server calculates the appropriate response timing based on the generated response text.

[0472] Specific operation: The server uses the timing calculation module to calculate the timing to respond 0.5 seconds after the user finishes speaking.

[0473] Input: Response text "Yes! It's really beautiful weather today!"

[0474] Output: Response timing (e.g., after 0.5 seconds)

[0475] Step 6:

[0476] The server generates voice data using a voice synthesis technique based on the response text and response timing.

[0477] Specific operation: The server calls the Google Cloud Text-to-Speech API or similar and generates the response text as voice data with a joyful tone added.

[0478] Input: Response text "Yes! It's really nice weather today!", Response timing

[0479] Output: Audio data

[0480] Step 7:

[0481] The terminal receives the generated voice data and plays it back to the user based on the calculated response timing.

[0482] Specific operation: The device stores the voice data in a buffer, checks the response timing, and then plays the voice through the built-in speaker.

[0483] Input: Voice data, response timing

[0484] Output: Audio output to the user

[0485] (Application example 1)

[0486] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0487] The goal is to provide an assistant system that makes interactions with customers in physical stores more natural and effective. In particular, when customers ask questions about products, it is required to provide smooth and easy-to-understand information. Conventional systems have difficulty increasing customer satisfaction and purchasing motivation because their responses to questions are unnatural and the information they provide is inaccurate.

[0488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0489] In this invention, the server includes a voice analysis means for capturing user voice data and performing voice recognition, a generation means for generating appropriate reaction content based on the voice recognition result, a timing calculation means for calculating the response timing of the generated reaction, a voice generation means for generating voice data based on the reaction content and the response timing, a playback means for playing back the generated voice data to the user, an information provision means for generating appropriate product information in response to the user's question, and a voice information playback means for playing back the information generated by the information provision means as voice, thereby enabling natural and quick responses to customer questions and the provision of accurate product information.

[0490] The "voice analysis means" is a means for capturing the user's voice data and performing voice recognition.

[0491] The "generation means" is a means for generating appropriate reaction content based on the speech recognition result.

[0492] The "timing calculation means" is a means for calculating the response timing of the generated reaction.

[0493] The "voice generating means" is a means for generating voice data based on the reaction content and response timing.

[0494] The "playback means" is a means for playing back the generated audio data to the user.

[0495] The "information providing means" is a means for generating appropriate product information in response to a user's question.

[0496] The "audio information reproducing means" is a means for reproducing the information generated by the information providing means as audio.

[0497] This embodiment will be described as a specific example of a voice assistant system that realizes natural conversations in a brick-and-mortar store.

[0498] The system uses smart glasses to support natural interactions between customers and store staff. Its main functions include voice analysis, reaction generation, timing calculation, voice generation, product information provision, and voice playback.

[0499] The server first uses a voice analysis means to capture the voice data sent from the smart glasses and performs voice recognition in real time. This voice recognition engine uses Google Cloud Speech-to-Text. Next, a generation means generates an appropriate reaction content based on the voice recognition results. This generation means uses OpenAI's GPT-4. The generated text data is passed to a timing calculation means to calculate the response timing of the reaction. Then, based on the calculated appropriate response timing, a voice generation means converts the reaction content into voice data. Google Cloud Text-to-Speech is used to generate this voice data. Finally, the generated voice data is played back to the user through the playback means of the smart glasses.

[0500] Furthermore, when a user asks a question about a product, the information providing means generates appropriate product information and plays it back as voice. Through this series of processes, customers can receive a quick and accurate response to their question, improving their shopping experience in physical stores.

[0501] As a concrete example, consider a case where a customer asks, "What is this item made of?" in a store. The voice assistant on the smart glasses captures this question, and Google Cloud Speech-to-Text converts it into text: "What is this item made of?" OpenAI GPT-4 then generates the appropriate response: "This item is made of 100% cotton." Google Cloud Text-to-Speech then converts this response into audio, which is played back to the customer through the smart glasses.

[0502] The following text is used as an example of a prompt sentence:

[0503] "A customer asks about the material of an item. Generate a natural response."

[0504] In this way, this system combines the functions of voice recognition, generative AI, timing calculation, voice generation, and information provision to enable natural conversations in physical stores.

[0505] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0506] Step 1:

[0507] The device (smart glasses) captures the user's voice. The input is the user's voice data, and the output is the captured voice data. Specifically, the microphone in the smart glasses records the user's speech in real time.

[0508] Step 2:

[0509] The device sends the captured voice data to the server, where speech recognition is performed using a speech analysis tool. The input is voice data, and the output is the speech recognition result (text data). Specifically, Google Cloud Speech-to-Text analyzes the voice data and generates text data such as "What is the material of this product?"

[0510] Step 3:

[0511] The server's generation means receives the speech recognition results (text data) and uses generation AI to generate an appropriate reaction. The input is text data, and the output is the generated reaction (text). Specifically, OpenAI's GPT-4 generates a response such as "This product is made of 100% cotton."

[0512] Step 4:

[0513] The server's timing calculation means calculates the response timing of the generated reaction. The input is the generated reaction content (text), and the output is the response timing (time interval). Specifically, it calculates the appropriate response timing (e.g., 0.5 seconds) after the user's utterance ends.

[0514] Step 5:

[0515] The server's voice generation means generates voice data based on the reaction content and response timing. The input is the generated reaction content (text) and response timing, and the output is voice data. Specifically, Google Cloud Text-to-Speech generates voice data saying, "This product is made of 100% cotton."

[0516] Step 6:

[0517] The server sends the generated voice data to the terminal. The input is the voice data, and the output is the completion of sending the voice data to the terminal. Specifically, the generated voice data is sent to the smart glasses via the network.

[0518] Step 7:

[0519] The device plays the generated voice data to the user based on the calculated response timing. The input is the voice data and the response timing, and the output is the voice playback. Specifically, the speaker on the smart glasses plays the voice, "This product is made of 100% cotton."

[0520] This series of steps allows users to get answers to their questions through natural dialogue.

[0521] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0522] This embodiment relates to a voice assistant system that automatically generates reaction content, response timing, and tone of voice using a generation AI and an emotion engine to realize natural dialogue with a user. This system mainly includes the following components:

[0523] 1. Audio analysis methods:

[0524] This is a function that allows the device to capture the user's voice data and perform speech recognition in real time. This allows the user's voice to be converted into text data. For example, if the user says, "The weather is nice today," the device will capture this utterance and generate the text data, "The weather is nice today."

[0525] 2. Emotion Engine:

[0526] This function recognizes the user's emotions by analyzing the text and voice data obtained by the server using voice analysis. This emotion engine determines the user's emotional state by taking into account not only the content of the speech but also the tone and pace of the voice.

[0527] 3. Generation means:

[0528] This function uses a generative AI to generate appropriate reactions based on the text data received by the server and the emotional data recognized by the emotion engine. The generative AI compares the content of the user's utterance with their emotions and generates an emotional expression that matches the context (e.g., a positive reaction such as "That's right! The weather is really nice today!").

[0529] 4. Timing calculation method:

[0530] This function calculates the appropriate response timing based on the reaction content generated by the server. For example, it calculates the timing to start the response approximately 0.5 seconds after the user finishes speaking.

[0531] 5. Sound Generation Means:

[0532] This function uses speech synthesis technology to generate the reaction content as voice based on the information from the generation means and timing calculation means. The emotions recognized by the emotion engine are also reflected. Therefore, the generated voice contains an appropriate emotional tone (in this example, a tone of joy).

[0533] 6. Regeneration means:

[0534] This function allows the terminal to play back the voice data sent from the server to the user. This playback function allows the user to experience natural conversation.

[0535] Program processing

[0536] The specific program processing is performed as follows: Below is a specific example of how the system responds to user utterances.

[0537] 1. The user says, "The weather is nice today."

[0538] 2. The device captures the user's voice and uses a speech recognition engine to generate text data such as "The weather is nice today."

[0539] 3. The device sends the generated text data to the back-end server, where it arrives via the network.

[0540] 4. The server receives the text data and analyzes the user's emotions using an emotion engine that works in conjunction with the generative AI. For example, the user's emotions are recognized as positive.

[0541] 5. The server uses the generative AI to generate an appropriate reaction. Based on the analysis results of the emotion engine, a response such as "Yes! It's really nice weather today!" is generated. This reaction corresponds to the user's emotions.

[0542] 6. The server uses the response timing calculation module to calculate an appropriate interval for the response (e.g., 0.5 seconds).

[0543] 7. The server uses the speech generation module to generate a reaction, "Yes! It's really nice weather today!", with a joyful tone added.

[0544] 8. The device receives the generated voice data and plays it back to the user based on the calculated response timing.

[0545] 9. The device will play "Yes! It's really nice weather today!" over the speaker, allowing the user to experience a natural conversation.

[0546] In this way, the system automatically performs a series of processes, from capturing and analyzing the user's speech to generating natural reactions at the appropriate time and responding as voice. This allows users to enjoy natural, stress-free conversations. The system can also achieve more accurate conversations by using call center recordings and other call data as training data.

[0547] The processing flow will be explained below.

[0548] Step 1:

[0549] The user speaks to the voice assistant. Specifically, the user says, "The weather is nice today."

[0550] Step 2:

[0551] The device captures the user's voice with a microphone and temporarily stores it as voice data. At the same time, the device's voice recognition engine activates and converts the voice data into text data in real time. For example, the text generated might be, "The weather is nice today."

[0552] Step 3:

[0553] The terminal sends the generated text data to the back-end server, where it arrives via the network.

[0554] Step 4:

[0555] The server receives the text data and begins analyzing it. As a preprocessing step for the generative AI, the text data is analyzed based on the context.

[0556] Step 5:

[0557] The server uses an emotion engine to analyze the user's emotions from the voice and text data, for example, recognizing that the user is in a positive emotion from the tone of voice and pace of speech.

[0558] Step 6:

[0559] The server uses generative AI to generate an appropriate reaction, taking into account the emotional data obtained from the emotion engine. Specifically, a positive reaction such as "Yes! It's really nice weather today!" is generated.

[0560] Step 7:

[0561] The server uses a response timing calculation module to calculate the optimal response timing for the generated reaction. It detects the end of the user's utterance and determines the timing to start the response, for example, 0.5 seconds later.

[0562] Step 8:

[0563] The server uses a voice generation module to convert the generated reaction text into voice data, which reflects emotional expressions such as a tone of joy.

[0564] Step 9:

[0565] The server sends the generated voice data to the terminal, which then receives the voice data via the network.

[0566] Step 10:

[0567] The device stores the received audio data in an internal buffer and waits until the calculated response timing. After a 0.5 second delay, the device prepares to play the audio data.

[0568] Step 11:

[0569] The device will play the voice "Yes! It's really nice weather today!" through the speaker, allowing the user to experience a natural and emotional interaction.

[0570] Step 12:

[0571] The user receives the voice assistant's response and returns to a system state where they can continue the conversation. The system waits for a new utterance.

[0572] In this way, the system automatically captures and analyzes the user's speech, recognizes their emotions, generates natural reactions at the appropriate time, and responds in voice, allowing users to enjoy natural, stress-free conversations.

[0573] Example 2

[0574] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0575] Conventional voice assistant systems often simply process user speech as text data and then react to it. This makes it difficult to understand and respond to the user's emotions and context, making it difficult to achieve natural dialogue with the user. Furthermore, the tone and timing of the generated voice can be mechanical, making the responses to the user seem unnatural.

[0576] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an analysis means for capturing user voice data and performing voice recognition, an analysis means for recognizing the user's emotion based on the voice recognition result and the voice data, a generation means for generating appropriate reaction content based on the analyzed emotion data and the voice recognition result, a calculation means for calculating the response timing of the generated reaction, a generation means for generating voice data based on the reaction content and response timing, and a playback means for playing the generated voice data to the user. This makes it possible to automatically generate natural reactions that take into account the user's utterance content and emotion, and play them as voice at appropriate timing.

[0577] "User" refers to an individual who uses this system to provide voice input.

[0578] "Voice Data" means information that is a digital recording of a user's speech.

[0579] "Analysis Means" refers to technical equipment or software for capturing audio data and performing voice recognition and emotion analysis.

[0580] "Text data" refers to character information converted from voice data by voice recognition.

[0581] "Emotion Engine" refers to a technical device or software for analyzing text and voice data to determine a user's emotional state.

[0582] "Generation means" refers to a technical device or software for automatically creating appropriate reaction content based on the analysis results of the emotion engine and the voice recognition results.

[0583] "Calculation means" refers to a technical device or software for calculating response timing based on reaction content.

[0584] "Generative AI" refers to artificial intelligence technology that generates appropriate reactions based on the user's speech and emotions.

[0585] A "prompt sentence" is an instruction sentence input to the generation AI that provides context for reaction generation.

[0586] "Speech generation means" refers to technical devices or software for generating reactions as synthetic voices based on text data and emotion data.

[0587] "Playback means" refers to a technical device for playing back the generated audio data in a form audible to the user.

[0588] This invention relates to a voice assistant system that analyzes a user's voice data, recognizes emotions, generates appropriate reactions, and provides the reactions to the user as voice. This system uses a generative AI model and an emotion engine to realize natural dialogue with the user.

[0589] 1. Audio analysis methods:

[0590] The device captures the user's voice data and performs real-time speech recognition. Specific speech recognition software used includes Google Speech-to-Text API and Amazon Transcribe. This speech analysis method converts the user's speech into text data. For example, if a user says, "The weather is nice today," the device captures this speech and generates the text data, "The weather is nice today."

[0591] 2. Emotion Engine:

[0592] The server analyzes the text and voice data obtained through the voice analysis tool to recognize the user's emotions. Specifically, emotion analysis software such as IBM Watson Tone Analyzer is used. This emotion engine determines the user's emotional state by taking into account not only the content of the speech but also the tone and pace of the voice.

[0593] 3. Generation means:

[0594] Based on the text data received by the server and the emotional data recognized by the emotion engine, an appropriate reaction is generated using a generative AI. The specific generative AI used includes OpenAI GPT-3. This generator compares the user's speech content with their emotions to generate emotional expressions that fit the context. For example, if the prompt text is input as "The user is in a positive mood and talking about the weather. Please generate an appropriate response," the response generated will be "Yes! The weather is really nice today!"

[0595] 4. Timing calculation method:

[0596] The server calculates an appropriate response timing based on the reaction content generated by the generation means. For example, a software module is provided to calculate the timing so that the response starts with a time interval of about 0.5 seconds after the user finishes speaking.

[0597] 5. Sound Generation Means:

[0598] The server uses speech synthesis technology based on the information from the generation method and timing calculation method to generate the reaction content as voice. Specific speech synthesis technologies used include Google Text-to-Speech API and Amazon Polly. The emotional tone recognized by the emotion engine is reflected in the voice. For example, the generated voice may include a tone of joy.

[0599] 6. Regeneration means:

[0600] The terminal reproduces the voice data sent from the server to the user, allowing the user to experience natural conversation.

[0601] By combining the above methods, the system can capture and analyze user utterances, and automatically generate natural reactions and respond as voice at the appropriate time. This allows users to enjoy natural, stress-free conversations. The system can also achieve higher accuracy by using call center recordings and other call data as training data.

[0602] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0603] Step 1:

[0604] The user inputs voice data. For example, the user might say, "The weather is nice today." This voice data becomes the input.

[0605] Step 2:

[0606] The device captures the user's voice data and converts it into text data using a speech recognition engine (e.g., Google Speech-to-Text API). The input is the user's voice data, and the output is the text data generated by speech recognition: "The weather is nice today."

[0607] Step 3:

[0608] The terminal sends the generated text data to the back-end server. The input is text data, and the output reaches the server via the network.

[0609] Step 4:

[0610] Based on the text data received by the server, an emotion engine (e.g., IBM Watson Tone Analyzer) is used to analyze the user's emotions. The input is text data, and "positive emotions" are recognized through data analysis. The output is the analyzed emotion data.

[0611] Step 5:

[0612] The server uses a generative AI (e.g., OpenAI GPT-3) to generate an appropriate reaction based on emotional and text data. The input is emotional and text data, and the prompt is, "The user is talking about the weather in a positive mood. Please generate an appropriate response." The output is the response generated by the generative AI: "Yes! The weather is really nice today!"

[0613] Step 6:

[0614] The server uses a response timing calculation module to calculate the appropriate interval (e.g., 0.5 seconds) for the response based on the reaction content. The input is the reaction content, and the output is the calculated response timing.

[0615] Step 7:

[0616] The server uses a speech generation module (e.g., Google Text-to-Speech API) to generate the reaction as speech data. The input is the text data "Yes! It's really nice weather today!" and an emotional tone (a tone of joy), and the output is the generated speech data.

[0617] Step 8:

[0618] The terminal receives the voice data sent from the server and plays it back to the user based on the calculated response timing. The input is the voice data and the response timing, and the output is audible voice for the user.

[0619] Step 9:

[0620] The device will play a voice message over the speaker saying, "Yes! It's really nice weather today!" This action allows the user to experience a natural conversation.

[0621] (Application example 2)

[0622] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0623] There is a demand for voice assistant systems that can provide more natural and effective customer service in brick-and-mortar stores. Conventional voice assistant systems have issues in that they are unable to fully achieve functions such as natural conversation that reflects the customer's emotions, or providing store guidance, product explanations, and promotional information.

[0624] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0625] In this invention, the server includes a voice analysis means for capturing user voice data and performing voice recognition, a generation means for generating appropriate reaction content based on the voice recognition result and the analyzed emotional state, a timing calculation means for calculating the response timing of the generated reaction, a voice generation means for generating voice data based on the reaction content and the response timing, a playback means for playing back the generated voice data to the user, and an information provision means for providing product information, store guidance, and promotion information. This allows for natural dialogue with customers and makes it possible to effectively provide in-store guidance, product explanations, and promotion information.

[0626] The "voice analysis means" is a function for capturing the user's voice data and performing voice recognition in real time.

[0627] The "generation means" is a function that generates appropriate reaction content based on the voice recognition results and the analyzed emotional state.

[0628] The "timing calculation means" is a function that calculates the response timing of the generated reaction.

[0629] The "voice generating means" is a function that generates voice data based on the reaction content and response timing.

[0630] The "playback means" is a function that plays back the generated audio data to the user.

[0631] "Information providing means" is a function that provides information about products, store guides, and promotional information.

[0632] To implement this invention, a specific system configuration and program are required. The hardware and software for realizing this system, as well as the methods for processing and calculating data, will be described below.

[0633] System Overview

[0634] The system has the following main components:

[0635] 1. Voice analysis means: Captures the user's voice data and performs voice recognition.

[0636] 2. Generation method: Generate appropriate reaction content based on the speech recognition results and the analyzed emotional state.

[0637] 3. Timing calculation means: Calculate the response timing of the generated reaction.

[0638] 4. Voice generation means: Generates voice data based on the reaction content and response timing.

[0639] 5. Playback means: Plays the generated audio data to the user.

[0640] 6. Information provision means: Providing product information, store guides, and promotional information.

[0641] Hardware and software used

[0642] Microphones and speakers: These are built into smartphones, smart glasses, and head-mounted displays.

[0643] Cloud servers: Used to run data analysis and generative AI models.

[0644] Generative AI models: Uses open-source GPT-3 and Azure Cognitive Services.

[0645] Emotion engine: Uses IBM Watson Tone Analyzer.

[0646] Speech recognition and synthesis: Uses Google Speech-to-Text and Amazon Polly.

[0647] Program processing flow

[0648] The program of this system operates as follows.

[0649] First, the user speaks to the voice assistant, and the voice data is captured through a microphone and converted into text data using a speech recognition method. Specifically, Google Speech-to-Text is used to convert voice to text.

[0650] This text data is then sent to a cloud server where an emotion engine analyzes the user's emotions, for example, using IBM Watson Tone Analyzer to determine whether the user is expressing positive emotions.

[0651] Based on the analyzed emotional state, the generator uses GPT-3 to generate an appropriate reaction, such as "This product uses new technology and is very durable."

[0652] Next, the timing calculation means calculates the response timing of approximately 0.5 seconds, and the voice generation means converts the generated reaction into voice data using Amazon Polly.

[0653] Finally, the playback means plays back the audio data through a speaker, allowing the user to experience natural dialogue.

[0654] Usage example

[0655] For example, consider a situation where a customer asks a voice assistant, "Tell me about the latest promotions." Here's a sample interaction:

[0656] Specific examples

[0657] 1. User: "Tell me about the latest promotions."

[0658] 2. Voice Capture: "What are the latest promotions?"

[0659] 3. Speech recognition: Generate text data for "Tell me the latest promotion information"

[0660] 4. Data Transmission

[0661] 5. Sentiment Analysis: Recognizing User Interest

[0662] 6. Reaction Generation: "These new shoes are currently 20% off!"

[0663] 7. Response timing calculation: approx. 0.5 seconds

[0664] 8. Speech synthesis: Convert "These new shoes are currently 20% off!" into speech data

[0665] 9. Audio playback

[0666] Prompt Sentence Examples

[0667] Customer: "Tell me about this product."

[0668] Analysis results:

[0669] Emotion: Interested

[0670] Product information: New product, durable, priced at $50

[0671] Prompt the generative AI model:

[0672] Input text: Please explain what happens when a customer expresses interest in a product that uses new technology.

[0673] Output: "This product uses the latest technology and is extremely durable. It costs $50."

[0674] In this way, the system can realize natural dialogue with customers and effectively provide in-store guidance, product explanations, and promotional information.

[0675] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0676] Step 1:

[0677] The user speaks to the voice assistant. The input is the user's voice data, and the specific action is the user saying, "Tell me about this product."

[0678] Step 2:

[0679] The device captures the user's voice data and performs speech recognition. The input is the user's voice data and the output is text data. The speech "Tell me about this product" is converted into text "Tell me about this product."

[0680] Step 3:

[0681] The device sends the generated text data to the cloud server. The input is text data, and the output is data transmission to the server.

[0682] Step 4:

[0683] The server receives the text data and uses the emotion engine to analyze the user's emotions. The input is text data, and the output is analyzed emotion data. Specifically, the text "Tell me about this product" is sent, and the emotion engine analyzes it as "Interested."

[0684] Step 5:

[0685] The server uses a generative AI model to generate appropriate reaction content based on the analyzed emotional state and text data. The input is text data and emotional data, and the output is the generated reaction content. Specifically, in response to the question "Tell me about this product," the server generates the response "This product uses the latest technology and is extremely durable."

[0686] Step 6:

[0687] The server uses the response timing calculation means to calculate the response timing of the generated reaction. The input is the generated reaction content, and the output is response timing information. Specifically, the server calculates the response timing of approximately 0.5 seconds.

[0688] Step 7:

[0689] The server uses a voice generation means to convert the generated reaction into voice data. The input is the generated reaction content and response timing information, and the output is voice data. Specifically, the text "This product uses the latest technology and is very durable" is converted into voice data.

[0690] Step 8:

[0691] The device plays back the audio data sent from the server. The input is audio data, and the output is audio playback to the user. Specifically, the device plays back through the speaker the audio, "This product uses the latest technology and is extremely durable."

[0692] This series of processing steps enables the system to have natural dialogue with the user, and effectively provide in-store guidance, product explanations, and promotional information.

[0693] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0694] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0695] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0696] [Third embodiment]

[0697] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0698] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0699] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0700] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0701] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0702] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0703] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0704] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0705] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0706] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0707] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0708] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0709] This embodiment relates to a voice assistant system that uses a generation AI to automatically generate reaction content, response timing, and tone of voice in order to realize natural dialogue with a user. This system mainly includes the following components:

[0710] 1. Audio analysis methods:

[0711] This is a function that allows the device to capture the user's voice data and perform speech recognition in real time. This allows the user's voice to be converted into text data. For example, if the user says, "The weather is nice today," the device will capture this utterance and generate the text data, "The weather is nice today."

[0712] 2. Generation means:

[0713] This function uses a generative AI to generate appropriate reactions based on the text data received by the server. The generative AI analyzes the user's speech and generates emotional expressions according to the context (e.g., a positive reaction such as "That's right! The weather is really nice today!").

[0714] 3. Timing calculation method:

[0715] This function calculates the appropriate response timing based on the reaction content generated by the server. For example, it calculates the timing to start the response approximately 0.5 seconds after the user finishes speaking.

[0716] 4. Sound Generation Means:

[0717] This function uses speech synthesis technology to generate reaction content as voice based on information from the generation method and timing calculation method. By adding tone of voice and emotion according to the reaction, natural dialogue voice can be generated. For example, if the reaction is "That's right! It's really nice weather today!", a tone of joy will be added to the generated voice.

[0718] 5. Regeneration means:

[0719] This function allows the terminal to play back the voice data sent from the server to the user. This playback function allows the user to experience natural conversation.

[0720] Program processing

[0721] The specific program processing is performed as follows: Below is a specific example of how the system responds to user utterances.

[0722] 1. The user says, "The weather is nice today."

[0723] 2. The device captures the user's voice and uses a speech recognition engine to generate text data such as "The weather is nice today."

[0724] 3. The device sends the generated text data to the server.

[0725] 4. The server receives the text data and uses generative AI to generate the most appropriate reaction for the context: "Yes! It's really nice weather today!"

[0726] 5. The server uses the response timing calculation module to calculate an appropriate interval for the response (e.g., 0.5 seconds).

[0727] 6. The server uses the speech generation module to generate a reaction, "Yes! It's really nice weather today!", with a joyful tone added.

[0728] 7. The device receives the generated voice data and plays it back to the user based on the calculated response timing.

[0729] This allows the user to experience a natural and smooth conversation with the voice assistant. The system of this embodiment automatically generates reactions, calculates response timing, and generates voices, so it can provide a natural dialogue environment without requiring user operation. The system can also achieve more accurate dialogue by using call center recording data and other call data as training data.

[0730] The processing flow will be explained below.

[0731] Step 1:

[0732] The user speaks to the voice assistant. Specifically, the user says, "The weather is nice today."

[0733] Step 2:

[0734] The device captures the user's voice with a microphone and temporarily stores it as voice data. At the same time, the device's voice recognition engine activates and converts the voice data into text data in real time. For example, the text generated might be, "The weather is nice today."

[0735] Step 3:

[0736] The terminal sends the generated text data to the back-end server, where it arrives via the network.

[0737] Step 4:

[0738] The server receives the text data. The generative AI module in the server analyzes the text data and understands the user's intent. Specifically, it extracts the positive context of "the weather is nice."

[0739] Step 5:

[0740] The server uses the generative AI module to generate an appropriate reaction, for example, "Yes! It's really nice weather today!" This reaction is intended to be a positive response to the user's comment.

[0741] Step 6:

[0742] The server uses a response timing calculation module to calculate the optimal timing for the reaction. Based on the speech analysis, it determines the timing to start the response, for example, 0.5 seconds later.

[0743] Step 7:

[0744] The server converts the generated text reaction into speech data using a speech generation module, where the generated speech has an appropriate emotional tone (in this example, a tone of joy).

[0745] Step 8:

[0746] The server sends the generated voice data to the terminal, which then receives the voice data via the network.

[0747] Step 9:

[0748] The device stores the received voice data in an internal buffer and plays it back through the speaker after the 0.5 second interval calculated based on the response timing has elapsed.

[0749] Step 10:

[0750] The device will play a voice message over the speaker saying, "Yes! It's really nice weather today!" This allows the user to experience a natural conversation.

[0751] Step 11:

[0752] The user receives a response from the voice assistant and can continue the conversation. The system waits for a new utterance.

[0753] In this way, the system automatically captures and analyzes the user's speech, generating natural reactions at the appropriate times and responding as voice, allowing users to enjoy natural, stress-free conversations.

[0754] Example 1

[0755] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0756] Conventional voice assistant systems often lack natural dialogue with users, particularly insufficient response timing and emotional expression. This leads to lower user satisfaction and a lower rate of continued use of the system. The goal of this invention is to achieve more natural, human-like dialogue and increase user satisfaction.

[0757] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0758] In this invention, the server includes a generation means for generating appropriate reaction content using a generation AI model based on the speech recognition result, a timing calculation means for calculating the response timing of the generated reaction, and a voice generation means for generating voice data based on the reaction content and response timing. This enables natural and smooth dialogue by analyzing and responding to the user's speech in real time and expressing emotions appropriately.

[0759] "User" refers to the individual or end user who uses the voice assistant system.

[0760] "Voice data" refers to data that records what a user says as a digital signal.

[0761] "Voice analysis means" refers to equipment or software for capturing a user's voice data and performing voice recognition.

[0762] A "generative AI model" refers to an artificial intelligence model that uses machine learning and neural network technology to generate optimal output for input data.

[0763] "Generation means" refers to a device or program that has the function of generating appropriate reaction content using a generative AI model based on the voice recognition results.

[0764] "Response timing" refers to the time interval between a user's utterance and the system's response.

[0765] "Timing calculation means" refers to a device or program that has the function of calculating the response timing of the generated reaction.

[0766] "Speech generation means" refers to a device or program that has the function of generating voice data based on the generated reaction content and response timing.

[0767] "Playback means" refers to equipment or software for playing back the generated audio data to the user.

[0768] "System" refers to the entire voice assistant, including the above means, that enables natural dialogue with the user.

[0769] The present invention relates to a voice assistant system that uses generation AI to automatically generate reaction content, response timing, and tone of voice in order to realize natural dialogue with users. This system mainly includes the following components:

[0770] 1. Audio analysis methods:

[0771] The device has the ability to capture the user's voice data and perform voice recognition in real time. This typically uses the device's built-in microphone and a voice recognition engine (such as a general voice recognition API). For example, if a user says, "The weather is nice today," the device captures this speech and generates "The weather is nice today" as text data.

[0772] 2. Generation means:

[0773] The system has the ability to generate appropriate reactions using a generative AI model based on the text data received by the server. This uses a generative AI model (e.g., a widely used generative AI technology). The server analyzes the user's speech and generates emotional expressions according to the context. For example, it generates a positive reaction such as, "Yes! It's really nice weather today!"

[0774] 3. Timing calculation method:

[0775] The server has a function to calculate the appropriate response timing based on the reaction content generated by the generation means. For example, it calculates the timing so that the response starts about 0.5 seconds after the user finishes speaking. For this, a timing calculation module for timing control within the program is used.

[0776] 4. Sound Generation Means:

[0777] The server has the function of generating reaction content as voice using voice synthesis technology based on information from the generation means and timing calculation means. In particular, it is possible to generate natural dialogue voice by adding emotions and tone of voice according to the reaction. This implementation uses a voice synthesis API (e.g., a widely used voice synthesis technology). Specifically, voice is generated by adding a tone of joy.

[0778] 5. Regeneration means:

[0779] The device has a function to play back the voice data sent from the server to the user. This playback means allows the user to experience a natural conversation. The voice is output using the device's built-in speaker.

[0780] Specific examples

[0781] For example, the following prompt sentence is input to a generative AI model:

[0782] "When a user says, 'The weather is nice today,' generate an appropriate response."

[0783] Given this input, the generator AI will output the following:

[0784] "Yes! It's really beautiful weather today!"

[0785] Based on the reactions generated in this way, the server calculates the response timing and generates voice data.Finally, the device receives the generated voice data and plays it back to the user at the appropriate time.

[0786] This allows users to have a natural and smooth conversation with the voice assistant. The system of the present invention provides a natural conversation environment by automatically processing each step, without requiring user operation.

[0787] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0788] Step 1:

[0789] The user says, "The weather is nice today."

[0790] Specific operation: The device's built-in microphone captures the user's speech.

[0791] Input: User's voice

[0792] Output: Audio data captured within the device

[0793] Step 2:

[0794] The voice data captured by the terminal is converted into text data using a voice recognition engine.

[0795] Specific operation: The device calls the Google Speech-to-Text API or similar and converts the voice data into text data such as "The weather is nice today."

[0796] Input: Audio data

[0797] Output: Text data "The weather is nice today."

[0798] Step 3:

[0799] The terminal transmits the generated text data to the server.

[0800] Specific operation: The terminal generates an HTTP request and sends text data to the server.

[0801] Input: Text data "The weather is nice today"

[0802] Output: Text data received by the server

[0803] Step 4:

[0804] The server uses a generative AI model to generate appropriate reaction content based on the text data received.

[0805] How it works: The server uses a generative AI model such as OpenAI GPT-4 to input text data as a prompt and generate a response text such as "Yes! It's really nice weather today!"

[0806] Input: Text data "The weather is nice today"

[0807] Output: Response text "Yes! It's really nice weather today!"

[0808] Step 5:

[0809] The server calculates the appropriate response timing based on the generated response text.

[0810] Specific operation: The server uses the timing calculation module to calculate the timing to respond 0.5 seconds after the user finishes speaking.

[0811] Input: Response text "Yes! It's really beautiful weather today!"

[0812] Output: Response timing (e.g., after 0.5 seconds)

[0813] Step 6:

[0814] The server generates voice data using a voice synthesis technique based on the response text and response timing.

[0815] Specific operation: The server calls the Google Cloud Text-to-Speech API or similar and generates the response text as voice data with a joyful tone added.

[0816] Input: Response text "Yes! It's really nice weather today!", Response timing

[0817] Output: Audio data

[0818] Step 7:

[0819] The terminal receives the generated voice data and plays it back to the user based on the calculated response timing.

[0820] Specific operation: The device stores the voice data in a buffer, checks the response timing, and then plays the voice through the built-in speaker.

[0821] Input: Voice data, response timing

[0822] Output: Audio output to the user

[0823] (Application example 1)

[0824] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0825] The goal is to provide an assistant system that makes interactions with customers in physical stores more natural and effective. In particular, when customers ask questions about products, it is required to provide smooth and easy-to-understand information. Conventional systems have difficulty increasing customer satisfaction and purchasing motivation because their responses to questions are unnatural and the information they provide is inaccurate.

[0826] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0827] In this invention, the server includes a voice analysis means for capturing user voice data and performing voice recognition, a generation means for generating appropriate reaction content based on the voice recognition result, a timing calculation means for calculating the response timing of the generated reaction, a voice generation means for generating voice data based on the reaction content and the response timing, a playback means for playing back the generated voice data to the user, an information provision means for generating appropriate product information in response to the user's question, and a voice information playback means for playing back the information generated by the information provision means as voice, thereby enabling natural and quick responses to customer questions and the provision of accurate product information.

[0828] The "voice analysis means" is a means for capturing the user's voice data and performing voice recognition.

[0829] The "generation means" is a means for generating appropriate reaction content based on the speech recognition result.

[0830] The "timing calculation means" is a means for calculating the response timing of the generated reaction.

[0831] The "voice generating means" is a means for generating voice data based on the reaction content and response timing.

[0832] The "playback means" is a means for playing back the generated audio data to the user.

[0833] The "information providing means" is a means for generating appropriate product information in response to a user's question.

[0834] The "audio information reproducing means" is a means for reproducing the information generated by the information providing means as audio.

[0835] This embodiment will be described as a specific example of a voice assistant system that realizes natural conversations in a brick-and-mortar store.

[0836] The system uses smart glasses to support natural interactions between customers and store staff. Its main functions include voice analysis, reaction generation, timing calculation, voice generation, product information provision, and voice playback.

[0837] The server first uses a voice analysis means to capture the voice data sent from the smart glasses and performs voice recognition in real time. This voice recognition engine uses Google Cloud Speech-to-Text. Next, a generation means generates an appropriate reaction content based on the voice recognition results. This generation means uses OpenAI's GPT-4. The generated text data is passed to a timing calculation means to calculate the response timing of the reaction. Then, based on the calculated appropriate response timing, a voice generation means converts the reaction content into voice data. Google Cloud Text-to-Speech is used to generate this voice data. Finally, the generated voice data is played back to the user through the playback means of the smart glasses.

[0838] Furthermore, when a user asks a question about a product, the information providing means generates appropriate product information and plays it back as voice. Through this series of processes, customers can receive a quick and accurate response to their question, improving their shopping experience in physical stores.

[0839] As a concrete example, consider a case where a customer asks, "What is this item made of?" in a store. The voice assistant on the smart glasses captures this question, and Google Cloud Speech-to-Text converts it into text: "What is this item made of?" OpenAI GPT-4 then generates the appropriate response: "This item is made of 100% cotton." Google Cloud Text-to-Speech then converts this response into audio, which is played back to the customer through the smart glasses.

[0840] The following text is used as an example of a prompt sentence:

[0841] "A customer asks about the material of an item. Generate a natural response."

[0842] In this way, this system combines the functions of voice recognition, generative AI, timing calculation, voice generation, and information provision to enable natural conversations in physical stores.

[0843] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0844] Step 1:

[0845] The device (smart glasses) captures the user's voice. The input is the user's voice data, and the output is the captured voice data. Specifically, the microphone in the smart glasses records the user's speech in real time.

[0846] Step 2:

[0847] The device sends the captured voice data to the server, where speech recognition is performed using a speech analysis tool. The input is voice data, and the output is the speech recognition result (text data). Specifically, Google Cloud Speech-to-Text analyzes the voice data and generates text data such as "What is the material of this product?"

[0848] Step 3:

[0849] The server's generation means receives the speech recognition results (text data) and uses generation AI to generate an appropriate reaction. The input is text data, and the output is the generated reaction (text). Specifically, OpenAI's GPT-4 generates a response such as "This product is made of 100% cotton."

[0850] Step 4:

[0851] The server's timing calculation means calculates the response timing of the generated reaction. The input is the generated reaction content (text), and the output is the response timing (time interval). Specifically, it calculates the appropriate response timing (e.g., 0.5 seconds) after the user's utterance ends.

[0852] Step 5:

[0853] The server's voice generation means generates voice data based on the reaction content and response timing. The input is the generated reaction content (text) and response timing, and the output is voice data. Specifically, Google Cloud Text-to-Speech generates voice data saying, "This product is made of 100% cotton."

[0854] Step 6:

[0855] The server sends the generated voice data to the terminal. The input is the voice data, and the output is the completion of sending the voice data to the terminal. Specifically, the generated voice data is sent to the smart glasses via the network.

[0856] Step 7:

[0857] The device plays the generated voice data to the user based on the calculated response timing. The input is the voice data and the response timing, and the output is the voice playback. Specifically, the speaker on the smart glasses plays the voice, "This product is made of 100% cotton."

[0858] This series of steps allows users to get answers to their questions through natural dialogue.

[0859] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0860] This embodiment relates to a voice assistant system that automatically generates reaction content, response timing, and tone of voice using a generation AI and an emotion engine to realize natural dialogue with a user. This system mainly includes the following components:

[0861] 1. Audio analysis methods:

[0862] This is a function that allows the device to capture the user's voice data and perform speech recognition in real time. This allows the user's voice to be converted into text data. For example, if the user says, "The weather is nice today," the device will capture this utterance and generate the text data, "The weather is nice today."

[0863] 2. Emotion Engine:

[0864] This function recognizes the user's emotions by analyzing the text and voice data obtained by the server using voice analysis. This emotion engine determines the user's emotional state by taking into account not only the content of the speech but also the tone and pace of the voice.

[0865] 3. Generation means:

[0866] This function uses a generative AI to generate appropriate reactions based on the text data received by the server and the emotional data recognized by the emotion engine. The generative AI compares the content of the user's utterance with their emotions and generates an emotional expression that matches the context (e.g., a positive reaction such as "That's right! The weather is really nice today!").

[0867] 4. Timing calculation method:

[0868] This function calculates the appropriate response timing based on the reaction content generated by the server. For example, it calculates the timing to start the response approximately 0.5 seconds after the user finishes speaking.

[0869] 5. Sound Generation Means:

[0870] This function uses speech synthesis technology to generate the reaction content as voice based on the information from the generation means and timing calculation means. The emotions recognized by the emotion engine are also reflected. Therefore, the generated voice contains an appropriate emotional tone (in this example, a tone of joy).

[0871] 6. Regeneration means:

[0872] This function allows the terminal to play back the voice data sent from the server to the user. This playback function allows the user to experience natural conversation.

[0873] Program processing

[0874] The specific program processing is performed as follows: Below is a specific example of how the system responds to user utterances.

[0875] 1. The user says, "The weather is nice today."

[0876] 2. The device captures the user's voice and uses a speech recognition engine to generate text data such as "The weather is nice today."

[0877] 3. The device sends the generated text data to the back-end server, where it arrives via the network.

[0878] 4. The server receives the text data and analyzes the user's emotions using an emotion engine that works in conjunction with the generative AI. For example, the user's emotions are recognized as positive.

[0879] 5. The server uses the generative AI to generate an appropriate reaction. Based on the analysis results of the emotion engine, a response such as "Yes! It's really nice weather today!" is generated. This reaction corresponds to the user's emotions.

[0880] 6. The server uses the response timing calculation module to calculate an appropriate interval for the response (e.g., 0.5 seconds).

[0881] 7. The server uses the speech generation module to generate a reaction, "Yes! It's really nice weather today!", with a joyful tone added.

[0882] 8. The device receives the generated voice data and plays it back to the user based on the calculated response timing.

[0883] 9. The device will play "Yes! It's really nice weather today!" over the speaker, allowing the user to experience a natural conversation.

[0884] In this way, the system automatically performs a series of processes, from capturing and analyzing the user's speech to generating natural reactions at the appropriate time and responding as voice. This allows users to enjoy natural, stress-free conversations. The system can also achieve more accurate conversations by using call center recordings and other call data as training data.

[0885] The processing flow will be explained below.

[0886] Step 1:

[0887] The user speaks to the voice assistant. Specifically, the user says, "The weather is nice today."

[0888] Step 2:

[0889] The device captures the user's voice with a microphone and temporarily stores it as voice data. At the same time, the device's voice recognition engine activates and converts the voice data into text data in real time. For example, the text generated might be, "The weather is nice today."

[0890] Step 3:

[0891] The terminal sends the generated text data to the back-end server, where it arrives via the network.

[0892] Step 4:

[0893] The server receives the text data and begins analyzing it. As a preprocessing step for the generative AI, the text data is analyzed based on the context.

[0894] Step 5:

[0895] The server uses an emotion engine to analyze the user's emotions from the voice and text data, for example, recognizing that the user is in a positive emotion from the tone of voice and pace of speech.

[0896] Step 6:

[0897] The server uses generative AI to generate an appropriate reaction, taking into account the emotional data obtained from the emotion engine. Specifically, a positive reaction such as "Yes! It's really nice weather today!" is generated.

[0898] Step 7:

[0899] The server uses a response timing calculation module to calculate the optimal response timing for the generated reaction. It detects the end of the user's utterance and determines the timing to start the response, for example, 0.5 seconds later.

[0900] Step 8:

[0901] The server uses a voice generation module to convert the generated reaction text into voice data, which reflects emotional expressions such as a tone of joy.

[0902] Step 9:

[0903] The server sends the generated voice data to the terminal, which then receives the voice data via the network.

[0904] Step 10:

[0905] The device stores the received audio data in an internal buffer and waits until the calculated response timing. After a 0.5 second delay, the device prepares to play the audio data.

[0906] Step 11:

[0907] The device will play the voice "Yes! It's really nice weather today!" through the speaker, allowing the user to experience a natural and emotional interaction.

[0908] Step 12:

[0909] The user receives the voice assistant's response and returns to a system state where they can continue the conversation. The system waits for a new utterance.

[0910] In this way, the system automatically captures and analyzes the user's speech, recognizes their emotions, generates natural reactions at the appropriate time, and responds in voice, allowing users to enjoy natural, stress-free conversations.

[0911] Example 2

[0912] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0913] Conventional voice assistant systems often simply process user speech as text data and then react to it. This makes it difficult to understand and respond to the user's emotions and context, making it difficult to achieve natural dialogue with the user. Furthermore, the tone and timing of the generated voice can be mechanical, making the responses to the user seem unnatural.

[0914] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an analysis means for capturing user voice data and performing voice recognition, an analysis means for recognizing the user's emotion based on the voice recognition result and the voice data, a generation means for generating appropriate reaction content based on the analyzed emotion data and the voice recognition result, a calculation means for calculating the response timing of the generated reaction, a generation means for generating voice data based on the reaction content and response timing, and a playback means for playing the generated voice data to the user. This makes it possible to automatically generate natural reactions that take into account the user's utterance content and emotion, and play them as voice at appropriate timing.

[0915] "User" refers to an individual who uses this system to provide voice input.

[0916] "Voice Data" means information that is a digital recording of a user's speech.

[0917] "Analysis Means" refers to technical equipment or software for capturing audio data and performing voice recognition and emotion analysis.

[0918] "Text data" refers to character information converted from voice data by voice recognition.

[0919] "Emotion Engine" refers to a technical device or software for analyzing text and voice data to determine a user's emotional state.

[0920] "Generation means" refers to a technical device or software for automatically creating appropriate reaction content based on the analysis results of the emotion engine and the voice recognition results.

[0921] "Calculation means" refers to a technical device or software for calculating response timing based on reaction content.

[0922] "Generative AI" refers to artificial intelligence technology that generates appropriate reactions based on the user's speech and emotions.

[0923] A "prompt sentence" is an instruction sentence input to the generation AI that provides context for reaction generation.

[0924] "Speech generation means" refers to technical devices or software for generating reactions as synthetic voices based on text data and emotion data.

[0925] "Playback means" refers to a technical device for playing back the generated audio data in a form audible to the user.

[0926] This invention relates to a voice assistant system that analyzes a user's voice data, recognizes emotions, generates appropriate reactions, and provides the reactions to the user as voice. This system uses a generative AI model and an emotion engine to realize natural dialogue with the user.

[0927] 1. Audio analysis methods:

[0928] The device captures the user's voice data and performs real-time speech recognition. Specific speech recognition software used includes Google Speech-to-Text API and Amazon Transcribe. This speech analysis method converts the user's speech into text data. For example, if a user says, "The weather is nice today," the device captures this speech and generates the text data, "The weather is nice today."

[0929] 2. Emotion Engine:

[0930] The server analyzes the text and voice data obtained through the voice analysis tool to recognize the user's emotions. Specifically, emotion analysis software such as IBM Watson Tone Analyzer is used. This emotion engine determines the user's emotional state by taking into account not only the content of the speech but also the tone and pace of the voice.

[0931] 3. Generation means:

[0932] Based on the text data received by the server and the emotional data recognized by the emotion engine, an appropriate reaction is generated using a generative AI. The specific generative AI used includes OpenAI GPT-3. This generator compares the user's speech content with their emotions to generate emotional expressions that fit the context. For example, if the prompt text is input as "The user is in a positive mood and talking about the weather. Please generate an appropriate response," the response generated will be "Yes! The weather is really nice today!"

[0933] 4. Timing calculation method:

[0934] The server calculates an appropriate response timing based on the reaction content generated by the generation means. For example, a software module is provided to calculate the timing so that the response starts with a time interval of about 0.5 seconds after the user finishes speaking.

[0935] 5. Sound Generation Means:

[0936] The server uses speech synthesis technology based on the information from the generation method and timing calculation method to generate the reaction content as voice. Specific speech synthesis technologies used include Google Text-to-Speech API and Amazon Polly. The emotional tone recognized by the emotion engine is reflected in the voice. For example, the generated voice may include a tone of joy.

[0937] 6. Regeneration means:

[0938] The terminal reproduces the voice data sent from the server to the user, allowing the user to experience natural conversation.

[0939] By combining the above methods, the system can capture and analyze user utterances, and automatically generate natural reactions and respond as voice at the appropriate time. This allows users to enjoy natural, stress-free conversations. The system can also achieve higher accuracy by using call center recordings and other call data as training data.

[0940] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0941] Step 1:

[0942] The user inputs voice data. For example, the user might say, "The weather is nice today." This voice data becomes the input.

[0943] Step 2:

[0944] The device captures the user's voice data and converts it into text data using a speech recognition engine (e.g., Google Speech-to-Text API). The input is the user's voice data, and the output is the text data generated by speech recognition: "The weather is nice today."

[0945] Step 3:

[0946] The terminal sends the generated text data to the back-end server. The input is text data, and the output reaches the server via the network.

[0947] Step 4:

[0948] Based on the text data received by the server, an emotion engine (e.g., IBM Watson Tone Analyzer) is used to analyze the user's emotions. The input is text data, and "positive emotions" are recognized through data analysis. The output is the analyzed emotion data.

[0949] Step 5:

[0950] The server uses a generative AI (e.g., OpenAI GPT-3) to generate an appropriate reaction based on emotional and text data. The input is emotional and text data, and the prompt is, "The user is talking about the weather in a positive mood. Please generate an appropriate response." The output is the response generated by the generative AI: "Yes! The weather is really nice today!"

[0951] Step 6:

[0952] The server uses a response timing calculation module to calculate the appropriate interval (e.g., 0.5 seconds) for the response based on the reaction content. The input is the reaction content, and the output is the calculated response timing.

[0953] Step 7:

[0954] The server uses a speech generation module (e.g., Google Text-to-Speech API) to generate the reaction as speech data. The input is the text data "Yes! It's really nice weather today!" and an emotional tone (a tone of joy), and the output is the generated speech data.

[0955] Step 8:

[0956] The terminal receives the voice data sent from the server and plays it back to the user based on the calculated response timing. The input is the voice data and the response timing, and the output is audible voice for the user.

[0957] Step 9:

[0958] The device will play a voice message over the speaker saying, "Yes! It's really nice weather today!" This action allows the user to experience a natural conversation.

[0959] (Application example 2)

[0960] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0961] There is a demand for voice assistant systems that can provide more natural and effective customer service in brick-and-mortar stores. Conventional voice assistant systems have issues in that they are unable to fully achieve functions such as natural conversation that reflects the customer's emotions, or providing store guidance, product explanations, and promotional information.

[0962] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0963] In this invention, the server includes a voice analysis means for capturing user voice data and performing voice recognition, a generation means for generating appropriate reaction content based on the voice recognition result and the analyzed emotional state, a timing calculation means for calculating the response timing of the generated reaction, a voice generation means for generating voice data based on the reaction content and the response timing, a playback means for playing back the generated voice data to the user, and an information provision means for providing product information, store guidance, and promotion information. This allows for natural dialogue with customers and makes it possible to effectively provide in-store guidance, product explanations, and promotion information.

[0964] The "voice analysis means" is a function for capturing the user's voice data and performing voice recognition in real time.

[0965] The "generation means" is a function that generates appropriate reaction content based on the voice recognition results and the analyzed emotional state.

[0966] The "timing calculation means" is a function that calculates the response timing of the generated reaction.

[0967] The "voice generating means" is a function that generates voice data based on the reaction content and response timing.

[0968] The "playback means" is a function that plays back the generated audio data to the user.

[0969] "Information providing means" is a function that provides information about products, store guides, and promotional information.

[0970] To implement this invention, a specific system configuration and program are required. The hardware and software for realizing this system, as well as the methods for processing and calculating data, will be described below.

[0971] System Overview

[0972] The system has the following main components:

[0973] 1. Voice analysis means: Captures the user's voice data and performs voice recognition.

[0974] 2. Generation method: Generate appropriate reaction content based on the speech recognition results and the analyzed emotional state.

[0975] 3. Timing calculation means: Calculate the response timing of the generated reaction.

[0976] 4. Voice generation means: Generates voice data based on the reaction content and response timing.

[0977] 5. Playback means: Plays the generated audio data to the user.

[0978] 6. Information provision means: Providing product information, store guides, and promotional information.

[0979] Hardware and software used

[0980] Microphones and speakers: These are built into smartphones, smart glasses, and head-mounted displays.

[0981] Cloud servers: Used to run data analysis and generative AI models.

[0982] Generative AI models: Uses open-source GPT-3 and Azure Cognitive Services.

[0983] Emotion engine: Uses IBM Watson Tone Analyzer.

[0984] Speech recognition and synthesis: Uses Google Speech-to-Text and Amazon Polly.

[0985] Program processing flow

[0986] The program of this system operates as follows.

[0987] First, the user speaks to the voice assistant, and the voice data is captured through a microphone and converted into text data using a speech recognition method. Specifically, Google Speech-to-Text is used to convert voice to text.

[0988] This text data is then sent to a cloud server where an emotion engine analyzes the user's emotions, for example, using IBM Watson Tone Analyzer to determine whether the user is expressing positive emotions.

[0989] Based on the analyzed emotional state, the generator uses GPT-3 to generate an appropriate reaction, such as "This product uses new technology and is very durable."

[0990] Next, the timing calculation means calculates the response timing of approximately 0.5 seconds, and the voice generation means converts the generated reaction into voice data using Amazon Polly.

[0991] Finally, the playback means plays back the audio data through a speaker, allowing the user to experience natural dialogue.

[0992] Usage example

[0993] For example, consider a situation where a customer asks a voice assistant, "Tell me about the latest promotions." Here's a sample interaction:

[0994] Specific examples

[0995] 1. User: "Tell me about the latest promotions."

[0996] 2. Voice Capture: "What are the latest promotions?"

[0997] 3. Speech recognition: Generate text data for "Tell me the latest promotion information"

[0998] 4. Data Transmission

[0999] 5. Sentiment Analysis: Recognizing User Interest

[1000] 6. Reaction Generation: "These new shoes are currently 20% off!"

[1001] 7. Response timing calculation: approx. 0.5 seconds

[1002] 8. Speech synthesis: Convert "These new shoes are currently 20% off!" into speech data

[1003] 9. Audio playback

[1004] Prompt Sentence Examples

[1005] Customer: "Tell me about this product."

[1006] Analysis results:

[1007] Emotion: Interested

[1008] Product information: New product, durable, priced at $50

[1009] Prompt the generative AI model:

[1010] Input text: Please explain what happens when a customer expresses interest in a product that uses new technology.

[1011] Output: "This product uses the latest technology and is extremely durable. It costs $50."

[1012] In this way, the system can realize natural dialogue with customers and effectively provide in-store guidance, product explanations, and promotional information.

[1013] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1014] Step 1:

[1015] The user speaks to the voice assistant. The input is the user's voice data, and the specific action is the user saying, "Tell me about this product."

[1016] Step 2:

[1017] The device captures the user's voice data and performs speech recognition. The input is the user's voice data and the output is text data. The speech "Tell me about this product" is converted into text "Tell me about this product."

[1018] Step 3:

[1019] The device sends the generated text data to the cloud server. The input is text data, and the output is data transmission to the server.

[1020] Step 4:

[1021] The server receives the text data and uses the emotion engine to analyze the user's emotions. The input is text data, and the output is analyzed emotion data. Specifically, the text "Tell me about this product" is sent, and the emotion engine analyzes it as "Interested."

[1022] Step 5:

[1023] The server uses a generative AI model to generate appropriate reaction content based on the analyzed emotional state and text data. The input is text data and emotional data, and the output is the generated reaction content. Specifically, in response to the question "Tell me about this product," the server generates the response "This product uses the latest technology and is extremely durable."

[1024] Step 6:

[1025] The server uses the response timing calculation means to calculate the response timing of the generated reaction. The input is the generated reaction content, and the output is response timing information. Specifically, the server calculates the response timing of approximately 0.5 seconds.

[1026] Step 7:

[1027] The server uses a voice generation means to convert the generated reaction into voice data. The input is the generated reaction content and response timing information, and the output is voice data. Specifically, the text "This product uses the latest technology and is very durable" is converted into voice data.

[1028] Step 8:

[1029] The device plays back the audio data sent from the server. The input is audio data, and the output is audio playback to the user. Specifically, the device plays back through the speaker the audio, "This product uses the latest technology and is extremely durable."

[1030] This series of processing steps enables the system to have natural dialogue with the user, and effectively provide in-store guidance, product explanations, and promotional information.

[1031] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1032] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1033] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1034] [Fourth embodiment]

[1035] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1036] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1037] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1038] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1039] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1040] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1041] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1042] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1043] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1044] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1045] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1046] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1047] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1048] This embodiment relates to a voice assistant system that uses a generation AI to automatically generate reaction content, response timing, and tone of voice in order to realize natural dialogue with a user. This system mainly includes the following components:

[1049] 1. Audio analysis methods:

[1050] This is a function that allows the device to capture the user's voice data and perform speech recognition in real time. This allows the user's voice to be converted into text data. For example, if the user says, "The weather is nice today," the device will capture this utterance and generate the text data, "The weather is nice today."

[1051] 2. Generation means:

[1052] This function uses a generative AI to generate appropriate reactions based on the text data received by the server. The generative AI analyzes the user's speech and generates emotional expressions according to the context (e.g., a positive reaction such as "That's right! The weather is really nice today!").

[1053] 3. Timing calculation method:

[1054] This function calculates the appropriate response timing based on the reaction content generated by the server. For example, it calculates the timing to start the response approximately 0.5 seconds after the user finishes speaking.

[1055] 4. Sound Generation Means:

[1056] This function uses speech synthesis technology to generate reaction content as voice based on information from the generation method and timing calculation method. By adding tone of voice and emotion according to the reaction, natural dialogue voice can be generated. For example, if the reaction is "That's right! It's really nice weather today!", a tone of joy will be added to the generated voice.

[1057] 5. Regeneration means:

[1058] This function allows the terminal to play back the voice data sent from the server to the user. This playback function allows the user to experience natural conversation.

[1059] Program processing

[1060] The specific program processing is performed as follows: Below is a specific example of how the system responds to user utterances.

[1061] 1. The user says, "The weather is nice today."

[1062] 2. The device captures the user's voice and uses a speech recognition engine to generate text data such as "The weather is nice today."

[1063] 3. The device sends the generated text data to the server.

[1064] 4. The server receives the text data and uses generative AI to generate the most appropriate reaction for the context: "Yes! It's really nice weather today!"

[1065] 5. The server uses the response timing calculation module to calculate an appropriate interval for the response (e.g., 0.5 seconds).

[1066] 6. The server uses the speech generation module to generate a reaction, "Yes! It's really nice weather today!", with a joyful tone added.

[1067] 7. The device receives the generated voice data and plays it back to the user based on the calculated response timing.

[1068] This allows the user to experience a natural and smooth conversation with the voice assistant. The system of this embodiment automatically generates reactions, calculates response timing, and generates voices, so it can provide a natural dialogue environment without requiring user operation. The system can also achieve more accurate dialogue by using call center recording data and other call data as training data.

[1069] The processing flow will be explained below.

[1070] Step 1:

[1071] The user speaks to the voice assistant. Specifically, the user says, "The weather is nice today."

[1072] Step 2:

[1073] The device captures the user's voice with a microphone and temporarily stores it as voice data. At the same time, the device's voice recognition engine activates and converts the voice data into text data in real time. For example, the text generated might be, "The weather is nice today."

[1074] Step 3:

[1075] The terminal sends the generated text data to the back-end server, where it arrives via the network.

[1076] Step 4:

[1077] The server receives the text data. The generative AI module in the server analyzes the text data and understands the user's intent. Specifically, it extracts the positive context of "the weather is nice."

[1078] Step 5:

[1079] The server uses the generative AI module to generate an appropriate reaction, for example, "Yes! It's really nice weather today!" This reaction is intended to be a positive response to the user's comment.

[1080] Step 6:

[1081] The server uses a response timing calculation module to calculate the optimal timing for the reaction. Based on the speech analysis, it determines the timing to start the response, for example, 0.5 seconds later.

[1082] Step 7:

[1083] The server converts the generated text reaction into speech data using a speech generation module, where the generated speech has an appropriate emotional tone (in this example, a tone of joy).

[1084] Step 8:

[1085] The server sends the generated voice data to the terminal, which then receives the voice data via the network.

[1086] Step 9:

[1087] The device stores the received voice data in an internal buffer and plays it back through the speaker after the 0.5 second interval calculated based on the response timing has elapsed.

[1088] Step 10:

[1089] The device will play a voice message over the speaker saying, "Yes! It's really nice weather today!" This allows the user to experience a natural conversation.

[1090] Step 11:

[1091] The user receives a response from the voice assistant and can continue the conversation. The system waits for a new utterance.

[1092] In this way, the system automatically captures and analyzes the user's speech, generating natural reactions at the appropriate times and responding as voice, allowing users to enjoy natural, stress-free conversations.

[1093] Example 1

[1094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1095] Conventional voice assistant systems often lack natural dialogue with users, particularly insufficient response timing and emotional expression. This leads to lower user satisfaction and a lower rate of continued use of the system. The goal of this invention is to achieve more natural, human-like dialogue and increase user satisfaction.

[1096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1097] In this invention, the server includes a generation means for generating appropriate reaction content using a generation AI model based on the speech recognition result, a timing calculation means for calculating the response timing of the generated reaction, and a voice generation means for generating voice data based on the reaction content and response timing. This enables natural and smooth dialogue by analyzing and responding to the user's speech in real time and expressing emotions appropriately.

[1098] "User" refers to the individual or end user who uses the voice assistant system.

[1099] "Voice data" refers to data that records what a user says as a digital signal.

[1100] "Voice analysis means" refers to equipment or software for capturing a user's voice data and performing voice recognition.

[1101] A "generative AI model" refers to an artificial intelligence model that uses machine learning and neural network technology to generate optimal output for input data.

[1102] "Generation means" refers to a device or program that has the function of generating appropriate reaction content using a generative AI model based on the voice recognition results.

[1103] "Response timing" refers to the time interval between a user's utterance and the system's response.

[1104] "Timing calculation means" refers to a device or program that has the function of calculating the response timing of the generated reaction.

[1105] "Speech generation means" refers to a device or program that has the function of generating voice data based on the generated reaction content and response timing.

[1106] "Playback means" refers to equipment or software for playing back the generated audio data to the user.

[1107] "System" refers to the entire voice assistant, including the above means, that enables natural dialogue with the user.

[1108] The present invention relates to a voice assistant system that uses generation AI to automatically generate reaction content, response timing, and tone of voice in order to realize natural dialogue with users. This system mainly includes the following components:

[1109] 1. Audio analysis methods:

[1110] The device has the ability to capture the user's voice data and perform voice recognition in real time. This typically uses the device's built-in microphone and a voice recognition engine (such as a general voice recognition API). For example, if a user says, "The weather is nice today," the device captures this speech and generates "The weather is nice today" as text data.

[1111] 2. Generation means:

[1112] The system has the ability to generate appropriate reactions using a generative AI model based on the text data received by the server. This uses a generative AI model (e.g., a widely used generative AI technology). The server analyzes the user's speech and generates emotional expressions according to the context. For example, it generates a positive reaction such as, "Yes! It's really nice weather today!"

[1113] 3. Timing calculation method:

[1114] The server has a function to calculate the appropriate response timing based on the reaction content generated by the generation means. For example, it calculates the timing so that the response starts about 0.5 seconds after the user finishes speaking. For this, a timing calculation module for timing control within the program is used.

[1115] 4. Sound Generation Means:

[1116] The server has the function of generating reaction content as voice using voice synthesis technology based on information from the generation means and timing calculation means. In particular, it is possible to generate natural dialogue voice by adding emotions and tone of voice according to the reaction. This implementation uses a voice synthesis API (e.g., a widely used voice synthesis technology). Specifically, voice is generated by adding a tone of joy.

[1117] 5. Regeneration means:

[1118] The device has a function to play back the voice data sent from the server to the user. This playback means allows the user to experience a natural conversation. The voice is output using the device's built-in speaker.

[1119] Specific examples

[1120] For example, the following prompt sentence is input to a generative AI model:

[1121] "When a user says, 'The weather is nice today,' generate an appropriate response."

[1122] Given this input, the generator AI will output the following:

[1123] "Yes! It's really beautiful weather today!"

[1124] Based on the reactions generated in this way, the server calculates the response timing and generates voice data.Finally, the device receives the generated voice data and plays it back to the user at the appropriate time.

[1125] This allows users to have a natural and smooth conversation with the voice assistant. The system of the present invention provides a natural conversation environment by automatically processing each step, without requiring user operation.

[1126] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1127] Step 1:

[1128] The user says, "The weather is nice today."

[1129] Specific operation: The device's built-in microphone captures the user's speech.

[1130] Input: User's voice

[1131] Output: Audio data captured within the device

[1132] Step 2:

[1133] The voice data captured by the terminal is converted into text data using a voice recognition engine.

[1134] Specific operation: The device calls the Google Speech-to-Text API or similar and converts the voice data into text data such as "The weather is nice today."

[1135] Input: Audio data

[1136] Output: Text data "The weather is nice today."

[1137] Step 3:

[1138] The terminal transmits the generated text data to the server.

[1139] Specific operation: The terminal generates an HTTP request and sends text data to the server.

[1140] Input: Text data "The weather is nice today"

[1141] Output: Text data received by the server

[1142] Step 4:

[1143] The server uses a generative AI model to generate appropriate reaction content based on the text data received.

[1144] How it works: The server uses a generative AI model such as OpenAI GPT-4 to input text data as a prompt and generate a response text such as "Yes! It's really nice weather today!"

[1145] Input: Text data "The weather is nice today"

[1146] Output: Response text "Yes! It's really nice weather today!"

[1147] Step 5:

[1148] The server calculates the appropriate response timing based on the generated response text.

[1149] Specific operation: The server uses the timing calculation module to calculate the timing to respond 0.5 seconds after the user finishes speaking.

[1150] Input: Response text "Yes! It's really beautiful weather today!"

[1151] Output: Response timing (e.g., after 0.5 seconds)

[1152] Step 6:

[1153] The server generates voice data using a voice synthesis technique based on the response text and response timing.

[1154] Specific operation: The server calls the Google Cloud Text-to-Speech API or similar and generates the response text as voice data with a joyful tone added.

[1155] Input: Response text "Yes! It's really nice weather today!", Response timing

[1156] Output: Audio data

[1157] Step 7:

[1158] The terminal receives the generated voice data and plays it back to the user based on the calculated response timing.

[1159] Specific operation: The device stores the voice data in a buffer, checks the response timing, and then plays the voice through the built-in speaker.

[1160] Input: Voice data, response timing

[1161] Output: Audio output to the user

[1162] (Application example 1)

[1163] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1164] The goal is to provide an assistant system that makes interactions with customers in physical stores more natural and effective. In particular, when customers ask questions about products, it is required to provide smooth and easy-to-understand information. Conventional systems have difficulty increasing customer satisfaction and purchasing motivation because their responses to questions are unnatural and the information they provide is inaccurate.

[1165] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1166] In this invention, the server includes a voice analysis means for capturing user voice data and performing voice recognition, a generation means for generating appropriate reaction content based on the voice recognition result, a timing calculation means for calculating the response timing of the generated reaction, a voice generation means for generating voice data based on the reaction content and the response timing, a playback means for playing back the generated voice data to the user, an information provision means for generating appropriate product information in response to the user's question, and a voice information playback means for playing back the information generated by the information provision means as voice, thereby enabling natural and quick responses to customer questions and the provision of accurate product information.

[1167] The "voice analysis means" is a means for capturing the user's voice data and performing voice recognition.

[1168] The "generation means" is a means for generating appropriate reaction content based on the speech recognition result.

[1169] The "timing calculation means" is a means for calculating the response timing of the generated reaction.

[1170] The "voice generating means" is a means for generating voice data based on the reaction content and response timing.

[1171] The "playback means" is a means for playing back the generated audio data to the user.

[1172] The "information providing means" is a means for generating appropriate product information in response to a user's question.

[1173] The "audio information reproducing means" is a means for reproducing the information generated by the information providing means as audio.

[1174] This embodiment will be described as a specific example of a voice assistant system that realizes natural conversations in a brick-and-mortar store.

[1175] The system uses smart glasses to support natural interactions between customers and store staff. Its main functions include voice analysis, reaction generation, timing calculation, voice generation, product information provision, and voice playback.

[1176] The server first uses a voice analysis means to capture the voice data sent from the smart glasses and performs voice recognition in real time. This voice recognition engine uses Google Cloud Speech-to-Text. Next, a generation means generates an appropriate reaction content based on the voice recognition results. This generation means uses OpenAI's GPT-4. The generated text data is passed to a timing calculation means to calculate the response timing of the reaction. Then, based on the calculated appropriate response timing, a voice generation means converts the reaction content into voice data. Google Cloud Text-to-Speech is used to generate this voice data. Finally, the generated voice data is played back to the user through the playback means of the smart glasses.

[1177] Furthermore, when a user asks a question about a product, the information providing means generates appropriate product information and plays it back as voice. Through this series of processes, customers can receive a quick and accurate response to their question, improving their shopping experience in physical stores.

[1178] As a concrete example, consider a case where a customer asks, "What is this item made of?" in a store. The voice assistant on the smart glasses captures this question, and Google Cloud Speech-to-Text converts it into text: "What is this item made of?" OpenAI GPT-4 then generates the appropriate response: "This item is made of 100% cotton." Google Cloud Text-to-Speech then converts this response into audio, which is played back to the customer through the smart glasses.

[1179] The following text is used as an example of a prompt sentence:

[1180] "A customer asks about the material of an item. Generate a natural response."

[1181] In this way, this system combines the functions of voice recognition, generative AI, timing calculation, voice generation, and information provision to enable natural conversations in physical stores.

[1182] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1183] Step 1:

[1184] The device (smart glasses) captures the user's voice. The input is the user's voice data, and the output is the captured voice data. Specifically, the microphone in the smart glasses records the user's speech in real time.

[1185] Step 2:

[1186] The device sends the captured voice data to the server, where speech recognition is performed using a speech analysis tool. The input is voice data, and the output is the speech recognition result (text data). Specifically, Google Cloud Speech-to-Text analyzes the voice data and generates text data such as "What is the material of this product?"

[1187] Step 3:

[1188] The server's generation means receives the speech recognition results (text data) and uses generation AI to generate an appropriate reaction. The input is text data, and the output is the generated reaction (text). Specifically, OpenAI's GPT-4 generates a response such as "This product is made of 100% cotton."

[1189] Step 4:

[1190] The server's timing calculation means calculates the response timing of the generated reaction. The input is the generated reaction content (text), and the output is the response timing (time interval). Specifically, it calculates the appropriate response timing (e.g., 0.5 seconds) after the user's utterance ends.

[1191] Step 5:

[1192] The server's voice generation means generates voice data based on the reaction content and response timing. The input is the generated reaction content (text) and response timing, and the output is voice data. Specifically, Google Cloud Text-to-Speech generates voice data saying, "This product is made of 100% cotton."

[1193] Step 6:

[1194] The server sends the generated voice data to the terminal. The input is the voice data, and the output is the completion of sending the voice data to the terminal. Specifically, the generated voice data is sent to the smart glasses via the network.

[1195] Step 7:

[1196] The device plays the generated voice data to the user based on the calculated response timing. The input is the voice data and the response timing, and the output is the voice playback. Specifically, the speaker on the smart glasses plays the voice, "This product is made of 100% cotton."

[1197] This series of steps allows users to get answers to their questions through natural dialogue.

[1198] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1199] This embodiment relates to a voice assistant system that automatically generates reaction content, response timing, and tone of voice using a generation AI and an emotion engine to realize natural dialogue with a user. This system mainly includes the following components:

[1200] 1. Audio analysis methods:

[1201] This is a function that allows the device to capture the user's voice data and perform speech recognition in real time. This allows the user's voice to be converted into text data. For example, if the user says, "The weather is nice today," the device will capture this utterance and generate the text data, "The weather is nice today."

[1202] 2. Emotion Engine:

[1203] This function recognizes the user's emotions by analyzing the text and voice data obtained by the server using voice analysis. This emotion engine determines the user's emotional state by taking into account not only the content of the speech but also the tone and pace of the voice.

[1204] 3. Generation means:

[1205] This function uses a generative AI to generate appropriate reactions based on the text data received by the server and the emotional data recognized by the emotion engine. The generative AI compares the content of the user's utterance with their emotions and generates an emotional expression that matches the context (e.g., a positive reaction such as "That's right! The weather is really nice today!").

[1206] 4. Timing calculation method:

[1207] This function calculates the appropriate response timing based on the reaction content generated by the server. For example, it calculates the timing to start the response approximately 0.5 seconds after the user finishes speaking.

[1208] 5. Sound Generation Means:

[1209] This function uses speech synthesis technology to generate the reaction content as voice based on the information from the generation means and timing calculation means. The emotions recognized by the emotion engine are also reflected. Therefore, the generated voice contains an appropriate emotional tone (in this example, a tone of joy).

[1210] 6. Regeneration means:

[1211] This function allows the terminal to play back the voice data sent from the server to the user. This playback function allows the user to experience natural conversation.

[1212] Program processing

[1213] The specific program processing is performed as follows: Below is a specific example of how the system responds to user utterances.

[1214] 1. The user says, "The weather is nice today."

[1215] 2. The device captures the user's voice and uses a speech recognition engine to generate text data such as "The weather is nice today."

[1216] 3. The device sends the generated text data to the back-end server, where it arrives via the network.

[1217] 4. The server receives the text data and analyzes the user's emotions using an emotion engine that works in conjunction with the generative AI. For example, the user's emotions are recognized as positive.

[1218] 5. The server uses the generative AI to generate an appropriate reaction. Based on the analysis results of the emotion engine, a response such as "Yes! It's really nice weather today!" is generated. This reaction corresponds to the user's emotions.

[1219] 6. The server uses the response timing calculation module to calculate an appropriate interval for the response (e.g., 0.5 seconds).

[1220] 7. The server uses the speech generation module to generate a reaction, "Yes! It's really nice weather today!", with a joyful tone added.

[1221] 8. The device receives the generated voice data and plays it back to the user based on the calculated response timing.

[1222] 9. The device will play "Yes! It's really nice weather today!" over the speaker, allowing the user to experience a natural conversation.

[1223] In this way, the system automatically performs a series of processes, from capturing and analyzing the user's speech to generating natural reactions at the appropriate time and responding as voice. This allows users to enjoy natural, stress-free conversations. The system can also achieve more accurate conversations by using call center recordings and other call data as training data.

[1224] The processing flow will be explained below.

[1225] Step 1:

[1226] The user speaks to the voice assistant. Specifically, the user says, "The weather is nice today."

[1227] Step 2:

[1228] The device captures the user's voice with a microphone and temporarily stores it as voice data. At the same time, the device's voice recognition engine activates and converts the voice data into text data in real time. For example, the text generated might be, "The weather is nice today."

[1229] Step 3:

[1230] The terminal sends the generated text data to the back-end server, where it arrives via the network.

[1231] Step 4:

[1232] The server receives the text data and begins analyzing it. As a preprocessing step for the generative AI, the text data is analyzed based on the context.

[1233] Step 5:

[1234] The server uses an emotion engine to analyze the user's emotions from the voice and text data, for example, recognizing that the user is in a positive emotion from the tone of voice and pace of speech.

[1235] Step 6:

[1236] The server uses generative AI to generate an appropriate reaction, taking into account the emotional data obtained from the emotion engine. Specifically, a positive reaction such as "Yes! It's really nice weather today!" is generated.

[1237] Step 7:

[1238] The server uses a response timing calculation module to calculate the optimal response timing for the generated reaction. It detects the end of the user's utterance and determines the timing to start the response, for example, 0.5 seconds later.

[1239] Step 8:

[1240] The server uses a voice generation module to convert the generated reaction text into voice data, which reflects emotional expressions such as a tone of joy.

[1241] Step 9:

[1242] The server sends the generated voice data to the terminal, which then receives the voice data via the network.

[1243] Step 10:

[1244] The device stores the received audio data in an internal buffer and waits until the calculated response timing. After a 0.5 second delay, the device prepares to play the audio data.

[1245] Step 11:

[1246] The device will play the voice "Yes! It's really nice weather today!" through the speaker, allowing the user to experience a natural and emotional interaction.

[1247] Step 12:

[1248] The user receives the voice assistant's response and returns to a system state where they can continue the conversation. The system waits for a new utterance.

[1249] In this way, the system automatically captures and analyzes the user's speech, recognizes their emotions, generates natural reactions at the appropriate time, and responds in voice, allowing users to enjoy natural, stress-free conversations.

[1250] Example 2

[1251] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1252] Conventional voice assistant systems often simply process user speech as text data and then react to it. This makes it difficult to understand and respond to the user's emotions and context, making it difficult to achieve natural dialogue with the user. Furthermore, the tone and timing of the generated voice can be mechanical, making the responses to the user seem unnatural.

[1253] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an analysis means for capturing user voice data and performing voice recognition, an analysis means for recognizing the user's emotion based on the voice recognition result and the voice data, a generation means for generating appropriate reaction content based on the analyzed emotion data and the voice recognition result, a calculation means for calculating the response timing of the generated reaction, a generation means for generating voice data based on the reaction content and response timing, and a playback means for playing the generated voice data to the user. This makes it possible to automatically generate natural reactions that take into account the user's utterance content and emotion, and play them as voice at appropriate timing.

[1254] "User" refers to an individual who uses this system to provide voice input.

[1255] "Voice Data" means information that is a digital recording of a user's speech.

[1256] "Analysis Means" refers to technical equipment or software for capturing audio data and performing voice recognition and emotion analysis.

[1257] "Text data" refers to character information converted from voice data by voice recognition.

[1258] "Emotion Engine" refers to a technical device or software for analyzing text and voice data to determine a user's emotional state.

[1259] "Generation means" refers to a technical device or software for automatically creating appropriate reaction content based on the analysis results of the emotion engine and the voice recognition results.

[1260] "Calculation means" refers to a technical device or software for calculating response timing based on reaction content.

[1261] "Generative AI" refers to artificial intelligence technology that generates appropriate reactions based on the user's speech and emotions.

[1262] A "prompt sentence" is an instruction sentence input to the generation AI that provides context for reaction generation.

[1263] "Speech generation means" refers to technical devices or software for generating reactions as synthetic voices based on text data and emotion data.

[1264] "Playback means" refers to a technical device for playing back the generated audio data in a form audible to the user.

[1265] This invention relates to a voice assistant system that analyzes a user's voice data, recognizes emotions, generates appropriate reactions, and provides the reactions to the user as voice. This system uses a generative AI model and an emotion engine to realize natural dialogue with the user.

[1266] 1. Audio analysis methods:

[1267] The device captures the user's voice data and performs real-time speech recognition. Specific speech recognition software used includes Google Speech-to-Text API and Amazon Transcribe. This speech analysis method converts the user's speech into text data. For example, if a user says, "The weather is nice today," the device captures this speech and generates the text data, "The weather is nice today."

[1268] 2. Emotion Engine:

[1269] The server analyzes the text and voice data obtained through the voice analysis tool to recognize the user's emotions. Specifically, emotion analysis software such as IBM Watson Tone Analyzer is used. This emotion engine determines the user's emotional state by taking into account not only the content of the speech but also the tone and pace of the voice.

[1270] 3. Generation means:

[1271] Based on the text data received by the server and the emotional data recognized by the emotion engine, an appropriate reaction is generated using a generative AI. The specific generative AI used includes OpenAI GPT-3. This generator compares the user's speech content with their emotions to generate emotional expressions that fit the context. For example, if the prompt text is input as "The user is in a positive mood and talking about the weather. Please generate an appropriate response," the response generated will be "Yes! The weather is really nice today!"

[1272] 4. Timing calculation method:

[1273] The server calculates an appropriate response timing based on the reaction content generated by the generation means. For example, a software module is provided to calculate the timing so that the response starts with a time interval of about 0.5 seconds after the user finishes speaking.

[1274] 5. Sound Generation Means:

[1275] The server uses speech synthesis technology based on the information from the generation method and timing calculation method to generate the reaction content as voice. Specific speech synthesis technologies used include Google Text-to-Speech API and Amazon Polly. The emotional tone recognized by the emotion engine is reflected in the voice. For example, the generated voice may include a tone of joy.

[1276] 6. Regeneration means:

[1277] The terminal reproduces the voice data sent from the server to the user, allowing the user to experience natural conversation.

[1278] By combining the above methods, the system can capture and analyze user utterances, and automatically generate natural reactions and respond as voice at the appropriate time. This allows users to enjoy natural, stress-free conversations. The system can also achieve higher accuracy by using call center recordings and other call data as training data.

[1279] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1280] Step 1:

[1281] The user inputs voice data. For example, the user might say, "The weather is nice today." This voice data becomes the input.

[1282] Step 2:

[1283] The device captures the user's voice data and converts it into text data using a speech recognition engine (e.g., Google Speech-to-Text API). The input is the user's voice data, and the output is the text data generated by speech recognition: "The weather is nice today."

[1284] Step 3:

[1285] The terminal sends the generated text data to the back-end server. The input is text data, and the output reaches the server via the network.

[1286] Step 4:

[1287] Based on the text data received by the server, an emotion engine (e.g., IBM Watson Tone Analyzer) is used to analyze the user's emotions. The input is text data, and "positive emotions" are recognized through data analysis. The output is the analyzed emotion data.

[1288] Step 5:

[1289] The server uses a generative AI (e.g., OpenAI GPT-3) to generate an appropriate reaction based on emotional and text data. The input is emotional and text data, and the prompt is, "The user is talking about the weather in a positive mood. Please generate an appropriate response." The output is the response generated by the generative AI: "Yes! The weather is really nice today!"

[1290] Step 6:

[1291] The server uses a response timing calculation module to calculate the appropriate interval (e.g., 0.5 seconds) for the response based on the reaction content. The input is the reaction content, and the output is the calculated response timing.

[1292] Step 7:

[1293] The server uses a speech generation module (e.g., Google Text-to-Speech API) to generate the reaction as speech data. The input is the text data "Yes! It's really nice weather today!" and an emotional tone (a tone of joy), and the output is the generated speech data.

[1294] Step 8:

[1295] The terminal receives the voice data sent from the server and plays it back to the user based on the calculated response timing. The input is the voice data and the response timing, and the output is audible voice for the user.

[1296] Step 9:

[1297] The device will play a voice message over the speaker saying, "Yes! It's really nice weather today!" This action allows the user to experience a natural conversation.

[1298] (Application example 2)

[1299] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1300] There is a demand for voice assistant systems that can provide more natural and effective customer service in brick-and-mortar stores. Conventional voice assistant systems have issues in that they are unable to fully achieve functions such as natural conversation that reflects the customer's emotions, or providing store guidance, product explanations, and promotional information.

[1301] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1302] In this invention, the server includes a voice analysis means for capturing user voice data and performing voice recognition, a generation means for generating appropriate reaction content based on the voice recognition result and the analyzed emotional state, a timing calculation means for calculating the response timing of the generated reaction, a voice generation means for generating voice data based on the reaction content and the response timing, a playback means for playing back the generated voice data to the user, and an information provision means for providing product information, store guidance, and promotion information. This allows for natural dialogue with customers and makes it possible to effectively provide in-store guidance, product explanations, and promotion information.

[1303] The "voice analysis means" is a function for capturing the user's voice data and performing voice recognition in real time.

[1304] The "generation means" is a function that generates appropriate reaction content based on the voice recognition results and the analyzed emotional state.

[1305] The "timing calculation means" is a function that calculates the response timing of the generated reaction.

[1306] The "voice generating means" is a function that generates voice data based on the reaction content and response timing.

[1307] The "playback means" is a function that plays back the generated audio data to the user.

[1308] "Information providing means" is a function that provides information about products, store guides, and promotional information.

[1309] To implement this invention, a specific system configuration and program are required. The hardware and software for realizing this system, as well as the methods for processing and calculating data, will be described below.

[1310] System Overview

[1311] The system has the following main components:

[1312] 1. Voice analysis means: Captures the user's voice data and performs voice recognition.

[1313] 2. Generation method: Generate appropriate reaction content based on the speech recognition results and the analyzed emotional state.

[1314] 3. Timing calculation means: Calculate the response timing of the generated reaction.

[1315] 4. Voice generation means: Generates voice data based on the reaction content and response timing.

[1316] 5. Playback means: Plays the generated audio data to the user.

[1317] 6. Information provision means: Providing product information, store guides, and promotional information.

[1318] Hardware and software used

[1319] Microphones and speakers: These are built into smartphones, smart glasses, and head-mounted displays.

[1320] Cloud servers: Used to run data analysis and generative AI models.

[1321] Generative AI models: Uses open-source GPT-3 and Azure Cognitive Services.

[1322] Emotion engine: Uses IBM Watson Tone Analyzer.

[1323] Speech recognition and synthesis: Uses Google Speech-to-Text and Amazon Polly.

[1324] Program processing flow

[1325] The program of this system operates as follows.

[1326] First, the user speaks to the voice assistant, and the voice data is captured through a microphone and converted into text data using a speech recognition method. Specifically, Google Speech-to-Text is used to convert voice to text.

[1327] This text data is then sent to a cloud server where an emotion engine analyzes the user's emotions, for example, using IBM Watson Tone Analyzer to determine whether the user is expressing positive emotions.

[1328] Based on the analyzed emotional state, the generator uses GPT-3 to generate an appropriate reaction, such as "This product uses new technology and is very durable."

[1329] Next, the timing calculation means calculates the response timing of approximately 0.5 seconds, and the voice generation means converts the generated reaction into voice data using Amazon Polly.

[1330] Finally, the playback means plays back the audio data through a speaker, allowing the user to experience natural dialogue.

[1331] Usage example

[1332] For example, consider a situation where a customer asks a voice assistant, "Tell me about the latest promotions." Here's a sample interaction:

[1333] Specific examples

[1334] 1. User: "Tell me about the latest promotions."

[1335] 2. Voice Capture: "What are the latest promotions?"

[1336] 3. Speech recognition: Generate text data for "Tell me the latest promotion information"

[1337] 4. Data Transmission

[1338] 5. Sentiment Analysis: Recognizing User Interest

[1339] 6. Reaction Generation: "These new shoes are currently 20% off!"

[1340] 7. Response timing calculation: approx. 0.5 seconds

[1341] 8. Speech synthesis: Convert "These new shoes are currently 20% off!" into speech data

[1342] 9. Audio playback

[1343] Prompt Sentence Examples

[1344] Customer: "Tell me about this product."

[1345] Analysis results:

[1346] Emotion: Interested

[1347] Product information: New product, durable, priced at $50

[1348] Prompt the generative AI model:

[1349] Input text: Please explain what happens when a customer expresses interest in a product that uses new technology.

[1350] Output: "This product uses the latest technology and is extremely durable. It costs $50."

[1351] In this way, the system can realize natural dialogue with customers and effectively provide in-store guidance, product explanations, and promotional information.

[1352] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1353] Step 1:

[1354] The user speaks to the voice assistant. The input is the user's voice data, and the specific action is the user saying, "Tell me about this product."

[1355] Step 2:

[1356] The device captures the user's voice data and performs speech recognition. The input is the user's voice data and the output is text data. The speech "Tell me about this product" is converted into text "Tell me about this product."

[1357] Step 3:

[1358] The device sends the generated text data to the cloud server. The input is text data, and the output is data transmission to the server.

[1359] Step 4:

[1360] The server receives the text data and uses the emotion engine to analyze the user's emotions. The input is text data, and the output is analyzed emotion data. Specifically, the text "Tell me about this product" is sent, and the emotion engine analyzes it as "Interested."

[1361] Step 5:

[1362] The server uses a generative AI model to generate appropriate reaction content based on the analyzed emotional state and text data. The input is text data and emotional data, and the output is the generated reaction content. Specifically, in response to the question "Tell me about this product," the server generates the response "This product uses the latest technology and is extremely durable."

[1363] Step 6:

[1364] The server uses the response timing calculation means to calculate the response timing of the generated reaction. The input is the generated reaction content, and the output is response timing information. Specifically, the server calculates the response timing of approximately 0.5 seconds.

[1365] Step 7:

[1366] The server uses a voice generation means to convert the generated reaction into voice data. The input is the generated reaction content and response timing information, and the output is voice data. Specifically, the text "This product uses the latest technology and is very durable" is converted into voice data.

[1367] Step 8:

[1368] The device plays back the audio data sent from the server. The input is audio data, and the output is audio playback to the user. Specifically, the device plays back through the speaker the audio, "This product uses the latest technology and is extremely durable."

[1369] This series of processing steps enables the system to have natural dialogue with the user, and effectively provide in-store guidance, product explanations, and promotional information.

[1370] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1371] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1372] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1373] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1374] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1375] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1376] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1377] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1378] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1379] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1380] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1381] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1382] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1383] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1384] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1385] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1386] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1387] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1388] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1389] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1390] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1391] The following is further disclosed regarding the above embodiment.

[1392] (Claim 1)

[1393] a voice analysis means for capturing user voice data and performing voice recognition;

[1394] A generating means for generating appropriate reaction content based on the speech recognition result;

[1395] a timing calculation means for calculating a response timing of the generated reaction;

[1396] a voice generating means for generating voice data based on the reaction content and response timing;

[1397] The system includes playback means for playing the generated audio data to a user.

[1398] (Claim 2)

[1399] 2. The system according to claim 1, wherein the timing calculation means detects the end of a user's speech and calculates a response timing having a predetermined interval.

[1400] (Claim 3)

[1401] 2. The system according to claim 1, wherein the generating means is capable of generating at least one emotional reaction such as laughter, surprise, or confusion.

[1402] "Example 1"

[1403] (Claim 1)

[1404] a voice analysis means for capturing user voice data and performing voice recognition;

[1405] A generation means for generating appropriate reaction content using a generation AI model based on the speech recognition result;

[1406] a timing calculation means for calculating a response timing of the generated reaction;

[1407] a voice generating means for generating voice data based on the reaction content and response timing;

[1408] The system includes playback means for playing the generated audio data to a user.

[1409] (Claim 2)

[1410] 2. The system according to claim 1, wherein the timing calculation means detects the end of a user's speech and calculates a response timing having a predetermined interval.

[1411] (Claim 3)

[1412] 2. The system according to claim 1, wherein the generating means is capable of generating at least one emotional reaction such as laughter, surprise, or confusion.

[1413] "Application Example 1"

[1414] (Claim 1)

[1415] a voice analysis means for capturing user voice data and performing voice recognition;

[1416] A generating means for generating appropriate reaction content based on the speech recognition result;

[1417] a timing calculation means for calculating a response timing of the generated reaction;

[1418] a voice generating means for generating voice data based on the reaction content and response timing;

[1419] playback means for playing the generated audio data to a user;

[1420] an information providing means for generating appropriate product information in response to a user's question;

[1421] an audio information reproducing means for reproducing the information generated by the information providing means as audio;

[1422] A system including:

[1423] (Claim 2)

[1424] 2. The system according to claim 1, wherein the timing calculation means detects the end of a user's speech and calculates a response timing having a predetermined interval.

[1425] (Claim 3)

[1426] The system described in claim 1, characterized in that the generation means is capable of generating emotional reactions such as at least laughter, surprise, or confusion, and provides information through natural dialogue in response to customer questions about products in a physical store.

[1427] "Example 2: Combining Emotion Engines"

[1428] (Claim 1)

[1429] an analyzing means for capturing user voice data and performing voice recognition;

[1430] an analysis means for recognizing a user's emotion based on the speech recognition result and the speech data;

[1431] A generating means for generating appropriate reaction content based on the analyzed emotion data and the speech recognition result;

[1432] a calculation means for calculating a response timing of the generated reaction;

[1433] generating means for generating voice data based on the reaction content and response timing;

[1434] The system includes playback means for playing the generated audio data to a user.

[1435] (Claim 2)

[1436] 2. The system according to claim 1, wherein the calculation means detects the end of a user's speech and calculates a response timing having a predetermined interval.

[1437] (Claim 3)

[1438] 10. The system of claim 1, wherein said generating means is capable of generating emotional reactions including at least a joyful tone.

[1439] "Application example 2 when combining emotion engines"

[1440] (Claim 1)

[1441] a voice analysis means for capturing user voice data and performing voice recognition;

[1442] generating means for generating appropriate reaction content based on the speech recognition result and the analyzed emotional state;

[1443] a timing calculation means for calculating a response timing of the generated reaction;

[1444] a voice generating means for generating voice data based on the reaction content and response timing;

[1445] playback means for playing the generated audio data to a user;

[1446] A system including an information providing means for providing product information, store guides, and promotional information.

[1447] (Claim 2)

[1448] 2. The system according to claim 1, wherein the timing calculation means detects the end of a user's speech and calculates a response timing having a predetermined interval.

[1449] (Claim 3)

[1450] The system according to claim 1, characterized in that the generation means is capable of generating emotional reactions such as at least laughter, surprise, or confusion, and is further capable of generating product descriptions and promotional information. [Explanation of symbols]

[1451] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a voice analysis means for capturing user voice data and performing voice recognition; A generating means for generating appropriate reaction content based on the speech recognition result; a timing calculation means for calculating a response timing of the generated reaction; a voice generating means for generating voice data based on the reaction content and response timing; The system includes playback means for playing the generated audio data to a user.

2. 2. The system according to claim 1, wherein said timing calculation means detects the end of a user's speech and calculates a response timing having a predetermined interval.

3. 2. The system of claim 1, wherein the generating means is capable of generating at least one emotional reaction such as laughter, surprise, or confusion.

Images