System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving a voice call from a user; means for recording an utterance of the user in the voice call; means for converting the recorded voice data into text data; means for generating an appropriate answer text using a generative model based on the text data; means for converting the generated answer text into voice data; and means for responding to the user with the converted voice data through a call line.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, many people face communication challenges, particularly the elderly, young people who feel lonely, people with dementia, foreign immigrants, and foreign language learners. Appropriate support methods are needed to alleviate the sense of isolation and stress these people feel and to address their diverse needs. However, previous voice call services have struggled to provide appropriate responses in real time or diverse functions. [Means for solving the problem]

[0005] The present invention provides a system that accepts voice calls from users, records the user's speech during the voice call, and converts the recorded voice data into text data. Based on this text data, a generative model is used to generate an appropriate response text, which is then converted into voice data. The converted voice data is then sent back to the user via a telephone line, enabling real-time voice dialogue and resolving communication challenges for the elderly, young people who feel lonely, dementia patients, foreign immigrants, foreign language learners, and others.

[0006] "User" refers to an individual who uses the System.

[0007] "Voice call" refers to telephone communication conducted via voice.

[0008] "Recording" refers to the act of saving sound as digital data.

[0009] "Audio data" refers to digital information of recorded audio.

[0010] "Text data" is voice data converted into character information.

[0011] A "generative model" refers to a machine learning model that generates appropriate output based on input data.

[0012] "Answer text" refers to the textual answer generated by a generative model in response to a user's question.

[0013] A "speech synthesis engine" refers to a system that converts text data into speech data.

[0014] "Call line" refers to the communications infrastructure for making voice calls.

[0015] "AI voice response" refers to a function that uses AI to automatically respond to users. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system that accepts voice calls from users, converts the voice data into text data, generates an appropriate response using a generative model, converts the generated response into voice data, and responds to the user over a telephone line. This system is composed of a server, a terminal, and a user.

[0038] Program processing explanation

[0039] 1. User Initiation and Initial Connection

[0040] The user calls the free call number 0800 or 0120.

[0041] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0042] 2. Collecting audio content and converting it into text

[0043] The user speaks a question or purpose (e.g., "What's the weather today?").

[0044] The terminal records the user's voice in real time.

[0045] The server transmits the recorded voice data to a voice recognition system, which converts the voice into text data.

[0046] 3. Input to the generative AI model and answer generation

[0047] The server sends the textual content to the generative model.

[0048] The generative model generates an appropriate answer text (e.g., "Today's weather is sunny and the temperature is 20 degrees").

[0049] 4. Converting to speech and responding to the user

[0050] The server sends the generated text to a speech synthesis engine to generate voice data.

[0051] The terminal responds to the user with the generated voice data via the telephone line (for example, by replying, "Today's weather is sunny and the temperature is 20 degrees").

[0052] Example: Elderly person asking about the weather

[0053] 1. The user (an elderly woman) calls 0800.

[0054] 2. The server receives the call and responds, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0055] 3. The user says, "What's the weather like today?"

[0056] 4. The device records the user's voice.

[0057] 5. The server sends the recorded data to a speech recognition system and converts it into text data such as "What is the weather today?"

[0058] 6. The server sends the text to the generative model and generates the answer "Today's weather is sunny and the temperature is 20 degrees."

[0059] 7. The server sends this text to a speech synthesis engine to generate voice data.

[0060] 8. The device responds to the user, "Today's weather is sunny and the temperature is 20 degrees."

[0061] This system provides natural voice dialogue in real time and can meet the diverse needs of the elderly, young people who feel lonely, dementia patients, foreign immigrants, foreign language learners, etc. The system is designed to reduce users' stress and improve the quality of communication, and is characterized by its ease and intuitive use.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] A user calls a free call number 0800 or 0120.

[0065] Step 2:

[0066] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0067] Step 3:

[0068] The user speaks a question or purpose (e.g., "What's the weather today?").

[0069] Step 4:

[0070] The device records the user's voice in real time.

[0071] Step 5:

[0072] The server transmits the recorded voice data to a voice recognition system, which converts the voice data into text data.

[0073] The voice data is sent via an API to a text recognition service, which generates the text "What is the weather today?"

[0074] Step 6:

[0075] The server sends the text data to the generative model.

[0076] An API request is made to the generative model to generate an answer based on the text "What is the weather today?"

[0077] Step 7:

[0078] The server receives the generated answer.

[0079] The generative model returns the text "Today's weather is sunny and the temperature is 20 degrees."

[0080] Step 8:

[0081] The server sends the generated text to a speech synthesis engine to generate voice data.

[0082] Send text to the speech synthesis API and get the corresponding audio data.

[0083] Step 9:

[0084] The terminal receives the generated voice data and responds to the user through the communication line.

[0085] The device plays audio data and tells the user, "Today's weather is sunny and the temperature is 20 degrees."

[0086] Step 10:

[0087] If the user has further questions or wants to continue the conversation, they can speak again (e.g., "What's the weather like tomorrow?").

[0088] Step 11:

[0089] The device will record the user's voice again and send it to the server.

[0090] Step 12:

[0091] The server again sends the new voice data to the voice recognition system and converts it into text data.

[0092] The user's utterance "Please tell me the weather tomorrow" is converted into text data.

[0093] Step 13:

[0094] The server sends new text data to the generative model, which generates an appropriate answer.

[0095] The generative model generates the text "It will rain tomorrow and the temperature will be 18 degrees."

[0096] Step 14:

[0097] The server sends the generated text to a speech synthesis engine to generate voice data.

[0098] The speech synthesis API generates speech data based on the text data.

[0099] Step 15:

[0100] The terminal returns the generated voice data to the user via the communication line.

[0101] "It's going to rain tomorrow and the temperature will be 18 degrees," the app replies to the user.

[0102] The above are the processing steps of the invention, and the system provides a user with a prompt and appropriate voice response.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] In modern society, users with diverse needs are increasingly using voice to ask questions and receive advice. Providing prompt and appropriate answers is essential for individual users who require specific support, such as the elderly, people with disabilities, and foreign language learners. However, conventional voice response systems struggle to accurately understand the user's intent and generate appropriate answers, resulting in inconsistent response quality. This has led to issues such as reduced user satisfaction and reduced reliance on continued use.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes: [means for accepting voice communication from a user;] [means for recording what the user says in the voice communication;] [means for converting the recorded voice data into text data;] [means for generating appropriate response text data using a generative AI model based on the text data;] [means for converting the generated response text data into voice data; and [means for responding to the user with the converted voice data via a communication line.] This makes it possible to quickly convert the user's voice data into text, generate appropriate responses using natural language processing technology, and respond as voice.

[0108] "User" refers to an individual human being who interacts with the system via voice communication.

[0109] "Voice communication" refers to the means of sending and receiving voice information over telephone lines or internet lines.

[0110] "Utterance" refers to words, questions, or instructions that a user conveys to the system through voice communication.

[0111] "Recording" refers to the means of preserving audio data conveyed through voice communications.

[0112] "Audio Data" means data collected as sound, in digital or analog form.

[0113] "Character data" refers to digital data that has been converted from voice data into character format.

[0114] A "generative AI model" refers to an artificial neural network model that has been pre-trained with a wide range of knowledge and diverse roles.

[0115] "Answer text data" refers to the text information of the answer generated by the generative AI model based on user input.

[0116] "Converting to audio data" refers to converting character data into a digital format that can be played as sound.

[0117] "Communication lines" refer to telephone lines and internet lines for sending and receiving voice and data.

[0118] "Replying" refers to conveying the system-generated answer to the user.

[0119] The present invention is a system that accepts voice communication from a user, converts the voice data into text data, generates an appropriate response using a generative AI model, converts the response into voice data, and responds to the user via a communication line. A specific embodiment of this system is described below.

[0120] This system consists of a server, a terminal, and a user. The server is equipped with software and hardware to perform multiple functions. Specifically, the server includes a communication module, a voice recognition module, a generative AI model, a text conversion module, and a voice synthesis module.

[0121] Communication Module

[0122] The server uses a communication module to accept voice communications from users. Communications are conducted via telephone lines or internet lines. When a user calls 0800 or 0120 toll-free, the server automatically recognizes the incoming call and begins answering.

[0123] Speech Recognition Module

[0124] The device records the user's speech in real time. The recorded voice data is sent to a server via a communication line. The server then converts the voice data into text data using a voice recognition module (e.g., Google Cloud Speech-to-Text).

[0125] Generative AI Models

[0126] The server generates appropriate response text data using a generative AI model (e.g., a pre-trained artificial neural network model) based on the text data. Specifically, the server inputs the following prompt sentence into the generative AI model:

[0127] "Please tell me the weather today."

[0128] The generative AI model generates an appropriate answer based on the given prompt, for example, "Today's weather is sunny and the temperature is 20 degrees."

[0129] Text-to-text and speech synthesis modules

[0130] The server sends the generated response text data to a text conversion module (e.g., Google Cloud Text-to-Speech) and converts it into voice data. The generated voice data is sent to the terminal via a communication line, and the terminal plays the voice data and responds to the user.

[0131] Example: Elderly person asking about the weather

[0132] 1. The user (an elderly woman) calls 0800.

[0133] 2. The server receives the call and responds, "Hello, welcome to our chat system. How can we help you today?"

[0134] 3. The user says, "What's the weather like today?"

[0135] 4. The device records the user's voice and sends it to the server.

[0136] 5. The server sends the voice data to the voice recognition module and converts it into text data such as "What is the weather today?"

[0137] 6. The server sends the text data to the generative AI model, which generates the answer, "Today's weather is sunny and the temperature is 20 degrees."

[0138] 7. The server sends this text data to a speech synthesis module to generate speech data.

[0139] 8. The device responds to the user, "Today's weather is sunny and the temperature is 20 degrees."

[0140] The system uses specific hardware and software to efficiently process users' voice data and provide prompt and appropriate responses. The system is designed to reduce users' stress and improve the quality of communication.

[0141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0142] Step 1:

[0143] User Initiation and Initial Connection

[0144] A user uses a telephone terminal to call the 0800 or 0120 toll-free number. The input is the user's speech (e.g., "What is the weather today?"). The server receives the incoming call via the communication line and activates the voice response system. The output is a greeting voice saying, "Hello. Welcome to the chat system. How can I help you today?"

[0145] Step 2:

[0146] Collecting audio content and recording audio data

[0147] The user receives a greeting from the server and then speaks out their question or purpose of use. The input is the user's speech. The terminal records this speech in real time and sends it to the server as voice data. Specifically, the terminal converts the voice into digital data through a microphone and sends it to the server via a communication line. The output is the voice data sent to the server.

[0148] Step 3:

[0149] Converting audio data to text

[0150] The server sends the received voice data to a voice recognition module. The input is voice data. The voice recognition module (e.g., Google Cloud Speech-to-Text API) converts the voice data into text data. Specifically, the voice data is analyzed and corresponding text is generated. The output is text data (e.g., "What is the weather today?").

[0151] Step 4:

[0152] Input to generative AI model and answer generation

[0153] The server sends the converted text data to the generative model. The input is text data. The generative AI model (e.g., a pre-trained artificial neural network model) generates an appropriate answer to the user's question. Specifically, the generative AI model analyzes the input text and generates an optimal answer based on the trained database. The output is the answer text (e.g., "Today's weather is sunny and the temperature is 20 degrees").

[0154] Step 5:

[0155] Converting response text to audio data

[0156] The server sends the generated answer text to a speech synthesis module. The input is the answer text. The speech synthesis module (e.g., Google Cloud Text-to-Speech) converts the text into speech data. Specifically, the speech synthesis module converts the text information into speech data using a speech synthesis algorithm. The output is speech data.

[0157] Step 6:

[0158] Replying to the user with voice data

[0159] The terminal plays the voice data received from the server and responds to the user. The input is voice data. The user can hear the response from the terminal. Specifically, the terminal plays the voice data through a speaker. The output is a voice response (e.g., "Today's weather is sunny and the temperature is 20 degrees"), which provides useful information to the user who hears it.

[0160] (Application example 1)

[0161] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0162] In physical stores, there is a demand for an environment where customers can efficiently obtain product information and location information without the assistance of store staff. It can be particularly difficult for customers to smoothly find the product they are looking for during busy times or when there is a shortage of staff. While multilingual guidance is also required, there is no easy way to achieve this. Therefore, it is important to build a system that allows customers to instantly obtain the information they need through their own devices or in-store equipment.

[0163] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0164] In this invention, the server includes means for accepting voice input from a user, means for recording the user's speech in the voice input, and means for converting the recorded voice data into text data. This allows users in a physical store to instantly obtain product information and location information, reducing the burden on store staff and improving customer satisfaction.

[0165] "User" refers to a customer or user who uses the system to input voice data.

[0166] "Voice input" refers to the voice signals that a user speaks to the system.

[0167] "Recording" is the process of recording a user's voice input in real time.

[0168] "Audio data" is data that represents an audio input signal in digital format.

[0169] "Text data" refers to data obtained by converting voice data into character information.

[0170] "Generative model" refers to a neural network model used to generate appropriate answer text based on given text data.

[0171] "Answer text" is text data containing an answer to a user's question created by a generative model.

[0172] The "audio output device" is a device such as a speaker or headphones that converts the generated answer text into audio data and responds to the user.

[0173] "Product information" refers to detailed information about products sold in physical stores, such as price, availability, and characteristics.

[0174] "Location information" refers to information about the location of a specific product within a physical store.

[0175] The term "system" refers to the entire set of mechanisms that includes the above means and provides voice responses to user questions.

[0176] This invention is a smart guidance system that supports customer service in brick-and-mortar stores. When a user asks a question by voice using a smartphone or a robot installed in the store, the voice is converted into text in real time and an appropriate answer is generated using a generative AI model. The generated answer is then converted into voice and returned to the user.

[0177] First, when a user speaks to a smartphone or in-store robot, the voice signal is received by the device, which then converts the voice signal into text data using Google Cloud Speech-to-Text or Amazon Transcribe. This speech recognition system then generates text from the voice signal.

[0178] The server then sends the text data to a generative AI model, such as OpenAI's GPT-4, which generates an appropriate answer. Because the generative AI model has extensive knowledge and learned a variety of roles in advance, it can generate an appropriate answer based on the user's question. For example, it can provide detailed guidance such as, "Take six steps from your current location, turn left, and then take the escalator in front of you to the third floor."

[0179] The generated text is then sent to a speech synthesis engine such as Google Cloud Text-to-Speech or Amazon Polly, where it is converted into voice data. The device then responds to the user through a voice output device (such as a speaker or headphones), allowing the user to receive the appropriate response via voice.

[0180] As a concrete example, consider a scenario in which a user opens a smartphone app and says, "Where is the chocolate section?" In this case, the smartphone app records the audio and transfers it to Google Cloud Speech-to-Text. The server then passes the received text data to GPT-4, which generates an appropriate answer. The generated answer, "The chocolate section is on the second floor, to the left of the food area," is then passed to Google Cloud Text-to-Speech, which generates audio data. Finally, the smartphone app plays the audio data and replies to the user, "The chocolate section is on the second floor, to the left of the food area."

[0181] Examples of prompts:

[0182] Question: 'Where is the drinking water section?'

[0183] Store map data: [List of corner names and locations]

[0184] Expected answer: 'The drinking water corner is in the center of the third floor.'

[0185] This system allows customers to quickly obtain the information they need, reducing the burden on store staff and improving customer satisfaction.

[0186] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0187] Step 1:

[0188] The user inputs voice to a smartphone or a robot in the store. Here, the user asks questions about product information or location information within the store. For example, the user might say, "Where is the chocolate section?" This voice signal is input to the terminal.

[0189] Step 2:

[0190] The device receives the user's voice input and records the speech in real time, which is then stored on the device and sent to a speech recognition system such as Google Cloud Speech-to-Text or Amazon Transcribe for appropriate processing.

[0191] Step 3:

[0192] A speech recognition system takes voice data as input and converts it into text data. Specifically, it analyzes the voice signal and converts its content into written information. This text data has the form of a user question, such as "Where is the chocolate corner?"

[0193] Step 4:

[0194] The server receives text data sent from the speech recognition system and sends it to OpenAI's GPT-4 or other generative AI models, which use extensive pre-trained knowledge to generate appropriate answers for the text data.

[0195] Step 5:

[0196] The generative AI model receives the text data "Where is the chocolate corner?" as input and generates the corresponding answer text "The chocolate corner is on the left side of the food area on the second floor." This answer text is output to the server.

[0197] Step 6:

[0198] The server receives the generated answer text and sends it to a speech synthesis engine such as Google Cloud Text-to-Speech or Amazon Polly, which converts the answer text into audio data.

[0199] Step 7:

[0200] The voice data generated by the speech synthesis engine is sent from the server to the device. The device then plays the voice data through a voice output device (speaker or headphones) and responds to the user. For example, the device might provide a voice prompt saying, "The chocolate section is on the left side of the food area on the second floor."

[0201] In this way, a series of processes from the user's voice input to the answer is carried out automatically, and information is provided quickly and accurately.

[0202] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0203] The present invention provides a system that converts voice data into text data through a user's voice call and generates appropriate responses using a generative model, and further combines it with an emotion engine to provide more advanced voice responses with emotion recognition. This system is composed of a server, a terminal, and a user.

[0204] Program processing explanation

[0205] 1. User Initiation and Initial Connection

[0206] A user calls a free call number 0800 or 0120.

[0207] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0208] 2. Speech content collection and emotion recognition

[0209] The user speaks a question or purpose (e.g., "What's the weather today?").

[0210] The device records the user's voice in real time.

[0211] The server sends the recorded voice data to the emotion engine to recognize the user's emotion. For example, if the user speaks with a slightly anxious voice, the emotion engine will recognize it as "anxiety."

[0212] 3. Converting voice data to text

[0213] The server sends the voice data to a voice recognition system, which converts the voice data into text data.

[0214] 4. Input to the generative AI model and answer generation

[0215] The server sends the text data to the generative model, which then uses the recognition results from the emotion engine to generate a more appropriate answer based on the emotion.

[0216] The generative model generates a response text such as "Today's weather is sunny and the temperature is 20 degrees." If the user feels anxious, they can add an additional comment such as "The weather is nice, so please relax."

[0217] 5. Converting to speech and responding to the user

[0218] The server transmits the generated answer text to a speech synthesis engine to generate speech data.

[0219] The terminal returns the generated voice data to the user via the communication line.

[0220] Example: Elderly person asking about the weather

[0221] 1. A user calls 0800.

[0222] 2. The server receives the call and responds, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0223] 3. The user says in a slightly anxious voice, "What's the weather like today?"

[0224] 4. The device records the user's voice.

[0225] 5. The server sends the recorded data to the emotion engine and recognizes that the user's emotion is "anxiety."

[0226] 6. The server sends the voice data to a voice recognition system and converts it into text data such as "What is the weather today?"

[0227] 7. The server sends the text data and emotion data to the generative model, which generates the answer: "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[0228] 8. The server sends this text to a speech synthesis engine to generate voice data.

[0229] 9. The device responds to the user, "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[0230] This system can significantly improve the quality of communication by recognizing emotions and providing appropriate responses to elderly people, lonely young people, dementia patients, foreign immigrants, foreign language learners, etc. Furthermore, by using the emotion engine, it is expected to reduce the stress and anxiety felt by users.

[0231] The processing flow will be explained below.

[0232] Step 1:

[0233] A user calls a free call number 0800 or 0120.

[0234] Step 2:

[0235] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0236] Step 3:

[0237] The user speaks a question or purpose (e.g., "What's the weather today?").

[0238] Step 4:

[0239] The device records the user's voice in real time.

[0240] Step 5:

[0241] The server sends the recorded voice data to the emotion engine to recognize the user's emotions.

[0242] Here, for example, if the user's voice is slightly trembling, the emotion engine will recognize this as "anxiety."

[0243] Step 6:

[0244] The server sends the recorded data to a voice recognition system, which converts the voice data into text data.

[0245] The voice data is sent via an API to a text recognition service, where it is converted into text such as "What is the weather today?"

[0246] Step 7:

[0247] The server sends the text data and the recognized emotion data to the generative model.

[0248] An API request is made to the generative model to generate an answer based on the text "What is the weather today?" and the emotion data "anxiety."

[0249] Step 8:

[0250] The server receives the generated answer.

[0251] The generative model returns the text, "Today's weather is sunny and the temperature is 20 degrees. Don't worry."

[0252] Step 9:

[0253] The server sends the generated text to a speech synthesis engine to generate voice data.

[0254] Text is sent to a speech synthesis API to generate voice data such as, "Today's weather is sunny and the temperature is 20 degrees. Don't worry."

[0255] Step 10:

[0256] The terminal receives the generated voice data and responds to the user through the communication line.

[0257] The device plays audio data and tells the user, "Today's weather is sunny and the temperature is 20 degrees. Don't worry."

[0258] Step 11:

[0259] If the user has further questions or wants to continue the conversation, they can speak again (e.g., "What's the weather like tomorrow?").

[0260] Step 12:

[0261] The device will record the user's voice again and send it to the server.

[0262] Step 13:

[0263] The server sends new voice data to the emotion engine, which then recognizes the user's emotion again.

[0264] If the new recording data contains, for example, a sense of elation, it will be recognized as "joy."

[0265] Step 14:

[0266] The server again sends the new voice data to the voice recognition system and converts it into text data.

[0267] The user's utterance "Please tell me the weather tomorrow" is converted into text data.

[0268] Step 15:

[0269] The server sends new text and emotion data to the generative model, which then generates an appropriate answer.

[0270] The generative model generates the text "It will rain tomorrow and the temperature will be 18 degrees."

[0271] Step 16:

[0272] The server sends the generated text to a speech synthesis engine to generate voice data.

[0273] The speech synthesis API generates speech data based on the text data.

[0274] Step 17:

[0275] The terminal returns the generated voice data to the user via the communication line.

[0276] "It's going to rain tomorrow and the temperature will be 18 degrees," the app replies to the user.

[0277] In this way, the emotion engine can be used to reduce the stress and anxiety felt by the user and provide a more appropriate response.

[0278] Example 2

[0279] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0280] Current voice response systems simply convert voice data into text and generate responses without recognizing the user's emotions, making it impossible to alleviate the stress and anxiety experienced by users. This is particularly insufficient for users who require emotional consideration, such as the elderly, users who feel lonely, and people with dementia. To solve this problem, an advanced system is needed that can recognize the user's emotions and provide responses that take them into account.

[0281] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0282] In this invention, the server includes: [means for accepting a voice call from a user;] [means for recording the user's speech during the voice call;] [means for transmitting the recorded voice data to an emotion recognition engine to recognize the user's emotions;] [means for converting the voice data into text data;] [means for generating an appropriate answer text using a generative AI model based on the text data and emotion data;] [means for converting the generated answer text into voice data; and [means for replying to the user with the converted voice data via a call line.] This makes it possible to recognize the user's emotions and generate an appropriate voice response corresponding to them.

[0283] "User" means an individual who accesses the System, makes voice calls and uses the Service.

[0284] "Voice call" refers to a communication method for sending and receiving voice data via telephone lines or Internet lines.

[0285] "Server" refers to a computer system for accepting voice calls and processing data.

[0286] "Terminal" refers to a device used by a user to make voice calls, such as a telephone or smartphone.

[0287] "Speech recognition system" refers to a software and hardware system for converting voice data into text data.

[0288] "Text data" refers to a data format in which voice data is converted into a string of characters.

[0289] An "emotion recognition engine" refers to a system that analyzes and recognizes a user's emotions from voice and text data.

[0290] "Emotion data" refers to emotional information obtained as a result of analysis by an emotion recognition engine.

[0291] A "generative AI model" refers to a neural network model that has been pre-trained with a wide range of knowledge and diverse roles.

[0292] "Answer text" refers to the response content generated by the generative AI model based on the user's question, expressed in text format.

[0293] "Speech synthesis engine" refers to a system for converting text data into speech data.

[0294] "Telephone line" refers to the communication line required to make a voice call.

[0295] A "prompt sentence" refers to text containing input instructions for a generative AI model.

[0296] The present invention is a system that converts voice data into text data through a user's voice call and generates an appropriate voice response based on the text data. This system includes multiple components, including a server, a terminal, and a user. The server accepts the voice call, converts the user's voice into text, uses a generative AI model to generate a response text, and converts it into voice to respond to the user.

[0297] Hardware and software used

[0298] Server: A high-performance computer system that runs speech recognition, generative AI models, emotion recognition engines, and speech synthesis engines.

[0299] Terminal: A device (telephone, smartphone, etc.) used by a user to make voice calls.

[0300] Speech recognition system: Software that converts voice data into text (e.g., Google Cloud Speech-to-Text API, Amazon Transcribe).

[0301] Emotion recognition engine: A system that analyzes user emotions from voice data (e.g., IBM Watson Tone Analyzer, Microsoft Azure Text Analytics).

[0302] Generative AI models: Neural network models pre-trained with a wide range of knowledge and diverse roles (e.g., GPT-3).

[0303] Speech synthesis engine: A system that converts text data into speech data (e.g., Google Text-to-Speech API, Amazon Polly).

[0304] System operation explanation

[0305] 1. User Initiation and Initial Connection

[0306] When a user calls the 0800 or 0120 toll-free number, the server receives the call and responds using a speech synthesis engine, saying, "Hello. Welcome to our chatty AI system. How can we help you today?"

[0307] 2. Speech content collection and emotion recognition

[0308] When a user speaks a question or request, the device records the voice in real time and sends it to the server. The server then sends the voice data to an emotion recognition engine, which analyzes the user's emotions. For example, if a user says, "What's the weather like today?" in a slightly anxious voice, the emotion recognition engine will identify the emotion "anxiety" from the voice.

[0309] 3. Converting voice data to text

[0310] The server converts the voice data into text data using a speech recognition system, which returns text such as "What's the weather like today?"

[0311] 4. Input to the generative AI model and answer generation

[0312] The server inputs text data and emotion data into the generative AI model to generate an appropriate answer. An example of a prompt sentence is, "The user's question is 'What's the weather like today?' The user is expressing the emotion 'anxiety.' The generated answer should include elements that reassure the user." The generative AI model generates the answer text, "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[0313] 5. Converting to speech and responding to the user

[0314] The server sends the generated response text to the speech synthesis engine, which generates voice data. The device responds to the user, saying, "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[0315] In this way, the present invention realizes a system that can recognize the user's emotions and provide an appropriate response accordingly, thereby reducing the stress and anxiety felt by the user.

[0316] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0317] Step 1:

[0318] User Initiation and Initial Connection

[0319] Input: A user calls a free call number 0800 or 0120.

[0320] How it works: The server receives the call and manages the incoming call using the telephone system interface.

[0321] Data processing: The server detects the incoming call and uses a speech synthesis engine to generate voice data as an initial response.

[0322] Output: Speech to the user: "Hello, welcome to our chatty AI system. How can we help you today?"

[0323] Step 2:

[0324] Speech content collection and emotion recognition

[0325] Input: The user speaks a question or request.

[0326] How it works: The device records the user's voice in real time and sends it to the server.

[0327] Data processing: The server sends the recorded voice data to an emotion recognition engine to analyze the user's emotions.

[0328] Output: Emotion data (e.g., "anxiety") analyzed by the emotion recognition engine is sent back to the server.

[0329] Step 3:

[0330] Converting audio data to text

[0331] Input: Recorded audio data.

[0332] How it works: The server sends voice data to a speech recognition system, which converts it into text data such as "What is the weather today?"

[0333] Data processing: The voice data is converted into text data using the speech recognition system's algorithm.

[0334] Output: Text data is sent back to the server.

[0335] Step 4:

[0336] Input to generative AI model and answer generation

[0337] Input: Text data and sentiment data.

[0338] How it works: The server inputs text data and emotion data into the generative AI model. It then generates a prompt sentence for the generative AI model.

[0339] Data processing: The generative AI model generates the optimal answer text based on the input data.

[0340] Output: The answer text (e.g. "Today's weather is sunny and the temperature is 20 degrees. Please relax.") is sent back to the server.

[0341] Step 5:

[0342] Converting to speech and responding to the user

[0343] Input: The generated answer text.

[0344] How it works: The server sends the answer text to a speech synthesis engine, which generates audio data.

[0345] Data processing: The text data is converted into voice data by a speech synthesis engine.

[0346] Output: The generated voice data is sent to the user over the telephone line.

[0347] Through the above processing steps, the system can provide an advanced response that takes into account the user's emotions and improve user satisfaction.

[0348] (Application example 2)

[0349] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0350] The present invention relates to a system that can recognize a user's emotions through voice calls and provide an appropriate response based on that emotion in real time. Conventional voice recognition systems have been unable to properly recognize a user's emotions and provide a response based on that emotion, which poses challenges, particularly in emergencies and situations where a response that reduces stress is required. Simply converting speech to text and returning a standard response often fails to fully address the user's emotions and state. This challenge must be addressed, particularly in security services, where prompt and emotionally responsive instructions are required.

[0351] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting a voice call from a user, means for recording the user's speech during the voice call, means for converting the recorded voice data into text data, means for generating an appropriate response text based on the text data using a generative model, means for converting the generated response text into voice data, means for responding to the user with the converted voice data via a communication line, and means for including an emotion engine that recognizes the user's emotions and generating a response corresponding to the emotions based on the emotion. The combination of emotion recognition and generative models in this invention makes it possible to provide users with a prompt and appropriate response. Particularly in security services, when a security guard on-site feels anxious in an emergency, the quality of on-site response can be improved by recognizing the guard's emotions and providing a response that includes instructions to relax.

[0352] A "user" is a person or organization that places a voice call and makes a question or request to the system.

[0353] "Voice call" refers to voice communication conducted over telephone lines or internet lines.

[0354] "Recording" is the act of storing audio data in digital form in real time.

[0355] "Audio data" refers to digital audio signals collected from an input device such as a microphone.

[0356] "Text data" is character string information converted from voice data by a voice recognition system.

[0357] A "generative model" is a neural network model that has been pre-trained with a wide range of knowledge and diverse roles, and is designed to generate appropriate answers based on input text data.

[0358] "Answer text" is text information that includes an answer to the user, generated by the generative model based on input text data.

[0359] A "speech synthesis engine" is a system that converts text data into speech data that humans can understand.

[0360] The "emotion engine" is a system that analyzes the user's voice data and recognizes the emotions contained in the voice in real time.

[0361] A "response" is an answer or instruction that the system provides to the user, and is provided as voice data.

[0362] A "communication line" is a communication means for sending and receiving voice data, and includes an internet line and a telephone line.

[0363] The system for implementing this invention is composed of a server, a terminal, and a user. Next, details of each component and their interactions will be explained.

[0364] Hardware and Software Configuration

[0365] 1. Server

[0366] The server is the central system that handles overall processing and data management. It uses Amazon EC2 instances.

[0367] The server is installed with software to perform the following functions:

[0368] Speech recognition system: Converts voice data into text data using the Google Speech-to-Text API.

[0369] Generative AI model: OpenAI GPT-4 is used to generate appropriate answers based on text data.

[0370] Emotion Engine: Uses the Affectiva API to recognize emotions from the user's voice in real time.

[0371] Speech synthesis engine: Amazon Polly is used to convert the generated response text into voice data.

[0372] 2. Terminal

[0373] The terminal is a device that collects audio and mediates communication between the user and the server. In this example, we use smart glasses (e.g., Google Glass Enterprise Edition 2).

[0374] It has the ability to collect audio using a built-in microphone and send the audio data to a server.

[0375] 3. Users

[0376] Users are security guards and other professionals who wear smart glasses to use the system, especially in on-site emergency response situations.

[0377] System Operation

[0378] 1. Audio collection

[0379] The user inputs voice through the smart glasses, which is collected by a built-in microphone.

[0380] For example, a security guard might say, "I need help!"

[0381] 2. Sending and analyzing audio data

[0382] The voice data collected from the terminal is sent to the server in real time.

[0383] The server first converts the audio data into text using a speech recognition system, in this case using the Google Speech-to-Text API.

[0384] 3. Emotion recognition

[0385] The voice data is sent in parallel to the emotion engine, which analyzes the user's emotion and uses the Affectiva API to identify the emotion as "impatience."

[0386] 4. Generating the Response

[0387] The server uses a generative AI model to generate an appropriate response based on text data and emotion recognition data. Through OpenAI GPT-4, the response generated is, "Assistance is on its way. Please remain calm."

[0388] 5. Audio Conversion

[0389] The generated answer text is converted into voice data through a speech synthesis engine, in this case Amazon Polly.

[0390] The generated voice data is returned to the user through the terminal.

[0391] Specific examples

[0392] For example, when a security guard at an emergency scene says "I need help!" in a panic, the response is generated in the following steps:

[0393] Prompt Sentence Examples

[0394] The user says "I need help!" with an impatient tone. Generate an appropriate response.

[0395] The server sends this prompt to the generative AI model, which generates a response saying, "Assistance is on its way. Please remain calm," and replies to the user via voice data. Through this process, the security guard on-site receives a prompt, emotionally appropriate response.

[0396] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0397] Step 1:

[0398] The user puts on the smart glasses and starts voice input. When reporting an emergency situation, the user speaks, "I need help!" The built-in microphone of the smart glasses collects voice data and starts recording in real time. The input is voice data, and the output is the voice data captured by the terminal. This voice data is sent to the next step in its original form.

[0399] Step 2:

[0400] The device sends the collected voice data to the server. Specifically, the voice data is uploaded to the server via an internet connection. The input is the voice data collected by the device, and the output is the voice data sent to the server. The server prepares this voice data for processing.

[0401] Step 3:

[0402] The server receives the voice data and inputs it into the voice recognition system. Here, the Google Speech-to-Text API is used to convert the voice data into text data. Specific operations include analyzing the voice waveform and identifying phonemes. The input is the voice data sent to the server, and the output is the converted text data.

[0403] Step 4:

[0404] The server sends the voice data to the emotion engine at the same time as the text data. The emotion engine (Affectiva API) analyzes the voice data in real time and recognizes the user's emotion. Specifically, it analyzes the tone, speed, and intensity of the voice. The input is the voice data, and the output is the recognized emotion data. In this case, the emotion recognized is "impatience."

[0405] Step 5:

[0406] The server inputs text data and emotion data into the generative AI model. OpenAI GPT-4 is used to generate a prompt sentence and generate an appropriate answer based on it. For example, a prompt might be generated such as, "The user said, 'I need help!' with an expression of impatience. Please generate an appropriate response." The input is text data and emotion data, and the output is the generated answer text. The generated response might be something like, "Assistance will arrive shortly. Please remain calm."

[0407] Step 6:

[0408] The server sends the generated answer text to a speech synthesis engine. Amazon Polly is used to convert the text data into speech data. Specifically, a speech waveform is generated based on the text information and output as natural-sounding speech. The input is the generated answer text, and the output is speech data.

[0409] Step 7:

[0410] The server sends the converted voice data to the terminal. The converted voice data is returned to the user through the telephone line. Specifically, the voice is played from the speaker of the smart glasses. The input is the voice data, and the output is the voice that the user can hear. The user can hear the response, "Assistance will arrive shortly. Please remain calm."

[0411] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0412] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0413] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0414] [Second embodiment]

[0415] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0416] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0417] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0418] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0419] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0420] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0421] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0422] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0423] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0424] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0425] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0426] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0427] The present invention is a system that accepts voice calls from users, converts the voice data into text data, generates an appropriate response using a generative model, converts the generated response into voice data, and responds to the user over a telephone line. This system is composed of a server, a terminal, and a user.

[0428] Program processing explanation

[0429] 1. User Initiation and Initial Connection

[0430] The user calls the free call number 0800 or 0120.

[0431] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0432] 2. Collecting audio content and converting it into text

[0433] The user speaks a question or purpose (e.g., "What's the weather today?").

[0434] The terminal records the user's voice in real time.

[0435] The server transmits the recorded voice data to a voice recognition system, which converts the voice into text data.

[0436] 3. Input to the generative AI model and answer generation

[0437] The server sends the textual content to the generative model.

[0438] The generative model generates an appropriate answer text (e.g., "Today's weather is sunny and the temperature is 20 degrees").

[0439] 4. Converting to speech and responding to the user

[0440] The server sends the generated text to a speech synthesis engine to generate voice data.

[0441] The terminal responds to the user with the generated voice data via the telephone line (for example, by replying, "Today's weather is sunny and the temperature is 20 degrees").

[0442] Example: Elderly person asking about the weather

[0443] 1. The user (an elderly woman) calls 0800.

[0444] 2. The server receives the call and responds, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0445] 3. The user says, "What's the weather like today?"

[0446] 4. The device records the user's voice.

[0447] 5. The server sends the recorded data to a speech recognition system and converts it into text data such as "What is the weather today?"

[0448] 6. The server sends the text to the generative model and generates the answer "Today's weather is sunny and the temperature is 20 degrees."

[0449] 7. The server sends this text to a speech synthesis engine to generate voice data.

[0450] 8. The device responds to the user, "Today's weather is sunny and the temperature is 20 degrees."

[0451] This system provides natural voice dialogue in real time and can meet the diverse needs of the elderly, young people who feel lonely, dementia patients, foreign immigrants, foreign language learners, etc. The system is designed to reduce users' stress and improve the quality of communication, and is characterized by its ease and intuitive use.

[0452] The processing flow will be explained below.

[0453] Step 1:

[0454] A user calls a free call number 0800 or 0120.

[0455] Step 2:

[0456] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0457] Step 3:

[0458] The user speaks a question or purpose (e.g., "What's the weather today?").

[0459] Step 4:

[0460] The device records the user's voice in real time.

[0461] Step 5:

[0462] The server transmits the recorded voice data to a voice recognition system, which converts the voice data into text data.

[0463] The voice data is sent via an API to a text recognition service, which generates the text "What is the weather today?"

[0464] Step 6:

[0465] The server sends the text data to the generative model.

[0466] An API request is made to the generative model to generate an answer based on the text "What is the weather today?"

[0467] Step 7:

[0468] The server receives the generated answer.

[0469] The generative model returns the text "Today's weather is sunny and the temperature is 20 degrees."

[0470] Step 8:

[0471] The server sends the generated text to a speech synthesis engine to generate voice data.

[0472] Send text to the speech synthesis API and get the corresponding audio data.

[0473] Step 9:

[0474] The terminal receives the generated voice data and responds to the user through the communication line.

[0475] The device plays audio data and tells the user, "Today's weather is sunny and the temperature is 20 degrees."

[0476] Step 10:

[0477] If the user has further questions or wants to continue the conversation, they can speak again (e.g., "What's the weather like tomorrow?").

[0478] Step 11:

[0479] The device will record the user's voice again and send it to the server.

[0480] Step 12:

[0481] The server again sends the new voice data to the voice recognition system and converts it into text data.

[0482] The user's utterance "Please tell me the weather tomorrow" is converted into text data.

[0483] Step 13:

[0484] The server sends new text data to the generative model, which generates an appropriate answer.

[0485] The generative model generates the text "It will rain tomorrow and the temperature will be 18 degrees."

[0486] Step 14:

[0487] The server sends the generated text to a speech synthesis engine to generate voice data.

[0488] The speech synthesis API generates speech data based on the text data.

[0489] Step 15:

[0490] The terminal returns the generated voice data to the user via the communication line.

[0491] "It's going to rain tomorrow and the temperature will be 18 degrees," the app replies to the user.

[0492] The above are the processing steps of the invention, and the system provides a user with a prompt and appropriate voice response.

[0493] Example 1

[0494] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0495] In modern society, users with diverse needs are increasingly using voice to ask questions and receive advice. Providing prompt and appropriate answers is essential for individual users who require specific support, such as the elderly, people with disabilities, and foreign language learners. However, conventional voice response systems struggle to accurately understand the user's intent and generate appropriate answers, resulting in inconsistent response quality. This has led to issues such as reduced user satisfaction and reduced reliance on continued use.

[0496] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0497] In this invention, the server includes: [means for accepting voice communication from a user;] [means for recording what the user says in the voice communication;] [means for converting the recorded voice data into text data;] [means for generating appropriate response text data using a generative AI model based on the text data;] [means for converting the generated response text data into voice data; and [means for responding to the user with the converted voice data via a communication line.] This makes it possible to quickly convert the user's voice data into text, generate appropriate responses using natural language processing technology, and respond as voice.

[0498] "User" refers to an individual human being who interacts with the system via voice communication.

[0499] "Voice communication" refers to the means of sending and receiving voice information over telephone lines or internet lines.

[0500] "Utterance" refers to words, questions, or instructions that a user conveys to the system through voice communication.

[0501] "Recording" refers to the means of preserving audio data conveyed through voice communications.

[0502] "Audio Data" means data collected as sound, in digital or analog form.

[0503] "Character data" refers to digital data that has been converted from voice data into character format.

[0504] A "generative AI model" refers to an artificial neural network model that has been pre-trained with a wide range of knowledge and diverse roles.

[0505] "Answer text data" refers to the text information of the answer generated by the generative AI model based on user input.

[0506] "Converting to audio data" refers to converting character data into a digital format that can be played as sound.

[0507] "Communication lines" refer to telephone lines and internet lines for sending and receiving voice and data.

[0508] "Replying" refers to conveying the system-generated answer to the user.

[0509] The present invention is a system that accepts voice communication from a user, converts the voice data into text data, generates an appropriate response using a generative AI model, converts the response into voice data, and responds to the user via a communication line. A specific embodiment of this system is described below.

[0510] This system consists of a server, a terminal, and a user. The server is equipped with software and hardware to perform multiple functions. Specifically, the server includes a communication module, a voice recognition module, a generative AI model, a text conversion module, and a voice synthesis module.

[0511] Communication Module

[0512] The server uses a communication module to accept voice communications from users. Communications are conducted via telephone lines or internet lines. When a user calls 0800 or 0120 toll-free, the server automatically recognizes the incoming call and begins answering.

[0513] Speech Recognition Module

[0514] The device records the user's speech in real time. The recorded voice data is sent to a server via a communication line. The server then converts the voice data into text data using a voice recognition module (e.g., Google Cloud Speech-to-Text).

[0515] Generative AI Models

[0516] The server generates appropriate response text data using a generative AI model (e.g., a pre-trained artificial neural network model) based on the text data. Specifically, the server inputs the following prompt sentence into the generative AI model:

[0517] "Please tell me the weather today."

[0518] The generative AI model generates an appropriate answer based on the given prompt, for example, "Today's weather is sunny and the temperature is 20 degrees."

[0519] Text-to-text and speech synthesis modules

[0520] The server sends the generated response text data to a text conversion module (e.g., Google Cloud Text-to-Speech) and converts it into voice data. The generated voice data is sent to the terminal via a communication line, and the terminal plays the voice data and responds to the user.

[0521] Example: Elderly person asking about the weather

[0522] 1. The user (an elderly woman) calls 0800.

[0523] 2. The server receives the call and responds, "Hello, welcome to our chat system. How can we help you today?"

[0524] 3. The user says, "What's the weather like today?"

[0525] 4. The device records the user's voice and sends it to the server.

[0526] 5. The server sends the voice data to the voice recognition module and converts it into text data such as "What is the weather today?"

[0527] 6. The server sends the text data to the generative AI model, which generates the answer, "Today's weather is sunny and the temperature is 20 degrees."

[0528] 7. The server sends this text data to a speech synthesis module to generate speech data.

[0529] 8. The device responds to the user, "Today's weather is sunny and the temperature is 20 degrees."

[0530] The system uses specific hardware and software to efficiently process users' voice data and provide prompt and appropriate responses. The system is designed to reduce users' stress and improve the quality of communication.

[0531] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0532] Step 1:

[0533] User Initiation and Initial Connection

[0534] A user uses a telephone terminal to call the 0800 or 0120 toll-free number. The input is the user's speech (e.g., "What is the weather today?"). The server receives the incoming call via the communication line and activates the voice response system. The output is a greeting voice saying, "Hello. Welcome to the chat system. How can I help you today?"

[0535] Step 2:

[0536] Collecting audio content and recording audio data

[0537] The user receives a greeting from the server and then speaks out their question or purpose of use. The input is the user's speech. The terminal records this speech in real time and sends it to the server as voice data. Specifically, the terminal converts the voice into digital data through a microphone and sends it to the server via a communication line. The output is the voice data sent to the server.

[0538] Step 3:

[0539] Converting audio data to text

[0540] The server sends the received voice data to a voice recognition module. The input is voice data. The voice recognition module (e.g., Google Cloud Speech-to-Text API) converts the voice data into text data. Specifically, the voice data is analyzed and corresponding text is generated. The output is text data (e.g., "What is the weather today?").

[0541] Step 4:

[0542] Input to generative AI model and answer generation

[0543] The server sends the converted text data to the generative model. The input is text data. The generative AI model (e.g., a pre-trained artificial neural network model) generates an appropriate answer to the user's question. Specifically, the generative AI model analyzes the input text and generates an optimal answer based on the trained database. The output is the answer text (e.g., "Today's weather is sunny and the temperature is 20 degrees").

[0544] Step 5:

[0545] Converting response text to audio data

[0546] The server sends the generated answer text to a speech synthesis module. The input is the answer text. The speech synthesis module (e.g., Google Cloud Text-to-Speech) converts the text into speech data. Specifically, the speech synthesis module converts the text information into speech data using a speech synthesis algorithm. The output is speech data.

[0547] Step 6:

[0548] Replying to the user with voice data

[0549] The terminal plays the voice data received from the server and responds to the user. The input is voice data. The user can hear the response from the terminal. Specifically, the terminal plays the voice data through a speaker. The output is a voice response (e.g., "Today's weather is sunny and the temperature is 20 degrees"), which provides useful information to the user who hears it.

[0550] (Application example 1)

[0551] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0552] In physical stores, there is a demand for an environment where customers can efficiently obtain product information and location information without the assistance of store staff. It can be particularly difficult for customers to smoothly find the product they are looking for during busy times or when there is a shortage of staff. While multilingual guidance is also required, there is no easy way to achieve this. Therefore, it is important to build a system that allows customers to instantly obtain the information they need through their own devices or in-store equipment.

[0553] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0554] In this invention, the server includes means for accepting voice input from a user, means for recording the user's speech in the voice input, and means for converting the recorded voice data into text data. This allows users in a physical store to instantly obtain product information and location information, reducing the burden on store staff and improving customer satisfaction.

[0555] "User" refers to a customer or user who uses the system to input voice data.

[0556] "Voice input" refers to the voice signals that a user speaks to the system.

[0557] "Recording" is the process of recording a user's voice input in real time.

[0558] "Audio data" is data that represents an audio input signal in digital format.

[0559] "Text data" refers to data obtained by converting voice data into character information.

[0560] "Generative model" refers to a neural network model used to generate appropriate answer text based on given text data.

[0561] "Answer text" is text data containing an answer to a user's question created by a generative model.

[0562] The "audio output device" is a device such as a speaker or headphones that converts the generated answer text into audio data and responds to the user.

[0563] "Product information" refers to detailed information about products sold in physical stores, such as price, availability, and characteristics.

[0564] "Location information" refers to information about the location of a specific product within a physical store.

[0565] The term "system" refers to the entire set of mechanisms that includes the above means and provides voice responses to user questions.

[0566] This invention is a smart guidance system that supports customer service in brick-and-mortar stores. When a user asks a question by voice using a smartphone or a robot installed in the store, the voice is converted into text in real time and an appropriate answer is generated using a generative AI model. The generated answer is then converted into voice and returned to the user.

[0567] First, when a user speaks to a smartphone or in-store robot, the voice signal is received by the device, which then converts the voice signal into text data using Google Cloud Speech-to-Text or Amazon Transcribe. This speech recognition system then generates text from the voice signal.

[0568] The server then sends the text data to a generative AI model, such as OpenAI's GPT-4, which generates an appropriate answer. Because the generative AI model has extensive knowledge and learned a variety of roles in advance, it can generate an appropriate answer based on the user's question. For example, it can provide detailed guidance such as, "Take six steps from your current location, turn left, and then take the escalator in front of you to the third floor."

[0569] The generated text is then sent to a speech synthesis engine such as Google Cloud Text-to-Speech or Amazon Polly, where it is converted into voice data. The device then responds to the user through a voice output device (such as a speaker or headphones), allowing the user to receive the appropriate response via voice.

[0570] As a concrete example, consider a scenario in which a user opens a smartphone app and says, "Where is the chocolate section?" In this case, the smartphone app records the audio and transfers it to Google Cloud Speech-to-Text. The server then passes the received text data to GPT-4, which generates an appropriate answer. The generated answer, "The chocolate section is on the second floor, to the left of the food area," is then passed to Google Cloud Text-to-Speech, which generates audio data. Finally, the smartphone app plays the audio data and replies to the user, "The chocolate section is on the second floor, to the left of the food area."

[0571] Examples of prompts:

[0572] Question: 'Where is the drinking water section?'

[0573] Store map data: [List of corner names and locations]

[0574] Expected answer: 'The drinking water corner is in the center of the third floor.'

[0575] This system allows customers to quickly obtain the information they need, reducing the burden on store staff and improving customer satisfaction.

[0576] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0577] Step 1:

[0578] The user inputs voice to a smartphone or a robot in the store. Here, the user asks questions about product information or location information within the store. For example, the user might say, "Where is the chocolate section?" This voice signal is input to the terminal.

[0579] Step 2:

[0580] The device receives the user's voice input and records the speech in real time, which is then stored on the device and sent to a speech recognition system such as Google Cloud Speech-to-Text or Amazon Transcribe for appropriate processing.

[0581] Step 3:

[0582] A speech recognition system takes voice data as input and converts it into text data. Specifically, it analyzes the voice signal and converts its content into written information. This text data has the form of a user question, such as "Where is the chocolate corner?"

[0583] Step 4:

[0584] The server receives text data sent from the speech recognition system and sends it to OpenAI's GPT-4 or other generative AI models, which use extensive pre-trained knowledge to generate appropriate answers for the text data.

[0585] Step 5:

[0586] The generative AI model receives the text data "Where is the chocolate corner?" as input and generates the corresponding answer text "The chocolate corner is on the left side of the food area on the second floor." This answer text is output to the server.

[0587] Step 6:

[0588] The server receives the generated answer text and sends it to a speech synthesis engine such as Google Cloud Text-to-Speech or Amazon Polly, which converts the answer text into audio data.

[0589] Step 7:

[0590] The voice data generated by the speech synthesis engine is sent from the server to the device. The device then plays the voice data through a voice output device (speaker or headphones) and responds to the user. For example, the device might provide a voice prompt saying, "The chocolate section is on the left side of the food area on the second floor."

[0591] In this way, a series of processes from the user's voice input to the answer is carried out automatically, and information is provided quickly and accurately.

[0592] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0593] The present invention provides a system that converts voice data into text data through a user's voice call and generates appropriate responses using a generative model, and further combines it with an emotion engine to provide more advanced voice responses with emotion recognition. This system is composed of a server, a terminal, and a user.

[0594] Program processing explanation

[0595] 1. User Initiation and Initial Connection

[0596] A user calls a free call number 0800 or 0120.

[0597] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0598] 2. Speech content collection and emotion recognition

[0599] The user speaks a question or purpose (e.g., "What's the weather today?").

[0600] The device records the user's voice in real time.

[0601] The server sends the recorded voice data to the emotion engine to recognize the user's emotion. For example, if the user speaks with a slightly anxious voice, the emotion engine will recognize it as "anxiety."

[0602] 3. Converting voice data to text

[0603] The server sends the voice data to a voice recognition system, which converts the voice data into text data.

[0604] 4. Input to the generative AI model and answer generation

[0605] The server sends the text data to the generative model, which then uses the recognition results from the emotion engine to generate a more appropriate answer based on the emotion.

[0606] The generative model generates a response text such as "Today's weather is sunny and the temperature is 20 degrees." If the user feels anxious, they can add an additional comment such as "The weather is nice, so please relax."

[0607] 5. Converting to speech and responding to the user

[0608] The server transmits the generated answer text to a speech synthesis engine to generate speech data.

[0609] The terminal returns the generated voice data to the user via the communication line.

[0610] Example: Elderly person asking about the weather

[0611] 1. A user calls 0800.

[0612] 2. The server receives the call and responds, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0613] 3. The user says in a slightly anxious voice, "What's the weather like today?"

[0614] 4. The device records the user's voice.

[0615] 5. The server sends the recorded data to the emotion engine and recognizes that the user's emotion is "anxiety."

[0616] 6. The server sends the voice data to a voice recognition system and converts it into text data such as "What is the weather today?"

[0617] 7. The server sends the text data and emotion data to the generative model, which generates the answer: "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[0618] 8. The server sends this text to a speech synthesis engine to generate voice data.

[0619] 9. The device responds to the user, "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[0620] This system can significantly improve the quality of communication by recognizing emotions and providing appropriate responses to elderly people, lonely young people, dementia patients, foreign immigrants, foreign language learners, etc. Furthermore, by using the emotion engine, it is expected to reduce the stress and anxiety felt by users.

[0621] The processing flow will be explained below.

[0622] Step 1:

[0623] A user calls a free call number 0800 or 0120.

[0624] Step 2:

[0625] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0626] Step 3:

[0627] The user speaks a question or purpose (e.g., "What's the weather today?").

[0628] Step 4:

[0629] The device records the user's voice in real time.

[0630] Step 5:

[0631] The server sends the recorded voice data to the emotion engine to recognize the user's emotions.

[0632] Here, for example, if the user's voice is slightly trembling, the emotion engine will recognize this as "anxiety."

[0633] Step 6:

[0634] The server sends the recorded data to a voice recognition system, which converts the voice data into text data.

[0635] The voice data is sent via an API to a text recognition service, where it is converted into text such as "What is the weather today?"

[0636] Step 7:

[0637] The server sends the text data and the recognized emotion data to the generative model.

[0638] An API request is made to the generative model to generate an answer based on the text "What is the weather today?" and the emotion data "anxiety."

[0639] Step 8:

[0640] The server receives the generated answer.

[0641] The generative model returns the text, "Today's weather is sunny and the temperature is 20 degrees. Don't worry."

[0642] Step 9:

[0643] The server sends the generated text to a speech synthesis engine to generate voice data.

[0644] Text is sent to a speech synthesis API to generate voice data such as, "Today's weather is sunny and the temperature is 20 degrees. Don't worry."

[0645] Step 10:

[0646] The terminal receives the generated voice data and responds to the user through the communication line.

[0647] The device plays audio data and tells the user, "Today's weather is sunny and the temperature is 20 degrees. Don't worry."

[0648] Step 11:

[0649] If the user has further questions or wants to continue the conversation, they can speak again (e.g., "What's the weather like tomorrow?").

[0650] Step 12:

[0651] The device will record the user's voice again and send it to the server.

[0652] Step 13:

[0653] The server sends new voice data to the emotion engine, which then recognizes the user's emotion again.

[0654] If the new recording data contains, for example, a sense of elation, it will be recognized as "joy."

[0655] Step 14:

[0656] The server again sends the new voice data to the voice recognition system and converts it into text data.

[0657] The user's utterance "Please tell me the weather tomorrow" is converted into text data.

[0658] Step 15:

[0659] The server sends new text and emotion data to the generative model, which then generates an appropriate answer.

[0660] The generative model generates the text "It will rain tomorrow and the temperature will be 18 degrees."

[0661] Step 16:

[0662] The server sends the generated text to a speech synthesis engine to generate voice data.

[0663] The speech synthesis API generates speech data based on the text data.

[0664] Step 17:

[0665] The terminal returns the generated voice data to the user via the communication line.

[0666] "It's going to rain tomorrow and the temperature will be 18 degrees," the app replies to the user.

[0667] In this way, the emotion engine can be used to reduce the stress and anxiety felt by the user and provide a more appropriate response.

[0668] Example 2

[0669] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0670] Current voice response systems simply convert voice data into text and generate responses without recognizing the user's emotions, making it impossible to alleviate the stress and anxiety experienced by users. This is particularly insufficient for users who require emotional consideration, such as the elderly, users who feel lonely, and people with dementia. To solve this problem, an advanced system is needed that can recognize the user's emotions and provide responses that take them into account.

[0671] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0672] In this invention, the server includes: [means for accepting a voice call from a user;] [means for recording the user's speech during the voice call;] [means for transmitting the recorded voice data to an emotion recognition engine to recognize the user's emotions;] [means for converting the voice data into text data;] [means for generating an appropriate answer text using a generative AI model based on the text data and emotion data;] [means for converting the generated answer text into voice data; and [means for replying to the user with the converted voice data via a call line.] This makes it possible to recognize the user's emotions and generate an appropriate voice response corresponding to them.

[0673] "User" means an individual who accesses the System, makes voice calls and uses the Service.

[0674] "Voice call" refers to a communication method for sending and receiving voice data via telephone lines or Internet lines.

[0675] "Server" refers to a computer system for accepting voice calls and processing data.

[0676] "Terminal" refers to a device used by a user to make voice calls, such as a telephone or smartphone.

[0677] "Speech recognition system" refers to a software and hardware system for converting voice data into text data.

[0678] "Text data" refers to a data format in which voice data is converted into a string of characters.

[0679] An "emotion recognition engine" refers to a system that analyzes and recognizes a user's emotions from voice and text data.

[0680] "Emotion data" refers to emotional information obtained as a result of analysis by an emotion recognition engine.

[0681] A "generative AI model" refers to a neural network model that has been pre-trained with a wide range of knowledge and diverse roles.

[0682] "Answer text" refers to the response content generated by the generative AI model based on the user's question, expressed in text format.

[0683] "Speech synthesis engine" refers to a system for converting text data into speech data.

[0684] "Telephone line" refers to the communication line required to make a voice call.

[0685] A "prompt sentence" refers to text containing input instructions for a generative AI model.

[0686] The present invention is a system that converts voice data into text data through a user's voice call and generates an appropriate voice response based on the text data. This system includes multiple components, including a server, a terminal, and a user. The server accepts the voice call, converts the user's voice into text, uses a generative AI model to generate a response text, and converts it into voice to respond to the user.

[0687] Hardware and software used

[0688] Server: A high-performance computer system that runs speech recognition, generative AI models, emotion recognition engines, and speech synthesis engines.

[0689] Terminal: A device (telephone, smartphone, etc.) used by a user to make voice calls.

[0690] Speech recognition system: Software that converts voice data into text (e.g., Google Cloud Speech-to-Text API, Amazon Transcribe).

[0691] Emotion recognition engine: A system that analyzes user emotions from voice data (e.g., IBM Watson Tone Analyzer, Microsoft Azure Text Analytics).

[0692] Generative AI models: Neural network models pre-trained with a wide range of knowledge and diverse roles (e.g., GPT-3).

[0693] Speech synthesis engine: A system that converts text data into speech data (e.g., Google Text-to-Speech API, Amazon Polly).

[0694] System operation explanation

[0695] 1. User Initiation and Initial Connection

[0696] When a user calls the 0800 or 0120 toll-free number, the server receives the call and responds using a speech synthesis engine, saying, "Hello. Welcome to our chatty AI system. How can we help you today?"

[0697] 2. Speech content collection and emotion recognition

[0698] When a user speaks a question or request, the device records the voice in real time and sends it to the server. The server then sends the voice data to an emotion recognition engine, which analyzes the user's emotions. For example, if a user says, "What's the weather like today?" in a slightly anxious voice, the emotion recognition engine will identify the emotion "anxiety" from the voice.

[0699] 3. Converting voice data to text

[0700] The server converts the voice data into text data using a speech recognition system, which returns text such as "What's the weather like today?"

[0701] 4. Input to the generative AI model and answer generation

[0702] The server inputs text data and emotion data into the generative AI model to generate an appropriate answer. An example of a prompt sentence is, "The user's question is 'What's the weather like today?' The user is expressing the emotion 'anxiety.' The generated answer should include elements that reassure the user." The generative AI model generates the answer text, "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[0703] 5. Converting to speech and responding to the user

[0704] The server sends the generated response text to the speech synthesis engine, which generates voice data. The device responds to the user, saying, "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[0705] In this way, the present invention realizes a system that can recognize the user's emotions and provide an appropriate response accordingly, thereby reducing the stress and anxiety felt by the user.

[0706] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0707] Step 1:

[0708] User Initiation and Initial Connection

[0709] Input: A user calls a free call number 0800 or 0120.

[0710] How it works: The server receives the call and manages the incoming call using the telephone system interface.

[0711] Data processing: The server detects the incoming call and uses a speech synthesis engine to generate voice data as an initial response.

[0712] Output: Speech to the user: "Hello, welcome to our chatty AI system. How can we help you today?"

[0713] Step 2:

[0714] Speech content collection and emotion recognition

[0715] Input: The user speaks a question or request.

[0716] How it works: The device records the user's voice in real time and sends it to the server.

[0717] Data processing: The server sends the recorded voice data to an emotion recognition engine to analyze the user's emotions.

[0718] Output: Emotion data (e.g., "anxiety") analyzed by the emotion recognition engine is sent back to the server.

[0719] Step 3:

[0720] Converting audio data to text

[0721] Input: Recorded audio data.

[0722] How it works: The server sends voice data to a speech recognition system, which converts it into text data such as "What is the weather today?"

[0723] Data processing: The voice data is converted into text data using the speech recognition system's algorithm.

[0724] Output: Text data is sent back to the server.

[0725] Step 4:

[0726] Input to generative AI model and answer generation

[0727] Input: Text data and sentiment data.

[0728] How it works: The server inputs text data and emotion data into the generative AI model. It then generates a prompt sentence for the generative AI model.

[0729] Data processing: The generative AI model generates the optimal answer text based on the input data.

[0730] Output: The answer text (e.g. "Today's weather is sunny and the temperature is 20 degrees. Please relax.") is sent back to the server.

[0731] Step 5:

[0732] Converting to speech and responding to the user

[0733] Input: The generated answer text.

[0734] How it works: The server sends the answer text to a speech synthesis engine, which generates audio data.

[0735] Data processing: The text data is converted into voice data by a speech synthesis engine.

[0736] Output: The generated voice data is sent to the user over the telephone line.

[0737] Through the above processing steps, the system can provide an advanced response that takes into account the user's emotions and improve user satisfaction.

[0738] (Application example 2)

[0739] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0740] The present invention relates to a system that can recognize a user's emotions through voice calls and provide an appropriate response based on that emotion in real time. Conventional voice recognition systems have been unable to properly recognize a user's emotions and provide a response based on that emotion, which poses challenges, particularly in emergencies and situations where a response that reduces stress is required. Simply converting speech to text and returning a standard response often fails to fully address the user's emotions and state. This challenge must be addressed, particularly in security services, where prompt and emotionally responsive instructions are required.

[0741] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting a voice call from a user, means for recording the user's speech during the voice call, means for converting the recorded voice data into text data, means for generating an appropriate response text based on the text data using a generative model, means for converting the generated response text into voice data, means for responding to the user with the converted voice data via a communication line, and means for including an emotion engine that recognizes the user's emotions and generating a response corresponding to the emotions based on the emotion. The combination of emotion recognition and generative models in this invention makes it possible to provide users with a prompt and appropriate response. Particularly in security services, when a security guard on-site feels anxious in an emergency, the quality of on-site response can be improved by recognizing the guard's emotions and providing a response that includes instructions to relax.

[0742] A "user" is a person or organization that places a voice call and makes a question or request to the system.

[0743] "Voice call" refers to voice communication conducted over telephone lines or internet lines.

[0744] "Recording" is the act of storing audio data in digital form in real time.

[0745] "Audio data" refers to digital audio signals collected from an input device such as a microphone.

[0746] "Text data" is character string information converted from voice data by a voice recognition system.

[0747] A "generative model" is a neural network model that has been pre-trained with a wide range of knowledge and diverse roles, and is designed to generate appropriate answers based on input text data.

[0748] "Answer text" is text information that includes an answer to the user, generated by the generative model based on input text data.

[0749] A "speech synthesis engine" is a system that converts text data into speech data that humans can understand.

[0750] The "emotion engine" is a system that analyzes the user's voice data and recognizes the emotions contained in the voice in real time.

[0751] A "response" is an answer or instruction that the system provides to the user, and is provided as voice data.

[0752] A "communication line" is a communication means for sending and receiving voice data, and includes an internet line and a telephone line.

[0753] The system for implementing this invention is composed of a server, a terminal, and a user. Next, details of each component and their interactions will be explained.

[0754] Hardware and Software Configuration

[0755] 1. Server

[0756] The server is the central system that handles overall processing and data management. It uses Amazon EC2 instances.

[0757] The server is installed with software to perform the following functions:

[0758] Speech recognition system: Converts voice data into text data using the Google Speech-to-Text API.

[0759] Generative AI model: OpenAI GPT-4 is used to generate appropriate answers based on text data.

[0760] Emotion Engine: Uses the Affectiva API to recognize emotions from the user's voice in real time.

[0761] Speech synthesis engine: Amazon Polly is used to convert the generated response text into voice data.

[0762] 2. Terminal

[0763] The terminal is a device that collects audio and mediates communication between the user and the server. In this example, we use smart glasses (e.g., Google Glass Enterprise Edition 2).

[0764] It has the ability to collect audio using a built-in microphone and send the audio data to a server.

[0765] 3. Users

[0766] Users are security guards and other professionals who wear smart glasses to use the system, especially in on-site emergency response situations.

[0767] System Operation

[0768] 1. Audio collection

[0769] The user inputs voice through the smart glasses, which is collected by a built-in microphone.

[0770] For example, a security guard might say, "I need help!"

[0771] 2. Sending and analyzing audio data

[0772] The voice data collected from the terminal is sent to the server in real time.

[0773] The server first converts the audio data into text using a speech recognition system, in this case using the Google Speech-to-Text API.

[0774] 3. Emotion recognition

[0775] The voice data is sent in parallel to the emotion engine, which analyzes the user's emotion and uses the Affectiva API to identify the emotion as "impatience."

[0776] 4. Generating the Response

[0777] The server uses a generative AI model to generate an appropriate response based on text data and emotion recognition data. Through OpenAI GPT-4, the response generated is, "Assistance is on its way. Please remain calm."

[0778] 5. Audio Conversion

[0779] The generated answer text is converted into voice data through a speech synthesis engine, in this case Amazon Polly.

[0780] The generated voice data is returned to the user through the terminal.

[0781] Specific examples

[0782] For example, when a security guard at an emergency scene says "I need help!" in a panic, the response is generated in the following steps:

[0783] Prompt Sentence Examples

[0784] The user says "I need help!" with an impatient tone. Generate an appropriate response.

[0785] The server sends this prompt to the generative AI model, which generates a response saying, "Assistance is on its way. Please remain calm," and replies to the user via voice data. Through this process, the security guard on-site receives a prompt, emotionally appropriate response.

[0786] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0787] Step 1:

[0788] The user puts on the smart glasses and starts voice input. When reporting an emergency situation, the user speaks, "I need help!" The built-in microphone of the smart glasses collects voice data and starts recording in real time. The input is voice data, and the output is the voice data captured by the terminal. This voice data is sent to the next step in its original form.

[0789] Step 2:

[0790] The device sends the collected voice data to the server. Specifically, the voice data is uploaded to the server via an internet connection. The input is the voice data collected by the device, and the output is the voice data sent to the server. The server prepares this voice data for processing.

[0791] Step 3:

[0792] The server receives the voice data and inputs it into the voice recognition system. Here, the Google Speech-to-Text API is used to convert the voice data into text data. Specific operations include analyzing the voice waveform and identifying phonemes. The input is the voice data sent to the server, and the output is the converted text data.

[0793] Step 4:

[0794] The server sends the voice data to the emotion engine at the same time as the text data. The emotion engine (Affectiva API) analyzes the voice data in real time and recognizes the user's emotion. Specifically, it analyzes the tone, speed, and intensity of the voice. The input is the voice data, and the output is the recognized emotion data. In this case, the emotion recognized is "impatience."

[0795] Step 5:

[0796] The server inputs text data and emotion data into the generative AI model. OpenAI GPT-4 is used to generate a prompt sentence and generate an appropriate answer based on it. For example, a prompt might be generated such as, "The user said, 'I need help!' with an expression of impatience. Please generate an appropriate response." The input is text data and emotion data, and the output is the generated answer text. The generated response might be something like, "Assistance will arrive shortly. Please remain calm."

[0797] Step 6:

[0798] The server sends the generated answer text to a speech synthesis engine. Amazon Polly is used to convert the text data into speech data. Specifically, a speech waveform is generated based on the text information and output as natural-sounding speech. The input is the generated answer text, and the output is speech data.

[0799] Step 7:

[0800] The server sends the converted voice data to the terminal. The converted voice data is returned to the user through the telephone line. Specifically, the voice is played from the speaker of the smart glasses. The input is the voice data, and the output is the voice that the user can hear. The user can hear the response, "Assistance will arrive shortly. Please remain calm."

[0801] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0802] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0803] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0804] [Third embodiment]

[0805] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0806] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0807] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0808] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0809] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0810] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0811] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0812] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0813] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0814] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0815] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0816] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0817] The present invention is a system that accepts voice calls from users, converts the voice data into text data, generates an appropriate response using a generative model, converts the generated response into voice data, and responds to the user over a telephone line. This system is composed of a server, a terminal, and a user.

[0818] Program processing explanation

[0819] 1. User Initiation and Initial Connection

[0820] The user calls the free call number 0800 or 0120.

[0821] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0822] 2. Collecting audio content and converting it into text

[0823] The user speaks a question or purpose (e.g., "What's the weather today?").

[0824] The terminal records the user's voice in real time.

[0825] The server transmits the recorded voice data to a voice recognition system, which converts the voice into text data.

[0826] 3. Input to the generative AI model and answer generation

[0827] The server sends the textual content to the generative model.

[0828] The generative model generates an appropriate answer text (e.g., "Today's weather is sunny and the temperature is 20 degrees").

[0829] 4. Converting to speech and responding to the user

[0830] The server sends the generated text to a speech synthesis engine to generate voice data.

[0831] The terminal responds to the user with the generated voice data via the telephone line (for example, by replying, "Today's weather is sunny and the temperature is 20 degrees").

[0832] Example: Elderly person asking about the weather

[0833] 1. The user (an elderly woman) calls 0800.

[0834] 2. The server receives the call and responds, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0835] 3. The user says, "What's the weather like today?"

[0836] 4. The device records the user's voice.

[0837] 5. The server sends the recorded data to a speech recognition system and converts it into text data such as "What is the weather today?"

[0838] 6. The server sends the text to the generative model and generates the answer "Today's weather is sunny and the temperature is 20 degrees."

[0839] 7. The server sends this text to a speech synthesis engine to generate voice data.

[0840] 8. The device responds to the user, "Today's weather is sunny and the temperature is 20 degrees."

[0841] This system provides natural voice dialogue in real time and can meet the diverse needs of the elderly, young people who feel lonely, dementia patients, foreign immigrants, foreign language learners, etc. The system is designed to reduce users' stress and improve the quality of communication, and is characterized by its ease and intuitive use.

[0842] The processing flow will be explained below.

[0843] Step 1:

[0844] A user calls a free call number 0800 or 0120.

[0845] Step 2:

[0846] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0847] Step 3:

[0848] The user speaks a question or purpose (e.g., "What's the weather today?").

[0849] Step 4:

[0850] The device records the user's voice in real time.

[0851] Step 5:

[0852] The server transmits the recorded voice data to a voice recognition system, which converts the voice data into text data.

[0853] The voice data is sent via an API to a text recognition service, which generates the text "What is the weather today?"

[0854] Step 6:

[0855] The server sends the text data to the generative model.

[0856] An API request is made to the generative model to generate an answer based on the text "What is the weather today?"

[0857] Step 7:

[0858] The server receives the generated answer.

[0859] The generative model returns the text "Today's weather is sunny and the temperature is 20 degrees."

[0860] Step 8:

[0861] The server sends the generated text to a speech synthesis engine to generate voice data.

[0862] Send text to the speech synthesis API and get the corresponding audio data.

[0863] Step 9:

[0864] The terminal receives the generated voice data and responds to the user through the communication line.

[0865] The device plays audio data and tells the user, "Today's weather is sunny and the temperature is 20 degrees."

[0866] Step 10:

[0867] If the user has further questions or wants to continue the conversation, they can speak again (e.g., "What's the weather like tomorrow?").

[0868] Step 11:

[0869] The device will record the user's voice again and send it to the server.

[0870] Step 12:

[0871] The server again sends the new voice data to the voice recognition system and converts it into text data.

[0872] The user's utterance "Please tell me the weather tomorrow" is converted into text data.

[0873] Step 13:

[0874] The server sends new text data to the generative model, which generates an appropriate answer.

[0875] The generative model generates the text "It will rain tomorrow and the temperature will be 18 degrees."

[0876] Step 14:

[0877] The server sends the generated text to a speech synthesis engine to generate voice data.

[0878] The speech synthesis API generates speech data based on the text data.

[0879] Step 15:

[0880] The terminal returns the generated voice data to the user via the communication line.

[0881] "It's going to rain tomorrow and the temperature will be 18 degrees," the app replies to the user.

[0882] The above are the processing steps of the invention, and the system provides a user with a prompt and appropriate voice response.

[0883] Example 1

[0884] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0885] In modern society, users with diverse needs are increasingly using voice to ask questions and receive advice. Providing prompt and appropriate answers is essential for individual users who require specific support, such as the elderly, people with disabilities, and foreign language learners. However, conventional voice response systems struggle to accurately understand the user's intent and generate appropriate answers, resulting in inconsistent response quality. This has led to issues such as reduced user satisfaction and reduced reliance on continued use.

[0886] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0887] In this invention, the server includes: [means for accepting voice communication from a user;] [means for recording what the user says in the voice communication;] [means for converting the recorded voice data into text data;] [means for generating appropriate response text data using a generative AI model based on the text data;] [means for converting the generated response text data into voice data; and [means for responding to the user with the converted voice data via a communication line.] This makes it possible to quickly convert the user's voice data into text, generate appropriate responses using natural language processing technology, and respond as voice.

[0888] "User" refers to an individual human being who interacts with the system via voice communication.

[0889] "Voice communication" refers to the means of sending and receiving voice information over telephone lines or internet lines.

[0890] "Utterance" refers to words, questions, or instructions that a user conveys to the system through voice communication.

[0891] "Recording" refers to the means of preserving audio data conveyed through voice communications.

[0892] "Audio Data" means data collected as sound, in digital or analog form.

[0893] "Character data" refers to digital data that has been converted from voice data into character format.

[0894] A "generative AI model" refers to an artificial neural network model that has been pre-trained with a wide range of knowledge and diverse roles.

[0895] "Answer text data" refers to the text information of the answer generated by the generative AI model based on user input.

[0896] "Converting to audio data" refers to converting character data into a digital format that can be played as sound.

[0897] "Communication lines" refer to telephone lines and internet lines for sending and receiving voice and data.

[0898] "Replying" refers to conveying the system-generated answer to the user.

[0899] The present invention is a system that accepts voice communication from a user, converts the voice data into text data, generates an appropriate response using a generative AI model, converts the response into voice data, and responds to the user via a communication line. A specific embodiment of this system is described below.

[0900] This system consists of a server, a terminal, and a user. The server is equipped with software and hardware to perform multiple functions. Specifically, the server includes a communication module, a voice recognition module, a generative AI model, a text conversion module, and a voice synthesis module.

[0901] Communication Module

[0902] The server uses a communication module to accept voice communications from users. Communications are conducted via telephone lines or internet lines. When a user calls 0800 or 0120 toll-free, the server automatically recognizes the incoming call and begins answering.

[0903] Speech Recognition Module

[0904] The device records the user's speech in real time. The recorded voice data is sent to a server via a communication line. The server then converts the voice data into text data using a voice recognition module (e.g., Google Cloud Speech-to-Text).

[0905] Generative AI Models

[0906] The server generates appropriate response text data using a generative AI model (e.g., a pre-trained artificial neural network model) based on the text data. Specifically, the server inputs the following prompt sentence into the generative AI model:

[0907] "Please tell me the weather today."

[0908] The generative AI model generates an appropriate answer based on the given prompt, for example, "Today's weather is sunny and the temperature is 20 degrees."

[0909] Text-to-text and speech synthesis modules

[0910] The server sends the generated response text data to a text conversion module (e.g., Google Cloud Text-to-Speech) and converts it into voice data. The generated voice data is sent to the terminal via a communication line, and the terminal plays the voice data and responds to the user.

[0911] Example: Elderly person asking about the weather

[0912] 1. The user (an elderly woman) calls 0800.

[0913] 2. The server receives the call and responds, "Hello, welcome to our chat system. How can we help you today?"

[0914] 3. The user says, "What's the weather like today?"

[0915] 4. The device records the user's voice and sends it to the server.

[0916] 5. The server sends the voice data to the voice recognition module and converts it into text data such as "What is the weather today?"

[0917] 6. The server sends the text data to the generative AI model, which generates the answer, "Today's weather is sunny and the temperature is 20 degrees."

[0918] 7. The server sends this text data to a speech synthesis module to generate speech data.

[0919] 8. The device responds to the user, "Today's weather is sunny and the temperature is 20 degrees."

[0920] The system uses specific hardware and software to efficiently process users' voice data and provide prompt and appropriate responses. The system is designed to reduce users' stress and improve the quality of communication.

[0921] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0922] Step 1:

[0923] User Initiation and Initial Connection

[0924] A user uses a telephone terminal to call the 0800 or 0120 toll-free number. The input is the user's speech (e.g., "What is the weather today?"). The server receives the incoming call via the communication line and activates the voice response system. The output is a greeting voice saying, "Hello. Welcome to the chat system. How can I help you today?"

[0925] Step 2:

[0926] Collecting audio content and recording audio data

[0927] The user receives a greeting from the server and then speaks out their question or purpose of use. The input is the user's speech. The terminal records this speech in real time and sends it to the server as voice data. Specifically, the terminal converts the voice into digital data through a microphone and sends it to the server via a communication line. The output is the voice data sent to the server.

[0928] Step 3:

[0929] Converting audio data to text

[0930] The server sends the received voice data to a voice recognition module. The input is voice data. The voice recognition module (e.g., Google Cloud Speech-to-Text API) converts the voice data into text data. Specifically, the voice data is analyzed and corresponding text is generated. The output is text data (e.g., "What is the weather today?").

[0931] Step 4:

[0932] Input to generative AI model and answer generation

[0933] The server sends the converted text data to the generative model. The input is text data. The generative AI model (e.g., a pre-trained artificial neural network model) generates an appropriate answer to the user's question. Specifically, the generative AI model analyzes the input text and generates an optimal answer based on the trained database. The output is the answer text (e.g., "Today's weather is sunny and the temperature is 20 degrees").

[0934] Step 5:

[0935] Converting response text to audio data

[0936] The server sends the generated answer text to a speech synthesis module. The input is the answer text. The speech synthesis module (e.g., Google Cloud Text-to-Speech) converts the text into speech data. Specifically, the speech synthesis module converts the text information into speech data using a speech synthesis algorithm. The output is speech data.

[0937] Step 6:

[0938] Replying to the user with voice data

[0939] The terminal plays the voice data received from the server and responds to the user. The input is voice data. The user can hear the response from the terminal. Specifically, the terminal plays the voice data through a speaker. The output is a voice response (e.g., "Today's weather is sunny and the temperature is 20 degrees"), which provides useful information to the user who hears it.

[0940] (Application example 1)

[0941] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0942] In physical stores, there is a demand for an environment where customers can efficiently obtain product information and location information without the assistance of store staff. It can be particularly difficult for customers to smoothly find the product they are looking for during busy times or when there is a shortage of staff. While multilingual guidance is also required, there is no easy way to achieve this. Therefore, it is important to build a system that allows customers to instantly obtain the information they need through their own devices or in-store equipment.

[0943] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0944] In this invention, the server includes means for accepting voice input from a user, means for recording the user's speech in the voice input, and means for converting the recorded voice data into text data. This allows users in a physical store to instantly obtain product information and location information, reducing the burden on store staff and improving customer satisfaction.

[0945] "User" refers to a customer or user who uses the system to input voice data.

[0946] "Voice input" refers to the voice signals that a user speaks to the system.

[0947] "Recording" is the process of recording a user's voice input in real time.

[0948] "Audio data" is data that represents an audio input signal in digital format.

[0949] "Text data" refers to data obtained by converting voice data into character information.

[0950] "Generative model" refers to a neural network model used to generate appropriate answer text based on given text data.

[0951] "Answer text" is text data containing an answer to a user's question created by a generative model.

[0952] The "audio output device" is a device such as a speaker or headphones that converts the generated answer text into audio data and responds to the user.

[0953] "Product information" refers to detailed information about products sold in physical stores, such as price, availability, and characteristics.

[0954] "Location information" refers to information about the location of a specific product within a physical store.

[0955] The term "system" refers to the entire set of mechanisms that includes the above means and provides voice responses to user questions.

[0956] This invention is a smart guidance system that supports customer service in brick-and-mortar stores. When a user asks a question by voice using a smartphone or a robot installed in the store, the voice is converted into text in real time and an appropriate answer is generated using a generative AI model. The generated answer is then converted into voice and returned to the user.

[0957] First, when a user speaks to a smartphone or in-store robot, the voice signal is received by the device, which then converts the voice signal into text data using Google Cloud Speech-to-Text or Amazon Transcribe. This speech recognition system then generates text from the voice signal.

[0958] The server then sends the text data to a generative AI model, such as OpenAI's GPT-4, which generates an appropriate answer. Because the generative AI model has extensive knowledge and learned a variety of roles in advance, it can generate an appropriate answer based on the user's question. For example, it can provide detailed guidance such as, "Take six steps from your current location, turn left, and then take the escalator in front of you to the third floor."

[0959] The generated text is then sent to a speech synthesis engine such as Google Cloud Text-to-Speech or Amazon Polly, where it is converted into voice data. The device then responds to the user through a voice output device (such as a speaker or headphones), allowing the user to receive the appropriate response via voice.

[0960] As a concrete example, consider a scenario in which a user opens a smartphone app and says, "Where is the chocolate section?" In this case, the smartphone app records the audio and transfers it to Google Cloud Speech-to-Text. The server then passes the received text data to GPT-4, which generates an appropriate answer. The generated answer, "The chocolate section is on the second floor, to the left of the food area," is then passed to Google Cloud Text-to-Speech, which generates audio data. Finally, the smartphone app plays the audio data and replies to the user, "The chocolate section is on the second floor, to the left of the food area."

[0961] Examples of prompts:

[0962] Question: 'Where is the drinking water section?'

[0963] Store map data: [List of corner names and locations]

[0964] Expected answer: 'The drinking water corner is in the center of the third floor.'

[0965] This system allows customers to quickly obtain the information they need, reducing the burden on store staff and improving customer satisfaction.

[0966] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0967] Step 1:

[0968] The user inputs voice to a smartphone or a robot in the store. Here, the user asks questions about product information or location information within the store. For example, the user might say, "Where is the chocolate section?" This voice signal is input to the terminal.

[0969] Step 2:

[0970] The device receives the user's voice input and records the speech in real time, which is then stored on the device and sent to a speech recognition system such as Google Cloud Speech-to-Text or Amazon Transcribe for appropriate processing.

[0971] Step 3:

[0972] A speech recognition system takes voice data as input and converts it into text data. Specifically, it analyzes the voice signal and converts its content into written information. This text data has the form of a user question, such as "Where is the chocolate corner?"

[0973] Step 4:

[0974] The server receives text data sent from the speech recognition system and sends it to OpenAI's GPT-4 or other generative AI models, which use extensive pre-trained knowledge to generate appropriate answers for the text data.

[0975] Step 5:

[0976] The generative AI model receives the text data "Where is the chocolate corner?" as input and generates the corresponding answer text "The chocolate corner is on the left side of the food area on the second floor." This answer text is output to the server.

[0977] Step 6:

[0978] The server receives the generated answer text and sends it to a speech synthesis engine such as Google Cloud Text-to-Speech or Amazon Polly, which converts the answer text into audio data.

[0979] Step 7:

[0980] The voice data generated by the speech synthesis engine is sent from the server to the device. The device then plays the voice data through a voice output device (speaker or headphones) and responds to the user. For example, the device might provide a voice prompt saying, "The chocolate section is on the left side of the food area on the second floor."

[0981] In this way, a series of processes from the user's voice input to the answer is carried out automatically, and information is provided quickly and accurately.

[0982] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0983] The present invention provides a system that converts voice data into text data through a user's voice call and generates appropriate responses using a generative model, and further combines it with an emotion engine to provide more advanced voice responses with emotion recognition. This system is composed of a server, a terminal, and a user.

[0984] Program processing explanation

[0985] 1. User Initiation and Initial Connection

[0986] A user calls a free call number 0800 or 0120.

[0987] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[0988] 2. Speech content collection and emotion recognition

[0989] The user speaks a question or purpose (e.g., "What's the weather today?").

[0990] The device records the user's voice in real time.

[0991] The server sends the recorded voice data to the emotion engine to recognize the user's emotion. For example, if the user speaks with a slightly anxious voice, the emotion engine will recognize it as "anxiety."

[0992] 3. Converting voice data to text

[0993] The server sends the voice data to a voice recognition system, which converts the voice data into text data.

[0994] 4. Input to the generative AI model and answer generation

[0995] The server sends the text data to the generative model, which then uses the recognition results from the emotion engine to generate a more appropriate answer based on the emotion.

[0996] The generative model generates a response text such as "Today's weather is sunny and the temperature is 20 degrees." If the user feels anxious, they can add an additional comment such as "The weather is nice, so please relax."

[0997] 5. Converting to speech and responding to the user

[0998] The server transmits the generated answer text to a speech synthesis engine to generate speech data.

[0999] The terminal returns the generated voice data to the user via the communication line.

[1000] Example: Elderly person asking about the weather

[1001] 1. A user calls 0800.

[1002] 2. The server receives the call and responds, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[1003] 3. The user says in a slightly anxious voice, "What's the weather like today?"

[1004] 4. The device records the user's voice.

[1005] 5. The server sends the recorded data to the emotion engine and recognizes that the user's emotion is "anxiety."

[1006] 6. The server sends the voice data to a voice recognition system and converts it into text data such as "What is the weather today?"

[1007] 7. The server sends the text data and emotion data to the generative model, which generates the answer: "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[1008] 8. The server sends this text to a speech synthesis engine to generate voice data.

[1009] 9. The device responds to the user, "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[1010] This system can significantly improve the quality of communication by recognizing emotions and providing appropriate responses to elderly people, lonely young people, dementia patients, foreign immigrants, foreign language learners, etc. Furthermore, by using the emotion engine, it is expected to reduce the stress and anxiety felt by users.

[1011] The processing flow will be explained below.

[1012] Step 1:

[1013] A user calls a free call number 0800 or 0120.

[1014] Step 2:

[1015] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[1016] Step 3:

[1017] The user speaks a question or purpose (e.g., "What's the weather today?").

[1018] Step 4:

[1019] The device records the user's voice in real time.

[1020] Step 5:

[1021] The server sends the recorded voice data to the emotion engine to recognize the user's emotions.

[1022] Here, for example, if the user's voice is slightly trembling, the emotion engine will recognize this as "anxiety."

[1023] Step 6:

[1024] The server sends the recorded data to a voice recognition system, which converts the voice data into text data.

[1025] The voice data is sent via an API to a text recognition service, where it is converted into text such as "What is the weather today?"

[1026] Step 7:

[1027] The server sends the text data and the recognized emotion data to the generative model.

[1028] An API request is made to the generative model to generate an answer based on the text "What is the weather today?" and the emotion data "anxiety."

[1029] Step 8:

[1030] The server receives the generated answer.

[1031] The generative model returns the text, "Today's weather is sunny and the temperature is 20 degrees. Don't worry."

[1032] Step 9:

[1033] The server sends the generated text to a speech synthesis engine to generate voice data.

[1034] Text is sent to a speech synthesis API to generate voice data such as, "Today's weather is sunny and the temperature is 20 degrees. Don't worry."

[1035] Step 10:

[1036] The terminal receives the generated voice data and responds to the user through the communication line.

[1037] The device plays audio data and tells the user, "Today's weather is sunny and the temperature is 20 degrees. Don't worry."

[1038] Step 11:

[1039] If the user has further questions or wants to continue the conversation, they can speak again (e.g., "What's the weather like tomorrow?").

[1040] Step 12:

[1041] The device will record the user's voice again and send it to the server.

[1042] Step 13:

[1043] The server sends new voice data to the emotion engine, which then recognizes the user's emotion again.

[1044] If the new recording data contains, for example, a sense of elation, it will be recognized as "joy."

[1045] Step 14:

[1046] The server again sends the new voice data to the voice recognition system and converts it into text data.

[1047] The user's utterance "Please tell me the weather tomorrow" is converted into text data.

[1048] Step 15:

[1049] The server sends new text and emotion data to the generative model, which then generates an appropriate answer.

[1050] The generative model generates the text "It will rain tomorrow and the temperature will be 18 degrees."

[1051] Step 16:

[1052] The server sends the generated text to a speech synthesis engine to generate voice data.

[1053] The speech synthesis API generates speech data based on the text data.

[1054] Step 17:

[1055] The terminal returns the generated voice data to the user via the communication line.

[1056] "It's going to rain tomorrow and the temperature will be 18 degrees," the app replies to the user.

[1057] In this way, the emotion engine can be used to reduce the stress and anxiety felt by the user and provide a more appropriate response.

[1058] Example 2

[1059] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1060] Current voice response systems simply convert voice data into text and generate responses without recognizing the user's emotions, making it impossible to alleviate the stress and anxiety experienced by users. This is particularly insufficient for users who require emotional consideration, such as the elderly, users who feel lonely, and people with dementia. To solve this problem, an advanced system is needed that can recognize the user's emotions and provide responses that take them into account.

[1061] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1062] In this invention, the server includes: [means for accepting a voice call from a user;] [means for recording the user's speech during the voice call;] [means for transmitting the recorded voice data to an emotion recognition engine to recognize the user's emotions;] [means for converting the voice data into text data;] [means for generating an appropriate answer text using a generative AI model based on the text data and emotion data;] [means for converting the generated answer text into voice data; and [means for replying to the user with the converted voice data via a call line.] This makes it possible to recognize the user's emotions and generate an appropriate voice response corresponding to them.

[1063] "User" means an individual who accesses the System, makes voice calls and uses the Service.

[1064] "Voice call" refers to a communication method for sending and receiving voice data via telephone lines or Internet lines.

[1065] "Server" refers to a computer system for accepting voice calls and processing data.

[1066] "Terminal" refers to a device used by a user to make voice calls, such as a telephone or smartphone.

[1067] "Speech recognition system" refers to a software and hardware system for converting voice data into text data.

[1068] "Text data" refers to a data format in which voice data is converted into a string of characters.

[1069] An "emotion recognition engine" refers to a system that analyzes and recognizes a user's emotions from voice and text data.

[1070] "Emotion data" refers to emotional information obtained as a result of analysis by an emotion recognition engine.

[1071] A "generative AI model" refers to a neural network model that has been pre-trained with a wide range of knowledge and diverse roles.

[1072] "Answer text" refers to the response content generated by the generative AI model based on the user's question, expressed in text format.

[1073] "Speech synthesis engine" refers to a system for converting text data into speech data.

[1074] "Telephone line" refers to the communication line required to make a voice call.

[1075] A "prompt sentence" refers to text containing input instructions for a generative AI model.

[1076] The present invention is a system that converts voice data into text data through a user's voice call and generates an appropriate voice response based on the text data. This system includes multiple components, including a server, a terminal, and a user. The server accepts the voice call, converts the user's voice into text, uses a generative AI model to generate a response text, and converts it into voice to respond to the user.

[1077] Hardware and software used

[1078] Server: A high-performance computer system that runs speech recognition, generative AI models, emotion recognition engines, and speech synthesis engines.

[1079] Terminal: A device (telephone, smartphone, etc.) used by a user to make voice calls.

[1080] Speech recognition system: Software that converts voice data into text (e.g., Google Cloud Speech-to-Text API, Amazon Transcribe).

[1081] Emotion recognition engine: A system that analyzes user emotions from voice data (e.g., IBM Watson Tone Analyzer, Microsoft Azure Text Analytics).

[1082] Generative AI models: Neural network models pre-trained with a wide range of knowledge and diverse roles (e.g., GPT-3).

[1083] Speech synthesis engine: A system that converts text data into speech data (e.g., Google Text-to-Speech API, Amazon Polly).

[1084] System operation explanation

[1085] 1. User Initiation and Initial Connection

[1086] When a user calls the 0800 or 0120 toll-free number, the server receives the call and responds using a speech synthesis engine, saying, "Hello. Welcome to our chatty AI system. How can we help you today?"

[1087] 2. Speech content collection and emotion recognition

[1088] When a user speaks a question or request, the device records the voice in real time and sends it to the server. The server then sends the voice data to an emotion recognition engine, which analyzes the user's emotions. For example, if a user says, "What's the weather like today?" in a slightly anxious voice, the emotion recognition engine will identify the emotion "anxiety" from the voice.

[1089] 3. Converting voice data to text

[1090] The server converts the voice data into text data using a speech recognition system, which returns text such as "What's the weather like today?"

[1091] 4. Input to the generative AI model and answer generation

[1092] The server inputs text data and emotion data into the generative AI model to generate an appropriate answer. An example of a prompt sentence is, "The user's question is 'What's the weather like today?' The user is expressing the emotion 'anxiety.' The generated answer should include elements that reassure the user." The generative AI model generates the answer text, "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[1093] 5. Converting to speech and responding to the user

[1094] The server sends the generated response text to the speech synthesis engine, which generates voice data. The device responds to the user, saying, "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[1095] In this way, the present invention realizes a system that can recognize the user's emotions and provide an appropriate response accordingly, thereby reducing the stress and anxiety felt by the user.

[1096] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1097] Step 1:

[1098] User Initiation and Initial Connection

[1099] Input: A user calls a free call number 0800 or 0120.

[1100] How it works: The server receives the call and manages the incoming call using the telephone system interface.

[1101] Data processing: The server detects the incoming call and uses a speech synthesis engine to generate voice data as an initial response.

[1102] Output: Speech to the user: "Hello, welcome to our chatty AI system. How can we help you today?"

[1103] Step 2:

[1104] Speech content collection and emotion recognition

[1105] Input: The user speaks a question or request.

[1106] How it works: The device records the user's voice in real time and sends it to the server.

[1107] Data processing: The server sends the recorded voice data to an emotion recognition engine to analyze the user's emotions.

[1108] Output: Emotion data (e.g., "anxiety") analyzed by the emotion recognition engine is sent back to the server.

[1109] Step 3:

[1110] Converting audio data to text

[1111] Input: Recorded audio data.

[1112] How it works: The server sends voice data to a speech recognition system, which converts it into text data such as "What is the weather today?"

[1113] Data processing: The voice data is converted into text data using the speech recognition system's algorithm.

[1114] Output: Text data is sent back to the server.

[1115] Step 4:

[1116] Input to generative AI model and answer generation

[1117] Input: Text data and sentiment data.

[1118] How it works: The server inputs text data and emotion data into the generative AI model. It then generates a prompt sentence for the generative AI model.

[1119] Data processing: The generative AI model generates the optimal answer text based on the input data.

[1120] Output: The answer text (e.g. "Today's weather is sunny and the temperature is 20 degrees. Please relax.") is sent back to the server.

[1121] Step 5:

[1122] Converting to speech and responding to the user

[1123] Input: The generated answer text.

[1124] How it works: The server sends the answer text to a speech synthesis engine, which generates audio data.

[1125] Data processing: The text data is converted into voice data by a speech synthesis engine.

[1126] Output: The generated voice data is sent to the user over the telephone line.

[1127] Through the above processing steps, the system can provide an advanced response that takes into account the user's emotions and improve user satisfaction.

[1128] (Application example 2)

[1129] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1130] The present invention relates to a system that can recognize a user's emotions through voice calls and provide an appropriate response based on that emotion in real time. Conventional voice recognition systems have been unable to properly recognize a user's emotions and provide a response based on that emotion, which poses challenges, particularly in emergencies and situations where a response that reduces stress is required. Simply converting speech to text and returning a standard response often fails to fully address the user's emotions and state. This challenge must be addressed, particularly in security services, where prompt and emotionally responsive instructions are required.

[1131] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting a voice call from a user, means for recording the user's speech during the voice call, means for converting the recorded voice data into text data, means for generating an appropriate response text based on the text data using a generative model, means for converting the generated response text into voice data, means for responding to the user with the converted voice data via a communication line, and means for including an emotion engine that recognizes the user's emotions and generating a response corresponding to the emotions based on the emotion. The combination of emotion recognition and generative models in this invention makes it possible to provide users with a prompt and appropriate response. Particularly in security services, when a security guard on-site feels anxious in an emergency, the quality of on-site response can be improved by recognizing the guard's emotions and providing a response that includes instructions to relax.

[1132] A "user" is a person or organization that places a voice call and makes a question or request to the system.

[1133] "Voice call" refers to voice communication conducted over telephone lines or internet lines.

[1134] "Recording" is the act of storing audio data in digital form in real time.

[1135] "Audio data" refers to digital audio signals collected from an input device such as a microphone.

[1136] "Text data" is character string information converted from voice data by a voice recognition system.

[1137] A "generative model" is a neural network model that has been pre-trained with a wide range of knowledge and diverse roles, and is designed to generate appropriate answers based on input text data.

[1138] "Answer text" is text information that includes an answer to the user, generated by the generative model based on input text data.

[1139] A "speech synthesis engine" is a system that converts text data into speech data that humans can understand.

[1140] The "emotion engine" is a system that analyzes the user's voice data and recognizes the emotions contained in the voice in real time.

[1141] A "response" is an answer or instruction that the system provides to the user, and is provided as voice data.

[1142] A "communication line" is a communication means for sending and receiving voice data, and includes an internet line and a telephone line.

[1143] The system for implementing this invention is composed of a server, a terminal, and a user. Next, details of each component and their interactions will be explained.

[1144] Hardware and Software Configuration

[1145] 1. Server

[1146] The server is the central system that handles overall processing and data management. It uses Amazon EC2 instances.

[1147] The server is installed with software to perform the following functions:

[1148] Speech recognition system: Converts voice data into text data using the Google Speech-to-Text API.

[1149] Generative AI model: OpenAI GPT-4 is used to generate appropriate answers based on text data.

[1150] Emotion Engine: Uses the Affectiva API to recognize emotions from the user's voice in real time.

[1151] Speech synthesis engine: Amazon Polly is used to convert the generated response text into voice data.

[1152] 2. Terminal

[1153] The terminal is a device that collects audio and mediates communication between the user and the server. In this example, we use smart glasses (e.g., Google Glass Enterprise Edition 2).

[1154] It has the ability to collect audio using a built-in microphone and send the audio data to a server.

[1155] 3. Users

[1156] Users are security guards and other professionals who wear smart glasses to use the system, especially in on-site emergency response situations.

[1157] System Operation

[1158] 1. Audio collection

[1159] The user inputs voice through the smart glasses, which is collected by a built-in microphone.

[1160] For example, a security guard might say, "I need help!"

[1161] 2. Sending and analyzing audio data

[1162] The voice data collected from the terminal is sent to the server in real time.

[1163] The server first converts the audio data into text using a speech recognition system, in this case using the Google Speech-to-Text API.

[1164] 3. Emotion recognition

[1165] The voice data is sent in parallel to the emotion engine, which analyzes the user's emotion and uses the Affectiva API to identify the emotion as "impatience."

[1166] 4. Generating the Response

[1167] The server uses a generative AI model to generate an appropriate response based on text data and emotion recognition data. Through OpenAI GPT-4, the response generated is, "Assistance is on its way. Please remain calm."

[1168] 5. Audio Conversion

[1169] The generated answer text is converted into voice data through a speech synthesis engine, in this case Amazon Polly.

[1170] The generated voice data is returned to the user through the terminal.

[1171] Specific examples

[1172] For example, when a security guard at an emergency scene says "I need help!" in a panic, the response is generated in the following steps:

[1173] Prompt Sentence Examples

[1174] The user says "I need help!" with an impatient tone. Generate an appropriate response.

[1175] The server sends this prompt to the generative AI model, which generates a response saying, "Assistance is on its way. Please remain calm," and replies to the user via voice data. Through this process, the security guard on-site receives a prompt, emotionally appropriate response.

[1176] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1177] Step 1:

[1178] The user puts on the smart glasses and starts voice input. When reporting an emergency situation, the user speaks, "I need help!" The built-in microphone of the smart glasses collects voice data and starts recording in real time. The input is voice data, and the output is the voice data captured by the terminal. This voice data is sent to the next step in its original form.

[1179] Step 2:

[1180] The device sends the collected voice data to the server. Specifically, the voice data is uploaded to the server via an internet connection. The input is the voice data collected by the device, and the output is the voice data sent to the server. The server prepares this voice data for processing.

[1181] Step 3:

[1182] The server receives the voice data and inputs it into the voice recognition system. Here, the Google Speech-to-Text API is used to convert the voice data into text data. Specific operations include analyzing the voice waveform and identifying phonemes. The input is the voice data sent to the server, and the output is the converted text data.

[1183] Step 4:

[1184] The server sends the voice data to the emotion engine at the same time as the text data. The emotion engine (Affectiva API) analyzes the voice data in real time and recognizes the user's emotion. Specifically, it analyzes the tone, speed, and intensity of the voice. The input is the voice data, and the output is the recognized emotion data. In this case, the emotion recognized is "impatience."

[1185] Step 5:

[1186] The server inputs text data and emotion data into the generative AI model. OpenAI GPT-4 is used to generate a prompt sentence and generate an appropriate answer based on it. For example, a prompt might be generated such as, "The user said, 'I need help!' with an expression of impatience. Please generate an appropriate response." The input is text data and emotion data, and the output is the generated answer text. The generated response might be something like, "Assistance will arrive shortly. Please remain calm."

[1187] Step 6:

[1188] The server sends the generated answer text to a speech synthesis engine. Amazon Polly is used to convert the text data into speech data. Specifically, a speech waveform is generated based on the text information and output as natural-sounding speech. The input is the generated answer text, and the output is speech data.

[1189] Step 7:

[1190] The server sends the converted voice data to the terminal. The converted voice data is returned to the user through the telephone line. Specifically, the voice is played from the speaker of the smart glasses. The input is the voice data, and the output is the voice that the user can hear. The user can hear the response, "Assistance will arrive shortly. Please remain calm."

[1191] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1192] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1193] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1194] [Fourth embodiment]

[1195] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1196] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1197] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1198] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1199] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1200] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1201] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1202] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1203] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1204] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1205] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1206] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1207] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1208] The present invention is a system that accepts voice calls from users, converts the voice data into text data, generates an appropriate response using a generative model, converts the generated response into voice data, and responds to the user over a telephone line. This system is composed of a server, a terminal, and a user.

[1209] Program processing explanation

[1210] 1. User Initiation and Initial Connection

[1211] The user calls the free call number 0800 or 0120.

[1212] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[1213] 2. Collecting audio content and converting it into text

[1214] The user speaks a question or purpose (e.g., "What's the weather today?").

[1215] The terminal records the user's voice in real time.

[1216] The server transmits the recorded voice data to a voice recognition system, which converts the voice into text data.

[1217] 3. Input to the generative AI model and answer generation

[1218] The server sends the textual content to the generative model.

[1219] The generative model generates an appropriate answer text (e.g., "Today's weather is sunny and the temperature is 20 degrees").

[1220] 4. Converting to speech and responding to the user

[1221] The server sends the generated text to a speech synthesis engine to generate voice data.

[1222] The terminal responds to the user with the generated voice data via the telephone line (for example, by replying, "Today's weather is sunny and the temperature is 20 degrees").

[1223] Example: Elderly person asking about the weather

[1224] 1. The user (an elderly woman) calls 0800.

[1225] 2. The server receives the call and responds, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[1226] 3. The user says, "What's the weather like today?"

[1227] 4. The device records the user's voice.

[1228] 5. The server sends the recorded data to a speech recognition system and converts it into text data such as "What is the weather today?"

[1229] 6. The server sends the text to the generative model and generates the answer "Today's weather is sunny and the temperature is 20 degrees."

[1230] 7. The server sends this text to a speech synthesis engine to generate voice data.

[1231] 8. The device responds to the user, "Today's weather is sunny and the temperature is 20 degrees."

[1232] This system provides natural voice dialogue in real time and can meet the diverse needs of the elderly, young people who feel lonely, dementia patients, foreign immigrants, foreign language learners, etc. The system is designed to reduce users' stress and improve the quality of communication, and is characterized by its ease and intuitive use.

[1233] The processing flow will be explained below.

[1234] Step 1:

[1235] A user calls a free call number 0800 or 0120.

[1236] Step 2:

[1237] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[1238] Step 3:

[1239] The user speaks a question or purpose (e.g., "What's the weather today?").

[1240] Step 4:

[1241] The device records the user's voice in real time.

[1242] Step 5:

[1243] The server transmits the recorded voice data to a voice recognition system, which converts the voice data into text data.

[1244] The voice data is sent via an API to a text recognition service, which generates the text "What is the weather today?"

[1245] Step 6:

[1246] The server sends the text data to the generative model.

[1247] An API request is made to the generative model to generate an answer based on the text "What is the weather today?"

[1248] Step 7:

[1249] The server receives the generated answer.

[1250] The generative model returns the text "Today's weather is sunny and the temperature is 20 degrees."

[1251] Step 8:

[1252] The server sends the generated text to a speech synthesis engine to generate voice data.

[1253] Send text to the speech synthesis API and get the corresponding audio data.

[1254] Step 9:

[1255] The terminal receives the generated voice data and responds to the user through the communication line.

[1256] The device plays audio data and tells the user, "Today's weather is sunny and the temperature is 20 degrees."

[1257] Step 10:

[1258] If the user has further questions or wants to continue the conversation, they can speak again (e.g., "What's the weather like tomorrow?").

[1259] Step 11:

[1260] The device will record the user's voice again and send it to the server.

[1261] Step 12:

[1262] The server again sends the new voice data to the voice recognition system and converts it into text data.

[1263] The user's utterance "Please tell me the weather tomorrow" is converted into text data.

[1264] Step 13:

[1265] The server sends new text data to the generative model, which generates an appropriate answer.

[1266] The generative model generates the text "It will rain tomorrow and the temperature will be 18 degrees."

[1267] Step 14:

[1268] The server sends the generated text to a speech synthesis engine to generate voice data.

[1269] The speech synthesis API generates speech data based on the text data.

[1270] Step 15:

[1271] The terminal returns the generated voice data to the user via the communication line.

[1272] "It's going to rain tomorrow and the temperature will be 18 degrees," the app replies to the user.

[1273] The above are the processing steps of the invention, and the system provides a user with a prompt and appropriate voice response.

[1274] Example 1

[1275] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1276] In modern society, users with diverse needs are increasingly using voice to ask questions and receive advice. Providing prompt and appropriate answers is essential for individual users who require specific support, such as the elderly, people with disabilities, and foreign language learners. However, conventional voice response systems struggle to accurately understand the user's intent and generate appropriate answers, resulting in inconsistent response quality. This has led to issues such as reduced user satisfaction and reduced reliance on continued use.

[1277] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1278] In this invention, the server includes: [means for accepting voice communication from a user;] [means for recording what the user says in the voice communication;] [means for converting the recorded voice data into text data;] [means for generating appropriate response text data using a generative AI model based on the text data;] [means for converting the generated response text data into voice data; and [means for responding to the user with the converted voice data via a communication line.] This makes it possible to quickly convert the user's voice data into text, generate appropriate responses using natural language processing technology, and respond as voice.

[1279] "User" refers to an individual human being who interacts with the system via voice communication.

[1280] "Voice communication" refers to the means of sending and receiving voice information over telephone lines or internet lines.

[1281] "Utterance" refers to words, questions, or instructions that a user conveys to the system through voice communication.

[1282] "Recording" refers to the means of preserving audio data conveyed through voice communications.

[1283] "Audio Data" means data collected as sound, in digital or analog form.

[1284] "Character data" refers to digital data that has been converted from voice data into character format.

[1285] A "generative AI model" refers to an artificial neural network model that has been pre-trained with a wide range of knowledge and diverse roles.

[1286] "Answer text data" refers to the text information of the answer generated by the generative AI model based on user input.

[1287] "Converting to audio data" refers to converting character data into a digital format that can be played as sound.

[1288] "Communication lines" refer to telephone lines and internet lines for sending and receiving voice and data.

[1289] "Replying" refers to conveying the system-generated answer to the user.

[1290] The present invention is a system that accepts voice communication from a user, converts the voice data into text data, generates an appropriate response using a generative AI model, converts the response into voice data, and responds to the user via a communication line. A specific embodiment of this system is described below.

[1291] This system consists of a server, a terminal, and a user. The server is equipped with software and hardware to perform multiple functions. Specifically, the server includes a communication module, a voice recognition module, a generative AI model, a text conversion module, and a voice synthesis module.

[1292] Communication Module

[1293] The server uses a communication module to accept voice communications from users. Communications are conducted via telephone lines or internet lines. When a user calls 0800 or 0120 toll-free, the server automatically recognizes the incoming call and begins answering.

[1294] Speech Recognition Module

[1295] The device records the user's speech in real time. The recorded voice data is sent to a server via a communication line. The server then converts the voice data into text data using a voice recognition module (e.g., Google Cloud Speech-to-Text).

[1296] Generative AI Models

[1297] The server generates appropriate response text data using a generative AI model (e.g., a pre-trained artificial neural network model) based on the text data. Specifically, the server inputs the following prompt sentence into the generative AI model:

[1298] "Please tell me the weather today."

[1299] The generative AI model generates an appropriate answer based on the given prompt, for example, "Today's weather is sunny and the temperature is 20 degrees."

[1300] Text-to-text and speech synthesis modules

[1301] The server sends the generated response text data to a text conversion module (e.g., Google Cloud Text-to-Speech) and converts it into voice data. The generated voice data is sent to the terminal via a communication line, and the terminal plays the voice data and responds to the user.

[1302] Example: Elderly person asking about the weather

[1303] 1. The user (an elderly woman) calls 0800.

[1304] 2. The server receives the call and responds, "Hello, welcome to our chat system. How can we help you today?"

[1305] 3. The user says, "What's the weather like today?"

[1306] 4. The device records the user's voice and sends it to the server.

[1307] 5. The server sends the voice data to the voice recognition module and converts it into text data such as "What is the weather today?"

[1308] 6. The server sends the text data to the generative AI model, which generates the answer, "Today's weather is sunny and the temperature is 20 degrees."

[1309] 7. The server sends this text data to a speech synthesis module to generate speech data.

[1310] 8. The device responds to the user, "Today's weather is sunny and the temperature is 20 degrees."

[1311] The system uses specific hardware and software to efficiently process users' voice data and provide prompt and appropriate responses. The system is designed to reduce users' stress and improve the quality of communication.

[1312] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1313] Step 1:

[1314] User Initiation and Initial Connection

[1315] A user uses a telephone terminal to call the 0800 or 0120 toll-free number. The input is the user's speech (e.g., "What is the weather today?"). The server receives the incoming call via the communication line and activates the voice response system. The output is a greeting voice saying, "Hello. Welcome to the chat system. How can I help you today?"

[1316] Step 2:

[1317] Collecting audio content and recording audio data

[1318] The user receives a greeting from the server and then speaks out their question or purpose of use. The input is the user's speech. The terminal records this speech in real time and sends it to the server as voice data. Specifically, the terminal converts the voice into digital data through a microphone and sends it to the server via a communication line. The output is the voice data sent to the server.

[1319] Step 3:

[1320] Converting audio data to text

[1321] The server sends the received voice data to a voice recognition module. The input is voice data. The voice recognition module (e.g., Google Cloud Speech-to-Text API) converts the voice data into text data. Specifically, the voice data is analyzed and corresponding text is generated. The output is text data (e.g., "What is the weather today?").

[1322] Step 4:

[1323] Input to generative AI model and answer generation

[1324] The server sends the converted text data to the generative model. The input is text data. The generative AI model (e.g., a pre-trained artificial neural network model) generates an appropriate answer to the user's question. Specifically, the generative AI model analyzes the input text and generates an optimal answer based on the trained database. The output is the answer text (e.g., "Today's weather is sunny and the temperature is 20 degrees").

[1325] Step 5:

[1326] Converting response text to audio data

[1327] The server sends the generated answer text to a speech synthesis module. The input is the answer text. The speech synthesis module (e.g., Google Cloud Text-to-Speech) converts the text into speech data. Specifically, the speech synthesis module converts the text information into speech data using a speech synthesis algorithm. The output is speech data.

[1328] Step 6:

[1329] Replying to the user with voice data

[1330] The terminal plays the voice data received from the server and responds to the user. The input is voice data. The user can hear the response from the terminal. Specifically, the terminal plays the voice data through a speaker. The output is a voice response (e.g., "Today's weather is sunny and the temperature is 20 degrees"), which provides useful information to the user who hears it.

[1331] (Application example 1)

[1332] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1333] In physical stores, there is a demand for an environment where customers can efficiently obtain product information and location information without the assistance of store staff. It can be particularly difficult for customers to smoothly find the product they are looking for during busy times or when there is a shortage of staff. While multilingual guidance is also required, there is no easy way to achieve this. Therefore, it is important to build a system that allows customers to instantly obtain the information they need through their own devices or in-store equipment.

[1334] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1335] In this invention, the server includes means for accepting voice input from a user, means for recording the user's speech in the voice input, and means for converting the recorded voice data into text data. This allows users in a physical store to instantly obtain product information and location information, reducing the burden on store staff and improving customer satisfaction.

[1336] "User" refers to a customer or user who uses the system to input voice data.

[1337] "Voice input" refers to the voice signals that a user speaks to the system.

[1338] "Recording" is the process of recording a user's voice input in real time.

[1339] "Audio data" is data that represents an audio input signal in digital format.

[1340] "Text data" refers to data obtained by converting voice data into character information.

[1341] "Generative model" refers to a neural network model used to generate appropriate answer text based on given text data.

[1342] "Answer text" is text data containing an answer to a user's question created by a generative model.

[1343] The "audio output device" is a device such as a speaker or headphones that converts the generated answer text into audio data and responds to the user.

[1344] "Product information" refers to detailed information about products sold in physical stores, such as price, availability, and characteristics.

[1345] "Location information" refers to information about the location of a specific product within a physical store.

[1346] The term "system" refers to the entire set of mechanisms that includes the above means and provides voice responses to user questions.

[1347] This invention is a smart guidance system that supports customer service in brick-and-mortar stores. When a user asks a question by voice using a smartphone or a robot installed in the store, the voice is converted into text in real time and an appropriate answer is generated using a generative AI model. The generated answer is then converted into voice and returned to the user.

[1348] First, when a user speaks to a smartphone or in-store robot, the voice signal is received by the device, which then converts the voice signal into text data using Google Cloud Speech-to-Text or Amazon Transcribe. This speech recognition system then generates text from the voice signal.

[1349] The server then sends the text data to a generative AI model, such as OpenAI's GPT-4, which generates an appropriate answer. Because the generative AI model has extensive knowledge and learned a variety of roles in advance, it can generate an appropriate answer based on the user's question. For example, it can provide detailed guidance such as, "Take six steps from your current location, turn left, and then take the escalator in front of you to the third floor."

[1350] The generated text is then sent to a speech synthesis engine such as Google Cloud Text-to-Speech or Amazon Polly, where it is converted into voice data. The device then responds to the user through a voice output device (such as a speaker or headphones), allowing the user to receive the appropriate response via voice.

[1351] As a concrete example, consider a scenario in which a user opens a smartphone app and says, "Where is the chocolate section?" In this case, the smartphone app records the audio and transfers it to Google Cloud Speech-to-Text. The server then passes the received text data to GPT-4, which generates an appropriate answer. The generated answer, "The chocolate section is on the second floor, to the left of the food area," is then passed to Google Cloud Text-to-Speech, which generates audio data. Finally, the smartphone app plays the audio data and replies to the user, "The chocolate section is on the second floor, to the left of the food area."

[1352] Examples of prompts:

[1353] Question: 'Where is the drinking water section?'

[1354] Store map data: [List of corner names and locations]

[1355] Expected answer: 'The drinking water corner is in the center of the third floor.'

[1356] This system allows customers to quickly obtain the information they need, reducing the burden on store staff and improving customer satisfaction.

[1357] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1358] Step 1:

[1359] The user inputs voice to a smartphone or a robot in the store. Here, the user asks questions about product information or location information within the store. For example, the user might say, "Where is the chocolate section?" This voice signal is input to the terminal.

[1360] Step 2:

[1361] The device receives the user's voice input and records the speech in real time, which is then stored on the device and sent to a speech recognition system such as Google Cloud Speech-to-Text or Amazon Transcribe for appropriate processing.

[1362] Step 3:

[1363] A speech recognition system takes voice data as input and converts it into text data. Specifically, it analyzes the voice signal and converts its content into written information. This text data has the form of a user question, such as "Where is the chocolate corner?"

[1364] Step 4:

[1365] The server receives text data sent from the speech recognition system and sends it to OpenAI's GPT-4 or other generative AI models, which use extensive pre-trained knowledge to generate appropriate answers for the text data.

[1366] Step 5:

[1367] The generative AI model receives the text data "Where is the chocolate corner?" as input and generates the corresponding answer text "The chocolate corner is on the left side of the food area on the second floor." This answer text is output to the server.

[1368] Step 6:

[1369] The server receives the generated answer text and sends it to a speech synthesis engine such as Google Cloud Text-to-Speech or Amazon Polly, which converts the answer text into audio data.

[1370] Step 7:

[1371] The voice data generated by the speech synthesis engine is sent from the server to the device. The device then plays the voice data through a voice output device (speaker or headphones) and responds to the user. For example, the device might provide a voice prompt saying, "The chocolate section is on the left side of the food area on the second floor."

[1372] In this way, a series of processes from the user's voice input to the answer is carried out automatically, and information is provided quickly and accurately.

[1373] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1374] The present invention provides a system that converts voice data into text data through a user's voice call and generates appropriate responses using a generative model, and further combines it with an emotion engine to provide more advanced voice responses with emotion recognition. This system is composed of a server, a terminal, and a user.

[1375] Program processing explanation

[1376] 1. User Initiation and Initial Connection

[1377] A user calls a free call number 0800 or 0120.

[1378] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[1379] 2. Speech content collection and emotion recognition

[1380] The user speaks a question or purpose (e.g., "What's the weather today?").

[1381] The device records the user's voice in real time.

[1382] The server sends the recorded voice data to the emotion engine to recognize the user's emotion. For example, if the user speaks with a slightly anxious voice, the emotion engine will recognize it as "anxiety."

[1383] 3. Converting voice data to text

[1384] The server sends the voice data to a voice recognition system, which converts the voice data into text data.

[1385] 4. Input to the generative AI model and answer generation

[1386] The server sends the text data to the generative model, which then uses the recognition results from the emotion engine to generate a more appropriate answer based on the emotion.

[1387] The generative model generates a response text such as "Today's weather is sunny and the temperature is 20 degrees." If the user feels anxious, they can add an additional comment such as "The weather is nice, so please relax."

[1388] 5. Converting to speech and responding to the user

[1389] The server transmits the generated answer text to a speech synthesis engine to generate speech data.

[1390] The terminal returns the generated voice data to the user via the communication line.

[1391] Example: Elderly person asking about the weather

[1392] 1. A user calls 0800.

[1393] 2. The server receives the call and responds, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[1394] 3. The user says in a slightly anxious voice, "What's the weather like today?"

[1395] 4. The device records the user's voice.

[1396] 5. The server sends the recorded data to the emotion engine and recognizes that the user's emotion is "anxiety."

[1397] 6. The server sends the voice data to a voice recognition system and converts it into text data such as "What is the weather today?"

[1398] 7. The server sends the text data and emotion data to the generative model, which generates the answer: "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[1399] 8. The server sends this text to a speech synthesis engine to generate voice data.

[1400] 9. The device responds to the user, "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[1401] This system can significantly improve the quality of communication by recognizing emotions and providing appropriate responses to elderly people, lonely young people, dementia patients, foreign immigrants, foreign language learners, etc. Furthermore, by using the emotion engine, it is expected to reduce the stress and anxiety felt by users.

[1402] The processing flow will be explained below.

[1403] Step 1:

[1404] A user calls a free call number 0800 or 0120.

[1405] Step 2:

[1406] The server receives the call and greets the user with an AI voice response saying, "Hello, welcome to Chatty GPT Plus. How can we help you today?"

[1407] Step 3:

[1408] The user speaks a question or purpose (e.g., "What's the weather today?").

[1409] Step 4:

[1410] The device records the user's voice in real time.

[1411] Step 5:

[1412] The server sends the recorded voice data to the emotion engine to recognize the user's emotions.

[1413] Here, for example, if the user's voice is slightly trembling, the emotion engine will recognize this as "anxiety."

[1414] Step 6:

[1415] The server sends the recorded data to a voice recognition system, which converts the voice data into text data.

[1416] The voice data is sent via an API to a text recognition service, where it is converted into text such as "What is the weather today?"

[1417] Step 7:

[1418] The server sends the text data and the recognized emotion data to the generative model.

[1419] An API request is made to the generative model to generate an answer based on the text "What is the weather today?" and the emotion data "anxiety."

[1420] Step 8:

[1421] The server receives the generated answer.

[1422] The generative model returns the text, "Today's weather is sunny and the temperature is 20 degrees. Don't worry."

[1423] Step 9:

[1424] The server sends the generated text to a speech synthesis engine to generate voice data.

[1425] Text is sent to a speech synthesis API to generate voice data such as, "Today's weather is sunny and the temperature is 20 degrees. Don't worry."

[1426] Step 10:

[1427] The terminal receives the generated voice data and responds to the user through the communication line.

[1428] The device plays audio data and tells the user, "Today's weather is sunny and the temperature is 20 degrees. Don't worry."

[1429] Step 11:

[1430] If the user has further questions or wants to continue the conversation, they can speak again (e.g., "What's the weather like tomorrow?").

[1431] Step 12:

[1432] The device will record the user's voice again and send it to the server.

[1433] Step 13:

[1434] The server sends new voice data to the emotion engine, which then recognizes the user's emotion again.

[1435] If the new recording data contains, for example, a sense of elation, it will be recognized as "joy."

[1436] Step 14:

[1437] The server again sends the new voice data to the voice recognition system and converts it into text data.

[1438] The user's utterance "Please tell me the weather tomorrow" is converted into text data.

[1439] Step 15:

[1440] The server sends new text and emotion data to the generative model, which then generates an appropriate answer.

[1441] The generative model generates the text "It will rain tomorrow and the temperature will be 18 degrees."

[1442] Step 16:

[1443] The server sends the generated text to a speech synthesis engine to generate voice data.

[1444] The speech synthesis API generates speech data based on the text data.

[1445] Step 17:

[1446] The terminal returns the generated voice data to the user via the communication line.

[1447] "It's going to rain tomorrow and the temperature will be 18 degrees," the app replies to the user.

[1448] In this way, the emotion engine can be used to reduce the stress and anxiety felt by the user and provide a more appropriate response.

[1449] Example 2

[1450] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1451] Current voice response systems simply convert voice data into text and generate responses without recognizing the user's emotions, making it impossible to alleviate the stress and anxiety experienced by users. This is particularly insufficient for users who require emotional consideration, such as the elderly, users who feel lonely, and people with dementia. To solve this problem, an advanced system is needed that can recognize the user's emotions and provide responses that take them into account.

[1452] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1453] In this invention, the server includes: [means for accepting a voice call from a user;] [means for recording the user's speech during the voice call;] [means for transmitting the recorded voice data to an emotion recognition engine to recognize the user's emotions;] [means for converting the voice data into text data;] [means for generating an appropriate answer text using a generative AI model based on the text data and emotion data;] [means for converting the generated answer text into voice data; and [means for replying to the user with the converted voice data via a call line.] This makes it possible to recognize the user's emotions and generate an appropriate voice response corresponding to them.

[1454] "User" means an individual who accesses the System, makes voice calls and uses the Service.

[1455] "Voice call" refers to a communication method for sending and receiving voice data via telephone lines or Internet lines.

[1456] "Server" refers to a computer system for accepting voice calls and processing data.

[1457] "Terminal" refers to a device used by a user to make voice calls, such as a telephone or smartphone.

[1458] "Speech recognition system" refers to a software and hardware system for converting voice data into text data.

[1459] "Text data" refers to a data format in which voice data is converted into a string of characters.

[1460] An "emotion recognition engine" refers to a system that analyzes and recognizes a user's emotions from voice and text data.

[1461] "Emotion data" refers to emotional information obtained as a result of analysis by an emotion recognition engine.

[1462] A "generative AI model" refers to a neural network model that has been pre-trained with a wide range of knowledge and diverse roles.

[1463] "Answer text" refers to the response content generated by the generative AI model based on the user's question, expressed in text format.

[1464] "Speech synthesis engine" refers to a system for converting text data into speech data.

[1465] "Telephone line" refers to the communication line required to make a voice call.

[1466] A "prompt sentence" refers to text containing input instructions for a generative AI model.

[1467] The present invention is a system that converts voice data into text data through a user's voice call and generates an appropriate voice response based on the text data. This system includes multiple components, including a server, a terminal, and a user. The server accepts the voice call, converts the user's voice into text, uses a generative AI model to generate a response text, and converts it into voice to respond to the user.

[1468] Hardware and software used

[1469] Server: A high-performance computer system that runs speech recognition, generative AI models, emotion recognition engines, and speech synthesis engines.

[1470] Terminal: A device (telephone, smartphone, etc.) used by a user to make voice calls.

[1471] Speech recognition system: Software that converts voice data into text (e.g., Google Cloud Speech-to-Text API, Amazon Transcribe).

[1472] Emotion recognition engine: A system that analyzes user emotions from voice data (e.g., IBM Watson Tone Analyzer, Microsoft Azure Text Analytics).

[1473] Generative AI models: Neural network models pre-trained with a wide range of knowledge and diverse roles (e.g., GPT-3).

[1474] Speech synthesis engine: A system that converts text data into speech data (e.g., Google Text-to-Speech API, Amazon Polly).

[1475] System operation explanation

[1476] 1. User Initiation and Initial Connection

[1477] When a user calls the 0800 or 0120 toll-free number, the server receives the call and responds using a speech synthesis engine, saying, "Hello. Welcome to our chatty AI system. How can we help you today?"

[1478] 2. Speech content collection and emotion recognition

[1479] When a user speaks a question or request, the device records the voice in real time and sends it to the server. The server then sends the voice data to an emotion recognition engine, which analyzes the user's emotions. For example, if a user says, "What's the weather like today?" in a slightly anxious voice, the emotion recognition engine will identify the emotion "anxiety" from the voice.

[1480] 3. Converting voice data to text

[1481] The server converts the voice data into text data using a speech recognition system, which returns text such as "What's the weather like today?"

[1482] 4. Input to the generative AI model and answer generation

[1483] The server inputs text data and emotion data into the generative AI model to generate an appropriate answer. An example of a prompt sentence is, "The user's question is 'What's the weather like today?' The user is expressing the emotion 'anxiety.' The generated answer should include elements that reassure the user." The generative AI model generates the answer text, "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[1484] 5. Converting to speech and responding to the user

[1485] The server sends the generated response text to the speech synthesis engine, which generates voice data. The device responds to the user, saying, "Today's weather is sunny and the temperature is 20 degrees. Please relax."

[1486] In this way, the present invention realizes a system that can recognize the user's emotions and provide an appropriate response accordingly, thereby reducing the stress and anxiety felt by the user.

[1487] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1488] Step 1:

[1489] User Initiation and Initial Connection

[1490] Input: A user calls a free call number 0800 or 0120.

[1491] How it works: The server receives the call and manages the incoming call using the telephone system interface.

[1492] Data processing: The server detects the incoming call and uses a speech synthesis engine to generate voice data as an initial response.

[1493] Output: Speech to the user: "Hello, welcome to our chatty AI system. How can we help you today?"

[1494] Step 2:

[1495] Speech content collection and emotion recognition

[1496] Input: The user speaks a question or request.

[1497] How it works: The device records the user's voice in real time and sends it to the server.

[1498] Data processing: The server sends the recorded voice data to an emotion recognition engine to analyze the user's emotions.

[1499] Output: Emotion data (e.g., "anxiety") analyzed by the emotion recognition engine is sent back to the server.

[1500] Step 3:

[1501] Converting audio data to text

[1502] Input: Recorded audio data.

[1503] How it works: The server sends voice data to a speech recognition system, which converts it into text data such as "What is the weather today?"

[1504] Data processing: The voice data is converted into text data using the speech recognition system's algorithm.

[1505] Output: Text data is sent back to the server.

[1506] Step 4:

[1507] Input to generative AI model and answer generation

[1508] Input: Text data and sentiment data.

[1509] How it works: The server inputs text data and emotion data into the generative AI model. It then generates a prompt sentence for the generative AI model.

[1510] Data processing: The generative AI model generates the optimal answer text based on the input data.

[1511] Output: The answer text (e.g. "Today's weather is sunny and the temperature is 20 degrees. Please relax.") is sent back to the server.

[1512] Step 5:

[1513] Converting to speech and responding to the user

[1514] Input: The generated answer text.

[1515] How it works: The server sends the answer text to a speech synthesis engine, which generates audio data.

[1516] Data processing: The text data is converted into voice data by a speech synthesis engine.

[1517] Output: The generated voice data is sent to the user over the telephone line.

[1518] Through the above processing steps, the system can provide an advanced response that takes into account the user's emotions and improve user satisfaction.

[1519] (Application example 2)

[1520] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1521] The present invention relates to a system that can recognize a user's emotions through voice calls and provide an appropriate response based on that emotion in real time. Conventional voice recognition systems have been unable to properly recognize a user's emotions and provide a response based on that emotion, which poses challenges, particularly in emergencies and situations where a response that reduces stress is required. Simply converting speech to text and returning a standard response often fails to fully address the user's emotions and state. This challenge must be addressed, particularly in security services, where prompt and emotionally responsive instructions are required.

[1522] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting a voice call from a user, means for recording the user's speech during the voice call, means for converting the recorded voice data into text data, means for generating an appropriate response text based on the text data using a generative model, means for converting the generated response text into voice data, means for responding to the user with the converted voice data via a communication line, and means for including an emotion engine that recognizes the user's emotions and generating a response corresponding to the emotions based on the emotion. The combination of emotion recognition and generative models in this invention makes it possible to provide users with a prompt and appropriate response. Particularly in security services, when a security guard on-site feels anxious in an emergency, the quality of on-site response can be improved by recognizing the guard's emotions and providing a response that includes instructions to relax.

[1523] A "user" is a person or organization that places a voice call and makes a question or request to the system.

[1524] "Voice call" refers to voice communication conducted over telephone lines or internet lines.

[1525] "Recording" is the act of storing audio data in digital form in real time.

[1526] "Audio data" refers to digital audio signals collected from an input device such as a microphone.

[1527] "Text data" is character string information converted from voice data by a voice recognition system.

[1528] A "generative model" is a neural network model that has been pre-trained with a wide range of knowledge and diverse roles, and is designed to generate appropriate answers based on input text data.

[1529] "Answer text" is text information that includes an answer to the user, generated by the generative model based on input text data.

[1530] A "speech synthesis engine" is a system that converts text data into speech data that humans can understand.

[1531] The "emotion engine" is a system that analyzes the user's voice data and recognizes the emotions contained in the voice in real time.

[1532] A "response" is an answer or instruction that the system provides to the user, and is provided as voice data.

[1533] A "communication line" is a communication means for sending and receiving voice data, and includes an internet line and a telephone line.

[1534] The system for implementing this invention is composed of a server, a terminal, and a user. Next, details of each component and their interactions will be explained.

[1535] Hardware and Software Configuration

[1536] 1. Server

[1537] The server is the central system that handles overall processing and data management. It uses Amazon EC2 instances.

[1538] The server is installed with software to perform the following functions:

[1539] Speech recognition system: Converts voice data into text data using the Google Speech-to-Text API.

[1540] Generative AI model: OpenAI GPT-4 is used to generate appropriate answers based on text data.

[1541] Emotion Engine: Uses the Affectiva API to recognize emotions from the user's voice in real time.

[1542] Speech synthesis engine: Amazon Polly is used to convert the generated response text into voice data.

[1543] 2. Terminal

[1544] The terminal is a device that collects audio and mediates communication between the user and the server. In this example, we use smart glasses (e.g., Google Glass Enterprise Edition 2).

[1545] It has the ability to collect audio using a built-in microphone and send the audio data to a server.

[1546] 3. Users

[1547] Users are security guards and other professionals who wear smart glasses to use the system, especially in on-site emergency response situations.

[1548] System Operation

[1549] 1. Audio collection

[1550] The user inputs voice through the smart glasses, which is collected by a built-in microphone.

[1551] For example, a security guard might say, "I need help!"

[1552] 2. Sending and analyzing audio data

[1553] The voice data collected from the terminal is sent to the server in real time.

[1554] The server first converts the audio data into text using a speech recognition system, in this case using the Google Speech-to-Text API.

[1555] 3. Emotion recognition

[1556] The voice data is sent in parallel to the emotion engine, which analyzes the user's emotion and uses the Affectiva API to identify the emotion as "impatience."

[1557] 4. Generating the Response

[1558] The server uses a generative AI model to generate an appropriate response based on text data and emotion recognition data. Through OpenAI GPT-4, the response generated is, "Assistance is on its way. Please remain calm."

[1559] 5. Audio Conversion

[1560] The generated answer text is converted into voice data through a speech synthesis engine, in this case Amazon Polly.

[1561] The generated voice data is returned to the user through the terminal.

[1562] Specific examples

[1563] For example, when a security guard at an emergency scene says "I need help!" in a panic, the response is generated in the following steps:

[1564] Prompt Sentence Examples

[1565] The user says "I need help!" with an impatient tone. Generate an appropriate response.

[1566] The server sends this prompt to the generative AI model, which generates a response saying, "Assistance is on its way. Please remain calm," and replies to the user via voice data. Through this process, the security guard on-site receives a prompt, emotionally appropriate response.

[1567] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1568] Step 1:

[1569] The user puts on the smart glasses and starts voice input. When reporting an emergency situation, the user speaks, "I need help!" The built-in microphone of the smart glasses collects voice data and starts recording in real time. The input is voice data, and the output is the voice data captured by the terminal. This voice data is sent to the next step in its original form.

[1570] Step 2:

[1571] The device sends the collected voice data to the server. Specifically, the voice data is uploaded to the server via an internet connection. The input is the voice data collected by the device, and the output is the voice data sent to the server. The server prepares this voice data for processing.

[1572] Step 3:

[1573] The server receives the voice data and inputs it into the voice recognition system. Here, the Google Speech-to-Text API is used to convert the voice data into text data. Specific operations include analyzing the voice waveform and identifying phonemes. The input is the voice data sent to the server, and the output is the converted text data.

[1574] Step 4:

[1575] The server sends the voice data to the emotion engine at the same time as the text data. The emotion engine (Affectiva API) analyzes the voice data in real time and recognizes the user's emotion. Specifically, it analyzes the tone, speed, and intensity of the voice. The input is the voice data, and the output is the recognized emotion data. In this case, the emotion recognized is "impatience."

[1576] Step 5:

[1577] The server inputs text data and emotion data into the generative AI model. OpenAI GPT-4 is used to generate a prompt sentence and generate an appropriate answer based on it. For example, a prompt might be generated such as, "The user said, 'I need help!' with an expression of impatience. Please generate an appropriate response." The input is text data and emotion data, and the output is the generated answer text. The generated response might be something like, "Assistance will arrive shortly. Please remain calm."

[1578] Step 6:

[1579] The server sends the generated answer text to a speech synthesis engine. Amazon Polly is used to convert the text data into speech data. Specifically, a speech waveform is generated based on the text information and output as natural-sounding speech. The input is the generated answer text, and the output is speech data.

[1580] Step 7:

[1581] The server sends the converted voice data to the terminal. The converted voice data is returned to the user through the telephone line. Specifically, the voice is played from the speaker of the smart glasses. The input is the voice data, and the output is the voice that the user can hear. The user can hear the response, "Assistance will arrive shortly. Please remain calm."

[1582] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1583] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1584] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1585] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1586] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1587] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1588] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1589] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1590] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1591] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1592] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1593] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1594] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1595] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1596] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1597] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1598] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1599] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1600] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1601] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1602] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1603] The following is further disclosed regarding the above embodiment.

[1604] (Claim 1)

[1605] [means for receiving a voice call from a user;

[1606] [Means for recording a user's speech during a voice call;

[1607] [Means for converting recorded audio data into text data;

[1608] [Means for generating appropriate answer text using a generative model based on text data;

[1609] [Means for converting the generated answer text into audio data;

[1610] [Means for returning the converted voice data to the user through a telephone line;

[1611] A system including:

[1612] (Claim 2)

[1613] The system of claim 1, wherein the generative model is a neural network model that has been pre-trained with extensive knowledge and diverse roles.

[1614] (Claim 3)

[1615] 2. The system according to claim 1, wherein the answer text generated based on the text data includes an appropriate answer to the user's question.

[1616] "Example 1"

[1617] (Claim 1)

[1618] [means for accepting voice communications from a user;

[1619] [Means for recording user utterances in voice communication;

[1620] [Means for converting recorded voice data into text data;

[1621] [Means for generating appropriate answer text data using a generation AI model based on text data;

[1622] [Means for converting the generated answer character data into voice data;

[1623] [Means for returning the converted voice data to the user through a communication line;

[1624] A system including:

[1625] (Claim 2)

[1626] [The system of claim 1, wherein the generative AI model is an artificial neural network model that has pre-trained a wide range of knowledge and diverse roles.

[1627] (Claim 3)

[1628] The system according to claim 1, wherein the answer character data generated based on the character data includes an appropriate answer to the user's question.

[1629] "Application Example 1"

[1630] (Claim 1)

[1631] [means for accepting voice input from a user;

[1632] [Means for recording user's speech in voice input;

[1633] [Means for converting recorded audio data into text data;

[1634] [Means for generating appropriate answer text using a generative model based on text data;

[1635] [Means for converting the generated answer text into audio data;

[1636] [Means for responding to the user with the converted voice data through a voice output device;

[1637] [Means of providing product information and location information to customers in physical stores,

[1638] A system including:

[1639] (Claim 2)

[1640] The system of claim 1, wherein the generative model is a neural network model that has pre-trained a wide range of knowledge and diverse roles.

[1641] (Claim 3)

[1642] The system according to claim 1, wherein the answer text generated based on the text data includes appropriate product information and location information in response to the user's question.

[1643] "Example 2: Combining Emotion Engines"

[1644] (Claim 1)

[1645] [means for receiving a voice call from a user;

[1646] [Means for recording a user's speech during a voice call;

[1647] [Means for transmitting recorded voice data to an emotion recognition engine to recognize the emotion of the user;

[1648] [Means for converting voice data into text data;

[1649] [Means for generating appropriate answer text using a generative AI model based on text data and emotion data;

[1650] [Means for converting the generated answer text into audio data;

[1651] [Means for returning the converted voice data to the user through a telephone line;

[1652] A system including:

[1653] (Claim 2)

[1654] The system of claim 1, wherein the generative AI model is a neural network model that has pre-trained extensive knowledge and diverse roles.

[1655] (Claim 3)

[1656] The system of claim 1, wherein the generative AI model generates answer text according to the user's emotions.

[1657] (Claim 4)

[1658] 2. The system according to claim 1, wherein the answer text generated based on the text data includes an appropriate answer to the user's question.

[1659] (Claim 5)

[1660] 2. The system of claim 1, wherein the emotion recognition engine categorizes a user's emotion from voice data.

[1661] "Application example 2 when combining emotion engines"

[1662] (Claim 1)

[1663] [means for receiving a voice call from a user;

[1664] [Means for recording a user's speech during a voice call;

[1665] [Means for converting recorded audio data into text data;

[1666] [Means for generating appropriate answer text using a generative model based on text data;

[1667] [Means for converting the generated answer text into audio data;

[1668] [Means for returning the converted voice data to the user through a telephone line;

[1669] [including an emotion engine that recognizes the user's emotion and generates a response corresponding to the emotion based on the emotion;

[1670] A system including:

[1671] (Claim 2)

[1672] The system of claim 1, characterized in that [the generative model is a neural network model that has been pre-trained with a wide range of knowledge and diverse roles, and the emotion engine recognizes emotions from the user's voice in real time].

[1673] (Claim 3)

[1674] The system of claim 1, wherein the answer text generated based on the text data includes an appropriate answer to the user's question and includes additional comments or instructions according to the emotion recognized by the emotion engine. [Explanation of symbols]

[1675] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for accepting a voice call from a user; a means for recording a user's speech during a voice call; A means for converting the recorded voice data into text data; A means for generating an appropriate answer text using a generative model based on text data; A means for converting the generated answer text into voice data; a means for returning the converted voice data to the user through a telephone line; A system including:

2. The system according to claim 1 , wherein the generative model is a neural network model that has been pre-trained with a wide range of knowledge and diverse roles.

3. 2. The system according to claim 1, wherein the answer text generated based on the text data includes an appropriate answer to the user's question.

Images