System

Abstract

A system is provided.SOLUTION: A system comprising: server means for holding a generative artificial intelligent model; robot means for converting a voice datum into a text datum and transmitting the text datum to the server means; and robot means for receiving a response from the server means, converting the response into a voice, and providing the voice to a user, wherein the robot means includes a voice recognizing engine and a speech synthesis engine.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, the number of seniors living alone is increasing, resulting in loneliness becoming a serious social problem. A lack of conversation partners is one of the causes of loneliness, and effective means to resolve this issue are needed. However, existing communication tools and robots have limited conversational capabilities, making it difficult to engage in natural conversation. Therefore, new technologies are needed to enable seniors to enjoy natural conversations and alleviate loneliness. [Means for solving the problem]

[0005] The present invention provides a system including a server means for storing a generative AI model, a robot means for converting voice data into text data and sending it to the server means, and a robot means for receiving a response from the server means, converting it into voice, and providing it to a user. The robot means includes a voice recognition engine and a voice synthesis engine, and the server means analyzes the text data using a natural language processing (NLP) engine and generates a response using the generative AI model. This allows for natural conversation to be provided to seniors, helping to alleviate feelings of loneliness.

[0006] The "server means" is a computing device that holds a generative AI model, analyzes voice data as text data, generates an appropriate response, and transmits it to the robot means.

[0007] The "robot means" is a device that converts voice input into text data and sends it to the server means, and also converts responses received from the server means into voice and provides them to the user.

[0008] A "voice recognition engine" is software or hardware for converting voice data into text data.

[0009] A "speech synthesis engine" is software or hardware for converting text data into voice data.

[0010] A "generative artificial intelligence model" is an AI model that generates appropriate responses based on natural language text data.

[0011] A "Natural Language Processing (NLP) engine" is software or hardware that analyzes natural language text data and understands its meaning.

[0012] "Users" are individuals, including seniors, who communicate using the system. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention is a system that includes a server that stores a generative AI model, a robot that converts voice data into text data and sends it to the server, and a robot that receives a response from the server, converts it into voice, and provides it to the user. Specific embodiments for implementing this system are described in detail below.

[0035] System Configuration

[0036] 1. Server Configuration

[0037] The server contains a generative AI model and a natural language processing (NLP) engine. The server performs the following functions:

[0038] When the voice data is transmitted as text data, the text content is analyzed.

[0039] Uses a Natural Language Processing (NLP) engine to understand the content of the text data sent to it.

[0040] An appropriate response is generated using a generative AI model, and the response is sent to the robot as text data.

[0041] 2. Robot Configuration

[0042] The robot performs the following functions:

[0043] It has a microphone for receiving voice input from the user.

[0044] A speech recognition engine is used to convert the user's speech into text data.

[0045] It has a communication means for transmitting the converted voice as text data to the server.

[0046] It has a communication means for receiving responses from the server.

[0047] A speech synthesis engine is used to convert the received text response into speech and provide it to the user through the speaker.

[0048] Program Processing

[0049] 1. Server-side processing

[0050] The server receives the text data sent by the robot and analyzes it, specifically as follows:

[0051] The server receives text data converted by a voice recognition engine from the robot.

[0052] The received text data is analyzed using a natural language processing (NLP) engine to understand its content.

[0053] Generate appropriate response text based on a generative AI model.

[0054] The generated response text is sent to the robot.

[0055] 2. Processing on the robot side

[0056] The robot converts the user's voice input into text data and sends it to the server. Specifically, it does the following:

[0057] The user speaks to the robot, which receives the voice through a microphone.

[0058] The robot's voice recognition engine converts the voice into text data.

[0059] The converted text data is sent to the server.

[0060] Receive a response from the server.

[0061] The received response text is converted into voice data by a speech synthesis engine and provided to the user through a speaker.

[0062] Specific examples

[0063] Example 1: A conversation about the weather

[0064] Consider a scenario in which a user asks a robot, "What's the weather like today?"

[0065] 1. The user says, "What's the weather like today?"

[0066] 2. The robot converts this speech into text data: "What's the weather like today?"

[0067] 3. The robot sends the text data to the server.

[0068] 4. The server receives the text data and analyzes it using the NLP engine.

[0069] 5. The server uses the generative AI model to generate a text response: "Today it's sunny and the temperature is 20 degrees."

[0070] 6. The server generates a text response and sends it to the robot.

[0071] 7. The robot converts the received text response, "Today is sunny and the temperature is 20 degrees," into voice data.

[0072] 8. The robot tells the user, "Today is sunny and the temperature is 20 degrees."

[0073] In this way, the present invention provides an effective means for seniors to overcome feelings of loneliness through natural conversation. By having the server and the robot work together, users can enjoy high-quality communication.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] The user speaks to the robot, "What's the weather like today?"

[0077] Step 2:

[0078] The robot uses a microphone to capture the user's voice.

[0079] Step 3:

[0080] The robot's voice recognition engine converts the captured voice data into text data such as "What's the weather like today?"

[0081] Step 4:

[0082] The robot sends the converted text data to a server via the Internet.

[0083] Step 5:

[0084] The server receives the text data sent from the robot.

[0085] Step 6:

[0086] The server's Natural Language Processing (NLP) engine analyzes the received text data.

[0087] Step 7:

[0088] The server uses an NLP engine to understand that the user is looking for information about the weather.

[0089] Step 8:

[0090] The server uses a generative AI model to generate a response text such as, "Today it's sunny and the temperature is 20 degrees."

[0091] Step 9:

[0092] The server sends the generated response text to the robot.

[0093] Step 10:

[0094] The robot receives the response text sent from the server.

[0095] Step 11:

[0096] The robot's speech synthesis engine converts the received text data, "Today is sunny and the temperature is 20 degrees," into speech data.

[0097] Step 12:

[0098] The robot provides the converted voice data to the user through a speaker.

[0099] Step 13:

[0100] The user hears a voice response from the robot: "Today is sunny and the temperature is 20 degrees."

[0101] Example 1

[0102] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0103] Conventional speech recognition and response generation systems lack the processing power and response quality required to realize natural conversations with users, resulting in the inability to fully alleviate users' feelings of loneliness. This has been a particular problem for seniors and users living alone, as it has been difficult to provide sufficient psychological support through conversation.

[0104] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0105] In this invention, the server includes means for storing a generative artificial intelligence model, robot means for converting voice data into text data and transmitting it, robot means for receiving a response, converting it into voice and providing it to the user, and means for analyzing the text data using a natural language processing engine and transmitting the generated response to the robot means. This enables natural conversation with the user and provides high-quality responses, thereby alleviating the user's sense of loneliness and providing psychological support.

[0106] A "generative artificial intelligence model" is an artificial intelligence algorithm for generating appropriate responses based on input data.

[0107] A "server means" is a computing device that analyzes data and generates responses.

[0108] A "robot means" is a device that acts as an interface with a user and receives, converts, transmits, and plays back audio.

[0109] A "voice recognition engine" is software for converting voice data into text data.

[0110] A "communication module" is a device or software for sending and receiving data.

[0111] A "speech synthesis engine" is software for converting text data into voice data.

[0112] A "natural language processing engine" is software for analyzing text data and understanding its content.

[0113] "Text data" is data expressed as character information.

[0114] "Audio data" is data that represents an audio signal in digital form.

[0115] "User" refers to a person who uses this system.

[0116] A "prompt sentence" is an input sentence that generates an appropriate response based on a generative artificial intelligence model.

[0117] "Response" refers to the system's response to a user's input.

[0118] This invention relates to a speech recognition and response generation system based on a generative AI model. This system has a series of functions: converting speech data into text data, analyzing the text data to generate an appropriate response, and then converting it back into speech data to provide to the user. The following describes in detail how this system is specifically implemented.

[0119] System Configuration

[0120] 1. Server configuration

[0121] The server contains the following main components:

[0122] Generative AI models: Includes models for generating appropriate responses based on input text data.

[0123] Natural Language Processing (NLP) engine: Includes an engine for analyzing text data and understanding its content.

[0124] The server receives the text data sent from the robot, analyzes it, generates a response using a generative artificial intelligence model, and sends the response back to the robot as text data.

[0125] 2. Robot Configuration

[0126] The robot includes the following main components:

[0127] Microphone: Hardware for receiving audio input from the user.

[0128] Speech recognition engine: Software for converting received voice data into text data (e.g., Google Cloud Speech-to-Text).

[0129] Communication module: A module for sending converted text data to the server and receiving responses from the server.

[0130] Speech synthesis engine: Software (e.g., Amazon Polly) that converts received text responses into speech.

[0131] Speaker: Hardware that outputs converted audio data to the user.

[0132] Specific system processing examples

[0133] As a specific example of system processing, a scenario of a conversation about the weather is shown below.

[0134] Prompt Sentence Examples

[0135] When a user asks the robot, "What's the weather like today?" the prompt is:

[0136] "What's the weather like today?"

[0137] Weather conversation process flow

[0138] 1. Voice input: The user speaks to the robot, "What's the weather like today?"

[0139] 2. Conversion of voice data: The robot receives voice using a microphone and converts it into text data, such as "What's the weather like today?", using a voice recognition engine.

[0140] 3. Sending text data: The robot sends the converted text data to the server via the communication module.

[0141] 4. Analyzing text data: The server receives the text data and analyzes it using the NLP engine.

[0142] 5. Generate a response: The server uses the generative AI model to generate a text response: "Today it's sunny and the temperature is 20 degrees."

[0143] 6. Send Response: The server generates a text response and sends it to the robot.

[0144] 7. Generate voice data: The text response received by the robot is converted into voice data using a voice synthesis engine.

[0145] 8. Voice response: The robot speaks to the user through the speaker, "Today is sunny and the temperature is 20 degrees."

[0146] In this way, the present invention is a system that can realize natural conversations with users and provide high-quality responses. By combining a generative AI model and an NLP engine, it is possible to perform highly accurate text analysis and response generation, thereby alleviating users' feelings of loneliness.

[0147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0148] Step 1:

[0149] The user speaks to the robot. For example, the user says, "What's the weather like today?"

[0150] Input: User's voice data

[0151] Specific operation: The user inputs voice by speaking to the robot.

[0152] Step 2:

[0153] The robot receives the user's voice using a microphone and converts it into text data using a voice recognition engine.

[0154] Input: User's voice data

[0155] Data processing: The speech recognition engine analyzes the voice signal and converts the voice data into text data.

[0156] Output: Text data "What's the weather like today?"

[0157] Specific operation: The robot's microphone receives the user's voice and the voice recognition engine converts it into text data.

[0158] Step 3:

[0159] The robot transmits the converted text data to the server via the communication module.

[0160] Input: Text data "What's the weather like today?"

[0161] Data processing: Converts text data into a format for sending to the server.

[0162] Output: Text data sent to the server

[0163] Specific operation: The robot's communication module sends text data to the server.

[0164] Step 4:

[0165] The server receives the text data and analyzes it using a natural language processing (NLP) engine.

[0166] Input: Text data sent to the server: "What's the weather like today?"

[0167] Data processing: An NLP engine analyzes the text to understand its intent and content.

[0168] Output: Analysis results (user question)

[0169] Specific operation: The server analyzes the text data received using an NLP engine and understands that the user's question is "about the weather."

[0170] Step 5:

[0171] The server uses the generative AI model to generate an appropriate response based on the analysis results.

[0172] Input: Analysis results (user question)

[0173] Data processing: The generative AI model generates an appropriate response based on the prompt.

[0174] Output: Response text "Today is sunny and the temperature is 20 degrees."

[0175] Specific behavior: The server's generative AI model generates the appropriate response: "Today is sunny and the temperature is 20 degrees."

[0176] Step 6:

[0177] The server generates a response text and sends it to the robot.

[0178] Input: Response text "It's sunny today and the temperature is 20 degrees."

[0179] Data processing: Converting response text into a format for sending to the robot.

[0180] Output: Response text sent to the robot

[0181] Specific operation: The text data generated by the server is sent to the robot via the communication module.

[0182] Step 7:

[0183] The robot converts the received response text into voice data using a speech synthesis engine.

[0184] Input: Response text "It's sunny today and the temperature is 20 degrees."

[0185] Data processing: The speech synthesis engine converts text data into speech data.

[0186] Output: Audio data

[0187] Specific operation: The robot's speech synthesis engine analyzes text data and converts it into voice data.

[0188] Step 8:

[0189] The robot provides voice data to the user through a speaker.

[0190] Input: Audio data

[0191] Output: A spoken response to the user: "It's sunny today and the temperature is 20 degrees."

[0192] Specific behavior: The robot's speaker outputs audio data to the user, providing a response.

[0193] In this way, each step works in cooperation to realize natural conversation with the user.

[0194] (Application example 1)

[0195] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0196] Conventional food delivery services require users to perform many manual operations when placing an order, which places a significant burden on users, especially elderly people and those unfamiliar with technology. Furthermore, there is a lack of systems that allow users to intuitively select from a wide variety of menus, hindering user convenience. Given this background, a new interactive ordering assistant is needed to improve user convenience.

[0197] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0198] In this invention, the server includes a means for storing a generative artificial intelligence model, an information processing device for converting voice data into text data and transmitting it, and an information processing device for receiving a response from the server, converting it into voice, and providing it to the user. This allows the user to complete a food delivery order while having a natural conversation through voice input. In addition, by using a natural language processing (NLP) engine and a voice synthesis engine, the user can proceed with the ordering process intuitively and quickly.

[0199] A "generative artificial intelligence model" is an algorithm that learns from huge data sets and performs natural language processing and data generation.

[0200] A "server means" is a computer system for processing data and communicating with other devices and resources.

[0201] "Information processing device means" refers to a device for converting voice data into text data and analyzing and processing the text data.

[0202] A "voice recognition engine" is a software system for converting input voice data into text data.

[0203] A "speech synthesis engine" is a software system for converting text data into speech data.

[0204] A "Natural Language Processing (NLP) engine" is a software system that understands and analyzes human language and generates appropriate responses.

[0205] "Food delivery service" refers to a service that allows users to order food online and have it delivered.

[0206] An "interactive order assistant" is a computer system that assists a user in the ordering process through natural conversation.

[0207] The system for realizing the "interactive order assistant" of the present invention uses the following hardware and software: An information processing device such as a smartphone acts as an interface with the user, and a server performs back-end processing.

[0208] Hardware and software used

[0209] Smartphone (information processing device)

[0210] microphone

[0211] speaker

[0212] server

[0213] Generative AI models (e.g., OpenAI GPT-3 / GPT-4)

[0214] Natural Language Processing (NLP) engines (e.g., Dialogflow)

[0215] Speech recognition engine (e.g. Google Cloud Speech-to-Text API)

[0216] Speech synthesis engine (e.g. Amazon Polly)

[0217] System Operation Overview

[0218] Smartphone side

[0219] When a user orders by voice, the smartphone's microphone captures the voice in real time. The captured voice is converted into text data using a speech recognition engine. This text data is then sent to the server.

[0220] Server side

[0221] The server receives the text data converted by the speech recognition engine. Using a natural language processing (NLP) engine, the server analyzes this text data and understands the user's intent. Based on the analyzed data, the server uses a generative AI model to generate an appropriate response text. This response text is then sent back to the smartphone.

[0222] Smartphone side

[0223] The smartphone receives the response text sent from the server. The received text is converted into voice data by a speech synthesis engine and provided to the user as voice through the smartphone speaker. Through this series of operations, the user can complete the food delivery order interactively.

[0224] Specific examples

[0225] The system is started when a user speaks into their smartphone saying, "I'd like to order a pizza." This speech is converted into text, and the text data "I'd like to order a pizza" is sent to the server. The server analyzes this text data and uses a generative AI model to generate an appropriate response. For example, it generates a response text such as "What kind of pizza would you like?" and sends it to the smartphone. The received text is converted into speech, and the user is asked aloud, "What kind of pizza would you like?" This allows the user to continue ordering by voice.

[0226] Prompt Sentence Examples

[0227] "When a user turns to a food delivery assistant and says, 'I want to order a pizza,' how do we respond?"

[0228] The prompts allow the generative AI model to generate appropriate responses to achieve natural interactions with the user, thus providing an intuitive and efficient food delivery ordering process for users.

[0229] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0230] Step 1:

[0231] A user speaks to a smartphone saying, "I'd like to order a pizza." The smartphone's microphone captures this speech in real time. The input is the user's voice data, and the output is an audio file for processing by the speech recognition engine.

[0232] Step 2:

[0233] The smartphone's speech recognition engine (e.g., Google Cloud Speech-to-Text API) converts the captured voice data into text data. Specifically, it samples the voice data, performs phonemic analysis, and generates the corresponding text data. The input is the captured voice data, and the output is the text data "I would like to order a pizza."

[0234] Step 3:

[0235] The smartphone sends the converted text data to the server. Specifically, it sends the text data to the server via an HTTP request. The input is the text data, and the output is an HTTP request to the server.

[0236] Step 4:

[0237] The server receives the text data sent from the smartphone. The input is the received text data, and the output is to pass the text data to the Natural Language Processing (NLP) system.

[0238] Step 5:

[0239] The server analyzes the text data using a natural language processing (NLP) engine (e.g., Dialogflow). Specifically, the NLP engine performs grammatical and semantic analysis of the text data to understand the user's intent. The input is the received text data, and the output is the analysis result.

[0240] Step 6:

[0241] The server uses a generative AI model (e.g., OpenAI GPT-3 / GPT-4) based on the analysis results to generate an appropriate response text. Specifically, the generative AI model receives the analysis results as a prompt and generates a natural response to the user. The input is the analysis results, and the output is a response text such as "What kind of pizza would you like?"

[0242] Step 7:

[0243] The server sends the generated response text to the smartphone. The input is the generated response text, and the output is the HTTP response to the smartphone.

[0244] Step 8:

[0245] The smartphone receives the response text sent from the server. The input is the received response text, and the output is text data for processing by the speech synthesis engine.

[0246] Step 9:

[0247] The smartphone's speech synthesis engine (e.g., Amazon Polly) converts the received text data into speech data. Specifically, it converts the text data into speech waveforms and generates an audio file. The input is the received text data, and the output is speech data.

[0248] Step 10:

[0249] Voice data is provided to the user through the smartphone speaker. Specifically, voice output is achieved by sending voice data to the speaker. The input is the generated voice data, and the output is the voice saying, "What kind of pizza would you like?"

[0250] The above processing steps enable the user to interactively place a food delivery order.

[0251] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0252] The present invention is a system including a server that stores a generative AI model, a robot that converts voice data into text data and sends it to the server, a robot that receives a response from the server, converts it into voice, and provides it to the user, and an emotion engine that recognizes the user's emotions. Specific embodiments for implementing this system are described in detail below.

[0253] System Configuration

[0254] 1. Server Configuration

[0255] The server contains a generative AI model and a natural language processing (NLP) engine. The server performs the following functions:

[0256] When the voice data is transmitted as text data, the text content is analyzed.

[0257] Uses a Natural Language Processing (NLP) engine to understand the content of the text data sent to it.

[0258] An appropriate response is generated using a generative AI model, and the response is sent to the robot as text data.

[0259] Adjust responses as needed based on emotion engine analysis.

[0260] 2. Robot Configuration

[0261] The robot performs the following functions:

[0262] It includes a microphone for receiving voice input from the user.

[0263] A speech recognition engine is used to convert the user's speech into text data.

[0264] The device includes a communication means for transmitting the converted voice data to the server as text data.

[0265] A communication means is provided for receiving a response from the server.

[0266] A speech synthesis engine is used to convert the received text response into speech and provide it to the user through the speaker.

[0267] An emotion engine is used to recognize the user's emotions from their voice tone and facial expressions, and the information is sent to the server.

[0268] Program Processing

[0269] 1. Server-side processing

[0270] The server receives the text data and emotion data sent by the robot and analyzes them. Specifically, it does the following:

[0271] The server receives from the robot the text data converted by the speech recognition engine and the emotion data analyzed by the emotion engine.

[0272] The received data is analyzed using a natural language processing (NLP) engine to understand its content.

[0273] Based on the generative AI model, an appropriate response text is generated based on the text content and emotional data.

[0274] The generated response text is sent to the robot.

[0275] 2. Processing on the robot side

[0276] The robot converts the user's voice input into text data, and also recognizes the user's emotions using its emotion engine and sends them to the server. Specifically, it works as follows:

[0277] The user speaks to the robot, which receives the voice through a microphone.

[0278] The robot's voice recognition engine converts the voice into text data.

[0279] The robot's emotion engine analyzes the user's tone of voice and facial expressions to generate emotion data.

[0280] The text data and the emotion data are transmitted to a server.

[0281] Receives the response text from the server.

[0282] The received response text is converted into voice data by a speech synthesis engine and provided to the user through a speaker.

[0283] Specific examples

[0284] Example 1: Conversation based on user sentiment

[0285] Consider a scenario where a user asks a robot, "What's the weather like today?", but the user's voice sounds tired.

[0286] 1. The user says, "What's the weather like today?"

[0287] 2. The robot converts this speech into text data: "What's the weather like today?"

[0288] 3. The robot's emotion engine analyzes the tone of the user's voice and generates emotion data such as "tired."

[0289] 4. The robot sends the text data and emotion data to the server.

[0290] 5. The server receives the text data and analyzes it using the NLP engine.

[0291] 6. The server uses the generative AI model to generate a text response: "Today it's sunny and the temperature is 20 degrees."

[0292] 7. Taking into account the emotional data, the server generates a response such as, "Are you feeling tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[0293] 8. The server sends the generated response text to the robot.

[0294] 9. The robot receives the response text and converts it into voice data using a speech synthesis engine.

[0295] 10. The robot asks the user, "Are you feeling tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[0296] In this way, the present invention helps seniors overcome feelings of loneliness through natural conversation and provides responses that take into account their emotional state. By working in tandem with the server and the robot, users can enjoy high-quality communication and emotional understanding.

[0297] The processing flow will be explained below.

[0298] Step 1:

[0299] The user speaks to the robot, "What's the weather like today?"

[0300] Step 2:

[0301] The robot uses a microphone to capture the user's voice.

[0302] Step 3:

[0303] The robot's voice recognition engine converts the captured voice data into text data such as "What's the weather like today?"

[0304] Step 4:

[0305] The robot's emotion engine analyzes the user's tone of voice and facial expressions to generate the emotion data "Tired."

[0306] Step 5:

[0307] The robot sends the converted text data and emotion data to a server via the Internet.

[0308] Step 6:

[0309] The server receives the text data and emotion data sent from the robot.

[0310] Step 7:

[0311] The server's Natural Language Processing (NLP) engine analyzes the received text data, "What's the weather like today?"

[0312] Step 8:

[0313] The server uses an NLP engine to understand that the user is looking for information about the weather.

[0314] Step 9:

[0315] The server's generative AI model generates the basic response text, "Today it's sunny and the temperature is 20 degrees."

[0316] Step 10:

[0317] The server takes into account the received emotion data "Tired" and generates a response text "Are you tired lately? It's sunny and 20 degrees today. It's a good day to relax."

[0318] Step 11:

[0319] The server sends the generated response text to the robot.

[0320] Step 12:

[0321] The robot receives the response text sent from the server.

[0322] Step 13:

[0323] The robot's speech synthesis engine converts the received text data, "Are you tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax." into speech data.

[0324] Step 14:

[0325] The robot provides the converted voice data to the user through a speaker.

[0326] Step 15:

[0327] The user hears a voice response from the robot: "Have you been feeling tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[0328] Example 2

[0329] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0330] Conventional voice assistant systems convert a user's voice into text data and generate responses, but they are unable to provide responses that take the user's emotional state into account, resulting in a problem of not being able to provide communication that fully satisfies the user. For this reason, there is a need for a system that can recognize the user's emotions and generate responses that correspond to those emotions. In addition, responses that are sensitive to the emotions of users who feel lonely, such as the elderly, are particularly important.

[0331] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0332] In this invention, the server includes means for storing a generative artificial intelligence model, means for adjusting responses based on emotional data, and means for analyzing text data using a natural language processing (NLP) engine, thereby generating responses based on the user's emotional state and enabling high-quality communication that gives the user a sense of satisfaction.

[0333] The "server means" is a device that holds a generative artificial intelligence model, analyzes and processes text data and emotional data, generates an appropriate response, and transmits it to the terminal means.

[0334] The "terminal means" is a device that receives voice input from the user, converts it into text data using a voice recognition engine, analyzes the user's emotions using an emotion engine, transmits the data to the server means, and converts the response received from the server means into voice using a voice synthesis engine to provide it to the user.

[0335] A "speech recognition engine" is software or hardware that converts voice input received from a user into text data.

[0336] A "speech synthesis engine" is software or hardware that converts text data into speech data.

[0337] An "emotion engine" is software or a system that analyzes a user's tone of voice and facial expressions, recognizes the user's emotional state, and generates emotion data.

[0338] A "Natural Language Processing (NLP) engine" is software or a system that analyzes natural language text data and understands grammar and meaning.

[0339] A "generative artificial intelligence model" is an algorithm or program that generates appropriate responses based on input data.

[0340] The present invention is a system that analyzes a user's voice input and generates a response according to the user's emotion. This system is composed of a server and a terminal. Specific embodiments of the system are described in detail below.

[0341] Server configuration

[0342] The server includes the following software and hardware:

[0343] 1. Generative AI model: A pre-trained generative AI model (e.g., GPT-3) is used to generate appropriate responses based on the text data sent by the user.

[0344] 2. Natural Language Processing (NLP) engine: Syntax and semantic analysis are performed using an engine for analyzing natural language text data (e.g., spaCy or NLTK).

[0345] 3. Emotional Data Processing Engine: Adjusts the generated response based on the emotional data sent from the terminal means.

[0346] 4. Communication means: Sends and receives data to and from the terminal means via HTTP requests or WebSockets.

[0347] Terminal means configuration

[0348] The terminal includes the following hardware and software:

[0349] 1. Speech recognition engine: An engine that converts voice data from the user into text data (e.g., Google Speech-to-Text API).

[0350] 2. Speech synthesis engine: An engine that converts text data received from a server into voice data (e.g., Microsoft Azure Text-to-Speech).

[0351] 3. Emotion engine: An engine that generates emotion data by analyzing the user's voice tone and facial expressions. It uses various phonological algorithms for voice analysis and facial recognition technology (e.g., OpenCV) for facial expression analysis.

[0352] 4. Communication method: Send and receive data with the server using HTTP requests and WebSockets.

[0353] 5. Microphone and speaker: Equipped with a microphone for receiving voice from the user and a speaker for outputting generated voice data.

[0354] System operation example

[0355] A specific example of operation is shown below.

[0356] Example 1: Conversation based on user sentiment

[0357] 1. The user asks the device, "What's the weather like today?"

[0358] 2. The terminal means receives this voice using its built-in microphone and converts it into text data such as "What's the weather like today?" using its voice recognition engine.

[0359] 3. The emotion engine of the terminal means analyzes the tone of the user's voice and generates emotion data such as "tired."

[0360] 4. The terminal means transmits the text data and emotion data to the server means.

[0361] 5. The server means receives the text data and emotion data, analyzes the text data using an NLP engine, and understands the meaning of "asking about the weather today."

[0362] 6. The server means uses the generative artificial intelligence model to generate a basic response text of "Today it's sunny and the temperature is 20 degrees."

[0363] 7. The server means considers the emotion data and generates a tailored response such as, "Are you feeling tired lately? It's sunny and 20 degrees today. It's a good day to relax."

[0364] 8. The server means transmits the generated response text to the terminal means.

[0365] 9. The terminal means receives the response text from the server means and generates a voice using a speech synthesis engine saying, "Are you tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[0366] 10. The terminal means provides the generated voice to the user through a speaker.

[0367] Examples of prompt statements

[0368] Example prompt 1:

[0369] "What is the robot's response when the user says, 'What's the weather like today?' in a tired tone?"

[0370] An example response generated:

[0371] "Have you been feeling tired lately? It's sunny and 20 degrees today. It's a good day to relax."

[0372] In this way, this system can provide high-quality communication by recognizing the user's emotions and generating appropriate responses based on those emotions.

[0373] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0374] Step 1:

[0375] Receiving audio input

[0376] The user speaks to the device to input voice. The device receives the user's voice using a built-in microphone. Voice data is obtained as input in this step.

[0377] Step 2:

[0378] Speech-to-text

[0379] The device analyzes the received voice data using a voice recognition engine (e.g., Google Speech-to-Text API) and converts it into text data. Specifically, it extracts features from the voice waveform and inputs them into a language model to obtain the corresponding text. The input is voice data, and the output is text data.

[0380] Step 3:

[0381] Emotion recognition

[0382] The device uses an emotion engine to generate emotion data from text data and the user's voice tone and facial expressions. Specifically, it recognizes emotions by analyzing non-verbal features of voice waveforms and visual features from image data. The input is text data and voice / image data, and the output is emotion data.

[0383] Step 4:

[0384] Sending text and emotion data

[0385] The device sends text data and emotion data to the server via HTTP POST requests or WebSockets. The input is the generated text data and emotion data, which are then sent to the server.

[0386] Step 5:

[0387] Receiving and analyzing data

[0388] The server receives the text data and emotion data sent from the device and analyzes the text data using an NLP engine (e.g., spaCy or NLTK). Specifically, it analyzes the text grammatically and extracts meaning. The input is text data, and the output is the semantic information of the analyzed text.

[0389] Step 6:

[0390] Generating a response

[0391] The server uses a generative artificial intelligence model (e.g., GPT-3) to generate a response based on the analyzed text and emotion data. Specifically, it inputs a prompt into the generative model to obtain an appropriate response text. The input is the semantic information and emotion data of the analyzed text, and the output is the response text.

[0392] Step 7:

[0393] Regulating responses based on emotions

[0394] The server adjusts the generated response text based on the emotion data. Specifically, it modifies the text by adding phrases that match the user's emotions or by adding kind words. The input is the generated response text and emotion data, and the output is the adjusted response text.

[0395] Step 8:

[0396] Sending response text

[0397] The server sends the adjusted response text to the terminal. The communication method is HTTP response or WebSocket. The input is the adjusted response text, which is then sent to the terminal.

[0398] Step 9:

[0399] Receiving reply text

[0400] The terminal receives the response text sent from the server via the communication means, and the input is the response text sent from the server.

[0401] Step 10:

[0402] Generate audio data

[0403] The device converts the received response text into voice data using a speech synthesis engine (e.g., Microsoft Azure Text-to-Speech). Specifically, it converts text data into a voice waveform and generates it. The input is the response text data, and the output is the generated voice data.

[0404] Step 11:

[0405] Audio output

[0406] The terminal provides the generated voice data to the user through a speaker. The input is the generated voice data, which is output to the user.

[0407] (Application example 2)

[0408] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0409] In conventional customer service, it has been difficult to properly recognize the user's emotional state and provide services based on that information. Furthermore, improving customer satisfaction has been a challenge because responses that are sensitive to the user's emotions are not provided. Therefore, there is a need for a system that can analyze the user's emotions and provide responses based on those emotions.

[0410] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion recognition engine and means for analyzing the user's emotional state, means for adjusting response text based on emotion data from the robot means, and means for analyzing text data using a natural language processing (NLP) engine and generating a response using a generative artificial intelligence model. This makes it possible to provide a response adapted to the user's emotion.

[0411] A "generative artificial intelligence model" is an AI system that learns from large amounts of data in advance and generates appropriate responses to input text.

[0412] "Server means" refers to a computer system that stores data, processes data, and performs network communication.

[0413] A "robot means" is a device that has the function of acquiring voice data, communicating with a server, and providing a generated response to a user.

[0414] "Voice data" is a digital representation of user-uttered voice information.

[0415] "Text data" refers to character information obtained by converting voice data into text.

[0416] A "speech recognition engine" is a software system that analyzes voice data and converts it into a corresponding string of characters.

[0417] A "speech synthesis engine" is a software system that converts text data into voice data and outputs it as voice.

[0418] An "emotion recognition engine" is a software system that analyzes a user's tone of voice and facial expressions to estimate their emotional state.

[0419] A "Natural Language Processing (NLP) engine" is a technology for analyzing natural language data and understanding its meaning and context.

[0420] "Customer service" is a general term for the product information, support, and assistance that stores and companies provide to users.

[0421] "Response text" is text information generated by the server means as a response to a user's input.

[0422] "Emotion data" is information that represents the emotional state of the user analyzed by the emotion recognition engine.

[0423] The present invention provides a system including server means for storing a generative artificial intelligence model, robot means for converting voice data into text data and sending it to the server means, robot means for converting a response from the server means into voice and providing it to the user, and an emotion recognition engine for analyzing the emotional state of the user. This system enables the provision of a response based on the emotional state of the user in customer service.

[0424] System configuration:

[0425] Specific configurations for implementing the present invention will be described below.

[0426] Server means:

[0427] 1. Generative AI Model:

[0428] The server means stores a generative artificial intelligence model (e.g., GPT-3) that has been trained in advance from large-scale data, and generates a response based on the input text.

[0429] 2. Natural Language Processing (NLP) Engine:

[0430] The server means uses a natural language processing (NLP) engine (e.g., BERT) to analyze the transmitted text data and understand its content.

[0431] 3. Response Adjustment:

[0432] The server means adjusts the generated response text based on the emotion data transmitted by the emotion recognition engine to generate an appropriate response text.

[0433] Robotic Means:

[0434] 1. Speech Recognition Engine:

[0435] The robotic means is equipped with a microphone for receiving voice input from the user and converts the voice into text data using a speech recognition engine (e.g., Google Speech-to-Text API).

[0436] 2. Emotion Recognition Engine:

[0437] The robotic means includes an emotion recognition engine (e.g., Microsoft Azure Emotion API) that analyzes the user's tone of voice and facial expressions to generate emotion data.

[0438] 3. Speech synthesis engine:

[0439] The robot means receives the response text sent from the server means, converts it into voice data using a voice synthesis engine (e.g., Google Text-to-Speech API), and provides it to the user through a speaker.

[0440] Example of how it works:

[0441] 1. User Input:

[0442] Consider a situation where a user speaks to a robot. For example, suppose the user says, "What's your recommendation for today?"

[0443] 2. Speech Recognition:

[0444] The robot means receives the user's voice through a microphone and converts it into text data such as "What's your recommendation today?" using a voice recognition engine.

[0445] 3. Emotion analysis:

[0446] The emotion recognition engine analyzes the user's tone of voice and facial expressions to generate emotion data such as "gratitude" or "interest."

[0447] 4. Data transmission:

[0448] The robot means transmits the generated text data and emotion data to the server means.

[0449] 5. Response Generation:

[0450] The server analyzes the text data using an NLP engine and generates an appropriate response text using a generative artificial intelligence model, adjusting the response by taking into account the emotional data, and generating a response such as "We have a special product for you today. We think you'll be interested."

[0451] 6. Speech synthesis:

[0452] The robot means receives the response text sent from the server means, converts it into voice data using a voice synthesis engine, and provides it to the user through a speaker.

[0453] Examples:

[0454] An example of a prompt is:

[0455] "User sentiment is interesting. Based on recent user sentiment, respond to the following question: What's your recommendation today?"

[0456] In this way, the system of the present invention can improve the quality of customer service by understanding the user's emotions and providing responses based on them.

[0457] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0458] Step 1:

[0459] The user speaks to the robot. Specifically, the user says, "What's your recommendation for today?" The user's voice input is received by the robot's microphone.

[0460] Step 2:

[0461] The robot's speech recognition engine (e.g., Google Speech-to-Text API) converts the voice data into text data. It receives the voice data as input, performs data analysis, and generates the text data "What's your recommendation today?" as output.

[0462] Step 3:

[0463] The robot's emotion recognition engine (e.g., Microsoft Azure Emotion API) analyzes the user's voice tone and facial expressions to generate emotional data. It receives voice tone and facial expression data as input, analyzes the data, and generates emotional data such as "interest" or "gratitude" as output.

[0464] Step 4:

[0465] The robot means transmits the text data and emotion data generated in the previous step to the server means, which receives the text data and emotion data as input and transmits them to the server using the communication means.

[0466] Step 5:

[0467] The server means receives the transmitted text data and emotion data, receives the data transmitted from the robot as input, and stores it as internal data.

[0468] Step 6:

[0469] The server's Natural Language Processing (NLP) engine (e.g., BERT) analyzes and understands the text data. It receives the text data as input, performs data analysis, and generates the processed results as output.

[0470] Step 7:

[0471] The server's artificial intelligence model (e.g., GPT-3) generates an appropriate response text based on the processing results of the NLP engine and the emotional data. It receives the analyzed text data and emotional data as input, performs data generation, and generates the response text as output. For example, it generates the response text, "Today we have a special product we highly recommend. We think you'll be interested."

[0472] Step 8:

[0473] The server means transmits the generated response text to the robot means, which receives the generated response text as input and transmits it to the robot using the communication means.

[0474] Step 9:

[0475] The robot means receives the response text from the server, and receives the response text sent from the server as input and stores it as internal data.

[0476] Step 10:

[0477] The robot's speech synthesis engine (e.g., Google Text-to-Speech API) converts the response text into voice data. It receives the response text as input, performs data conversion, and generates voice data as output.

[0478] Step 11:

[0479] The robot means provides the generated voice data to the user through the speaker. The robot means receives the voice data as input and outputs it through the speaker. The user can receive a voice response such as, "Today we have a special product we highly recommend. I'm sure you'll be interested."

[0480] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0481] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0482] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0483] [Second embodiment]

[0484] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0485] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0486] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0487] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0488] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0489] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0490] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0491] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0492] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0493] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0494] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0495] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0496] The present invention is a system that includes a server that stores a generative AI model, a robot that converts voice data into text data and sends it to the server, and a robot that receives a response from the server, converts it into voice, and provides it to the user. Specific embodiments for implementing this system are described in detail below.

[0497] System Configuration

[0498] 1. Server Configuration

[0499] The server contains a generative AI model and a natural language processing (NLP) engine. The server performs the following functions:

[0500] When the voice data is transmitted as text data, the text content is analyzed.

[0501] Uses a Natural Language Processing (NLP) engine to understand the content of the text data sent to it.

[0502] An appropriate response is generated using a generative AI model, and the response is sent to the robot as text data.

[0503] 2. Robot Configuration

[0504] The robot performs the following functions:

[0505] It has a microphone for receiving voice input from the user.

[0506] A speech recognition engine is used to convert the user's speech into text data.

[0507] It has a communication means for transmitting the converted voice as text data to the server.

[0508] It has a communication means for receiving responses from the server.

[0509] A speech synthesis engine is used to convert the received text response into speech and provide it to the user through the speaker.

[0510] Program Processing

[0511] 1. Server-side processing

[0512] The server receives the text data sent by the robot and analyzes it, specifically as follows:

[0513] The server receives text data converted by a voice recognition engine from the robot.

[0514] The received text data is analyzed using a natural language processing (NLP) engine to understand its content.

[0515] Generate appropriate response text based on a generative AI model.

[0516] The generated response text is sent to the robot.

[0517] 2. Processing on the robot side

[0518] The robot converts the user's voice input into text data and sends it to the server. Specifically, it does the following:

[0519] The user speaks to the robot, which receives the voice through a microphone.

[0520] The robot's voice recognition engine converts the voice into text data.

[0521] The converted text data is sent to the server.

[0522] Receive a response from the server.

[0523] The received response text is converted into voice data by a speech synthesis engine and provided to the user through a speaker.

[0524] Specific examples

[0525] Example 1: A conversation about the weather

[0526] Consider a scenario in which a user asks a robot, "What's the weather like today?"

[0527] 1. The user says, "What's the weather like today?"

[0528] 2. The robot converts this speech into text data: "What's the weather like today?"

[0529] 3. The robot sends the text data to the server.

[0530] 4. The server receives the text data and analyzes it using the NLP engine.

[0531] 5. The server uses the generative AI model to generate a text response: "Today it's sunny and the temperature is 20 degrees."

[0532] 6. The server generates a text response and sends it to the robot.

[0533] 7. The robot converts the received text response, "Today is sunny and the temperature is 20 degrees," into voice data.

[0534] 8. The robot tells the user, "Today is sunny and the temperature is 20 degrees."

[0535] In this way, the present invention provides an effective means for seniors to overcome feelings of loneliness through natural conversation. By having the server and the robot work together, users can enjoy high-quality communication.

[0536] The processing flow will be explained below.

[0537] Step 1:

[0538] The user speaks to the robot, "What's the weather like today?"

[0539] Step 2:

[0540] The robot uses a microphone to capture the user's voice.

[0541] Step 3:

[0542] The robot's voice recognition engine converts the captured voice data into text data such as "What's the weather like today?"

[0543] Step 4:

[0544] The robot sends the converted text data to a server via the Internet.

[0545] Step 5:

[0546] The server receives the text data sent from the robot.

[0547] Step 6:

[0548] The server's Natural Language Processing (NLP) engine analyzes the received text data.

[0549] Step 7:

[0550] The server uses an NLP engine to understand that the user is looking for information about the weather.

[0551] Step 8:

[0552] The server uses a generative AI model to generate a response text such as, "Today it's sunny and the temperature is 20 degrees."

[0553] Step 9:

[0554] The server sends the generated response text to the robot.

[0555] Step 10:

[0556] The robot receives the response text sent from the server.

[0557] Step 11:

[0558] The robot's speech synthesis engine converts the received text data, "Today is sunny and the temperature is 20 degrees," into speech data.

[0559] Step 12:

[0560] The robot provides the converted voice data to the user through a speaker.

[0561] Step 13:

[0562] The user hears a voice response from the robot: "Today is sunny and the temperature is 20 degrees."

[0563] Example 1

[0564] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0565] Conventional speech recognition and response generation systems lack the processing power and response quality required to realize natural conversations with users, resulting in the inability to fully alleviate users' feelings of loneliness. This has been a particular problem for seniors and users living alone, as it has been difficult to provide sufficient psychological support through conversation.

[0566] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0567] In this invention, the server includes means for storing a generative artificial intelligence model, robot means for converting voice data into text data and transmitting it, robot means for receiving a response, converting it into voice and providing it to the user, and means for analyzing the text data using a natural language processing engine and transmitting the generated response to the robot means. This enables natural conversation with the user and provides high-quality responses, thereby alleviating the user's sense of loneliness and providing psychological support.

[0568] A "generative artificial intelligence model" is an artificial intelligence algorithm for generating appropriate responses based on input data.

[0569] A "server means" is a computing device that analyzes data and generates responses.

[0570] A "robot means" is a device that acts as an interface with a user and receives, converts, transmits, and plays back audio.

[0571] A "voice recognition engine" is software for converting voice data into text data.

[0572] A "communication module" is a device or software for sending and receiving data.

[0573] A "speech synthesis engine" is software for converting text data into voice data.

[0574] A "natural language processing engine" is software for analyzing text data and understanding its content.

[0575] "Text data" is data expressed as character information.

[0576] "Audio data" is data that represents an audio signal in digital form.

[0577] "User" refers to a person who uses this system.

[0578] A "prompt sentence" is an input sentence that generates an appropriate response based on a generative artificial intelligence model.

[0579] "Response" refers to the system's response to a user's input.

[0580] This invention relates to a speech recognition and response generation system based on a generative AI model. This system has a series of functions: converting speech data into text data, analyzing the text data to generate an appropriate response, and then converting it back into speech data to provide to the user. The following describes in detail how this system is specifically implemented.

[0581] System Configuration

[0582] 1. Server configuration

[0583] The server contains the following main components:

[0584] Generative AI models: Includes models for generating appropriate responses based on input text data.

[0585] Natural Language Processing (NLP) engine: Includes an engine for analyzing text data and understanding its content.

[0586] The server receives the text data sent from the robot, analyzes it, generates a response using a generative artificial intelligence model, and sends the response back to the robot as text data.

[0587] 2. Robot Configuration

[0588] The robot includes the following main components:

[0589] Microphone: Hardware for receiving audio input from the user.

[0590] Speech recognition engine: Software for converting received voice data into text data (e.g., Google Cloud Speech-to-Text).

[0591] Communication module: A module for sending converted text data to the server and receiving responses from the server.

[0592] Speech synthesis engine: Software (e.g., Amazon Polly) that converts received text responses into speech.

[0593] Speaker: Hardware that outputs converted audio data to the user.

[0594] Specific system processing examples

[0595] As a specific example of system processing, a scenario of a conversation about the weather is shown below.

[0596] Prompt Sentence Examples

[0597] When a user asks the robot, "What's the weather like today?" the prompt is:

[0598] "What's the weather like today?"

[0599] Weather conversation process flow

[0600] 1. Voice input: The user speaks to the robot, "What's the weather like today?"

[0601] 2. Conversion of voice data: The robot receives voice using a microphone and converts it into text data, such as "What's the weather like today?", using a voice recognition engine.

[0602] 3. Sending text data: The robot sends the converted text data to the server via the communication module.

[0603] 4. Analyzing text data: The server receives the text data and analyzes it using the NLP engine.

[0604] 5. Generate a response: The server uses the generative AI model to generate a text response: "Today it's sunny and the temperature is 20 degrees."

[0605] 6. Send Response: The server generates a text response and sends it to the robot.

[0606] 7. Generate voice data: The text response received by the robot is converted into voice data using a voice synthesis engine.

[0607] 8. Voice response: The robot speaks to the user through the speaker, "Today is sunny and the temperature is 20 degrees."

[0608] In this way, the present invention is a system that can realize natural conversations with users and provide high-quality responses. By combining a generative AI model and an NLP engine, it is possible to perform highly accurate text analysis and response generation, thereby alleviating users' feelings of loneliness.

[0609] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0610] Step 1:

[0611] The user speaks to the robot. For example, the user says, "What's the weather like today?"

[0612] Input: User's voice data

[0613] Specific operation: The user inputs voice by speaking to the robot.

[0614] Step 2:

[0615] The robot receives the user's voice using a microphone and converts it into text data using a voice recognition engine.

[0616] Input: User's voice data

[0617] Data processing: The speech recognition engine analyzes the voice signal and converts the voice data into text data.

[0618] Output: Text data "What's the weather like today?"

[0619] Specific operation: The robot's microphone receives the user's voice and the voice recognition engine converts it into text data.

[0620] Step 3:

[0621] The robot transmits the converted text data to the server via the communication module.

[0622] Input: Text data "What's the weather like today?"

[0623] Data processing: Converts text data into a format for sending to the server.

[0624] Output: Text data sent to the server

[0625] Specific operation: The robot's communication module sends text data to the server.

[0626] Step 4:

[0627] The server receives the text data and analyzes it using a natural language processing (NLP) engine.

[0628] Input: Text data sent to the server: "What's the weather like today?"

[0629] Data processing: An NLP engine analyzes the text to understand its intent and content.

[0630] Output: Analysis results (user question)

[0631] Specific operation: The server analyzes the text data received using an NLP engine and understands that the user's question is "about the weather."

[0632] Step 5:

[0633] The server uses the generative AI model to generate an appropriate response based on the analysis results.

[0634] Input: Analysis results (user question)

[0635] Data processing: The generative AI model generates an appropriate response based on the prompt.

[0636] Output: Response text "Today is sunny and the temperature is 20 degrees."

[0637] Specific behavior: The server's generative AI model generates the appropriate response: "Today is sunny and the temperature is 20 degrees."

[0638] Step 6:

[0639] The server generates a response text and sends it to the robot.

[0640] Input: Response text "It's sunny today and the temperature is 20 degrees."

[0641] Data processing: Converting response text into a format for sending to the robot.

[0642] Output: Response text sent to the robot

[0643] Specific operation: The text data generated by the server is sent to the robot via the communication module.

[0644] Step 7:

[0645] The robot converts the received response text into voice data using a speech synthesis engine.

[0646] Input: Response text "It's sunny today and the temperature is 20 degrees."

[0647] Data processing: The speech synthesis engine converts text data into speech data.

[0648] Output: Audio data

[0649] Specific operation: The robot's speech synthesis engine analyzes text data and converts it into voice data.

[0650] Step 8:

[0651] The robot provides voice data to the user through a speaker.

[0652] Input: Audio data

[0653] Output: A spoken response to the user: "It's sunny today and the temperature is 20 degrees."

[0654] Specific behavior: The robot's speaker outputs audio data to the user, providing a response.

[0655] In this way, each step works in cooperation to realize natural conversation with the user.

[0656] (Application example 1)

[0657] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0658] Conventional food delivery services require users to perform many manual operations when placing an order, which places a significant burden on users, especially elderly people and those unfamiliar with technology. Furthermore, there is a lack of systems that allow users to intuitively select from a wide variety of menus, hindering user convenience. Given this background, a new interactive ordering assistant is needed to improve user convenience.

[0659] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0660] In this invention, the server includes a means for storing a generative artificial intelligence model, an information processing device for converting voice data into text data and transmitting it, and an information processing device for receiving a response from the server, converting it into voice, and providing it to the user. This allows the user to complete a food delivery order while having a natural conversation through voice input. In addition, by using a natural language processing (NLP) engine and a voice synthesis engine, the user can proceed with the ordering process intuitively and quickly.

[0661] A "generative artificial intelligence model" is an algorithm that learns from huge data sets and performs natural language processing and data generation.

[0662] A "server means" is a computer system for processing data and communicating with other devices and resources.

[0663] "Information processing device means" refers to a device for converting voice data into text data and analyzing and processing the text data.

[0664] A "voice recognition engine" is a software system for converting input voice data into text data.

[0665] A "speech synthesis engine" is a software system for converting text data into speech data.

[0666] A "Natural Language Processing (NLP) engine" is a software system that understands and analyzes human language and generates appropriate responses.

[0667] "Food delivery service" refers to a service that allows users to order food online and have it delivered.

[0668] An "interactive order assistant" is a computer system that assists a user in the ordering process through natural conversation.

[0669] The system for realizing the "interactive order assistant" of the present invention uses the following hardware and software: An information processing device such as a smartphone acts as an interface with the user, and a server performs back-end processing.

[0670] Hardware and software used

[0671] Smartphone (information processing device)

[0672] microphone

[0673] speaker

[0674] server

[0675] Generative AI models (e.g., OpenAI GPT-3 / GPT-4)

[0676] Natural Language Processing (NLP) engines (e.g., Dialogflow)

[0677] Speech recognition engine (e.g. Google Cloud Speech-to-Text API)

[0678] Speech synthesis engine (e.g. Amazon Polly)

[0679] System Operation Overview

[0680] Smartphone side

[0681] When a user orders by voice, the smartphone's microphone captures the voice in real time. The captured voice is converted into text data using a speech recognition engine. This text data is then sent to the server.

[0682] Server side

[0683] The server receives the text data converted by the speech recognition engine. Using a natural language processing (NLP) engine, the server analyzes this text data and understands the user's intent. Based on the analyzed data, the server uses a generative AI model to generate an appropriate response text. This response text is then sent back to the smartphone.

[0684] Smartphone side

[0685] The smartphone receives the response text sent from the server. The received text is converted into voice data by a speech synthesis engine and provided to the user as voice through the smartphone speaker. Through this series of operations, the user can complete the food delivery order interactively.

[0686] Specific examples

[0687] The system is started when a user speaks into their smartphone saying, "I'd like to order a pizza." This speech is converted into text, and the text data "I'd like to order a pizza" is sent to the server. The server analyzes this text data and uses a generative AI model to generate an appropriate response. For example, it generates a response text such as "What kind of pizza would you like?" and sends it to the smartphone. The received text is converted into speech, and the user is asked aloud, "What kind of pizza would you like?" This allows the user to continue ordering by voice.

[0688] Prompt Sentence Examples

[0689] "When a user turns to a food delivery assistant and says, 'I want to order a pizza,' how do we respond?"

[0690] The prompts allow the generative AI model to generate appropriate responses to achieve natural interactions with the user, thus providing an intuitive and efficient food delivery ordering process for users.

[0691] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0692] Step 1:

[0693] A user speaks to a smartphone saying, "I'd like to order a pizza." The smartphone's microphone captures this speech in real time. The input is the user's voice data, and the output is an audio file for processing by the speech recognition engine.

[0694] Step 2:

[0695] The smartphone's speech recognition engine (e.g., Google Cloud Speech-to-Text API) converts the captured voice data into text data. Specifically, it samples the voice data, performs phonemic analysis, and generates the corresponding text data. The input is the captured voice data, and the output is the text data "I would like to order a pizza."

[0696] Step 3:

[0697] The smartphone sends the converted text data to the server. Specifically, it sends the text data to the server via an HTTP request. The input is the text data, and the output is an HTTP request to the server.

[0698] Step 4:

[0699] The server receives the text data sent from the smartphone. The input is the received text data, and the output is to pass the text data to the Natural Language Processing (NLP) system.

[0700] Step 5:

[0701] The server analyzes the text data using a natural language processing (NLP) engine (e.g., Dialogflow). Specifically, the NLP engine performs grammatical and semantic analysis of the text data to understand the user's intent. The input is the received text data, and the output is the analysis result.

[0702] Step 6:

[0703] The server uses a generative AI model (e.g., OpenAI GPT-3 / GPT-4) based on the analysis results to generate an appropriate response text. Specifically, the generative AI model receives the analysis results as a prompt and generates a natural response to the user. The input is the analysis results, and the output is a response text such as "What kind of pizza would you like?"

[0704] Step 7:

[0705] The server sends the generated response text to the smartphone. The input is the generated response text, and the output is the HTTP response to the smartphone.

[0706] Step 8:

[0707] The smartphone receives the response text sent from the server. The input is the received response text, and the output is text data for processing by the speech synthesis engine.

[0708] Step 9:

[0709] The smartphone's speech synthesis engine (e.g., Amazon Polly) converts the received text data into speech data. Specifically, it converts the text data into speech waveforms and generates an audio file. The input is the received text data, and the output is speech data.

[0710] Step 10:

[0711] Voice data is provided to the user through the smartphone speaker. Specifically, voice output is achieved by sending voice data to the speaker. The input is the generated voice data, and the output is the voice saying, "What kind of pizza would you like?"

[0712] The above processing steps enable the user to interactively place a food delivery order.

[0713] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0714] The present invention is a system including a server that stores a generative AI model, a robot that converts voice data into text data and sends it to the server, a robot that receives a response from the server, converts it into voice, and provides it to the user, and an emotion engine that recognizes the user's emotions. Specific embodiments for implementing this system are described in detail below.

[0715] System Configuration

[0716] 1. Server Configuration

[0717] The server contains a generative AI model and a natural language processing (NLP) engine. The server performs the following functions:

[0718] When the voice data is transmitted as text data, the text content is analyzed.

[0719] Uses a Natural Language Processing (NLP) engine to understand the content of the text data sent to it.

[0720] An appropriate response is generated using a generative AI model, and the response is sent to the robot as text data.

[0721] Adjust responses as needed based on emotion engine analysis.

[0722] 2. Robot Configuration

[0723] The robot performs the following functions:

[0724] It includes a microphone for receiving voice input from the user.

[0725] A speech recognition engine is used to convert the user's speech into text data.

[0726] The device includes a communication means for transmitting the converted voice data to the server as text data.

[0727] A communication means is provided for receiving a response from the server.

[0728] A speech synthesis engine is used to convert the received text response into speech and provide it to the user through the speaker.

[0729] An emotion engine is used to recognize the user's emotions from their voice tone and facial expressions, and the information is sent to the server.

[0730] Program Processing

[0731] 1. Server-side processing

[0732] The server receives the text data and emotion data sent by the robot and analyzes them. Specifically, it does the following:

[0733] The server receives from the robot the text data converted by the speech recognition engine and the emotion data analyzed by the emotion engine.

[0734] The received data is analyzed using a natural language processing (NLP) engine to understand its content.

[0735] Based on the generative AI model, an appropriate response text is generated based on the text content and emotional data.

[0736] The generated response text is sent to the robot.

[0737] 2. Processing on the robot side

[0738] The robot converts the user's voice input into text data, and also recognizes the user's emotions using its emotion engine and sends them to the server. Specifically, it works as follows:

[0739] The user speaks to the robot, which receives the voice through a microphone.

[0740] The robot's voice recognition engine converts the voice into text data.

[0741] The robot's emotion engine analyzes the user's tone of voice and facial expressions to generate emotion data.

[0742] The text data and the emotion data are transmitted to a server.

[0743] Receives the response text from the server.

[0744] The received response text is converted into voice data by a speech synthesis engine and provided to the user through a speaker.

[0745] Specific examples

[0746] Example 1: Conversation based on user sentiment

[0747] Consider a scenario where a user asks a robot, "What's the weather like today?", but the user's voice sounds tired.

[0748] 1. The user says, "What's the weather like today?"

[0749] 2. The robot converts this speech into text data: "What's the weather like today?"

[0750] 3. The robot's emotion engine analyzes the tone of the user's voice and generates emotion data such as "tired."

[0751] 4. The robot sends the text data and emotion data to the server.

[0752] 5. The server receives the text data and analyzes it using the NLP engine.

[0753] 6. The server uses the generative AI model to generate a text response: "Today it's sunny and the temperature is 20 degrees."

[0754] 7. Taking into account the emotional data, the server generates a response such as, "Are you feeling tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[0755] 8. The server sends the generated response text to the robot.

[0756] 9. The robot receives the response text and converts it into voice data using a speech synthesis engine.

[0757] 10. The robot asks the user, "Are you feeling tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[0758] In this way, the present invention helps seniors overcome feelings of loneliness through natural conversation and provides responses that take into account their emotional state. By working in tandem with the server and the robot, users can enjoy high-quality communication and emotional understanding.

[0759] The processing flow will be explained below.

[0760] Step 1:

[0761] The user speaks to the robot, "What's the weather like today?"

[0762] Step 2:

[0763] The robot uses a microphone to capture the user's voice.

[0764] Step 3:

[0765] The robot's voice recognition engine converts the captured voice data into text data such as "What's the weather like today?"

[0766] Step 4:

[0767] The robot's emotion engine analyzes the user's tone of voice and facial expressions to generate the emotion data "Tired."

[0768] Step 5:

[0769] The robot sends the converted text data and emotion data to a server via the Internet.

[0770] Step 6:

[0771] The server receives the text data and emotion data sent from the robot.

[0772] Step 7:

[0773] The server's Natural Language Processing (NLP) engine analyzes the received text data, "What's the weather like today?"

[0774] Step 8:

[0775] The server uses an NLP engine to understand that the user is looking for information about the weather.

[0776] Step 9:

[0777] The server's generative AI model generates the basic response text, "Today it's sunny and the temperature is 20 degrees."

[0778] Step 10:

[0779] The server takes into account the received emotion data "Tired" and generates a response text "Are you tired lately? It's sunny and 20 degrees today. It's a good day to relax."

[0780] Step 11:

[0781] The server sends the generated response text to the robot.

[0782] Step 12:

[0783] The robot receives the response text sent from the server.

[0784] Step 13:

[0785] The robot's speech synthesis engine converts the received text data, "Are you tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax." into speech data.

[0786] Step 14:

[0787] The robot provides the converted voice data to the user through a speaker.

[0788] Step 15:

[0789] The user hears a voice response from the robot: "Have you been feeling tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[0790] Example 2

[0791] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0792] Conventional voice assistant systems convert a user's voice into text data and generate responses, but they are unable to provide responses that take the user's emotional state into account, resulting in a problem of not being able to provide communication that fully satisfies the user. For this reason, there is a need for a system that can recognize the user's emotions and generate responses that correspond to those emotions. In addition, responses that are sensitive to the emotions of users who feel lonely, such as the elderly, are particularly important.

[0793] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0794] In this invention, the server includes means for storing a generative artificial intelligence model, means for adjusting responses based on emotional data, and means for analyzing text data using a natural language processing (NLP) engine, thereby generating responses based on the user's emotional state and enabling high-quality communication that gives the user a sense of satisfaction.

[0795] The "server means" is a device that holds a generative artificial intelligence model, analyzes and processes text data and emotional data, generates an appropriate response, and transmits it to the terminal means.

[0796] The "terminal means" is a device that receives voice input from the user, converts it into text data using a voice recognition engine, analyzes the user's emotions using an emotion engine, transmits the data to the server means, and converts the response received from the server means into voice using a voice synthesis engine to provide it to the user.

[0797] A "speech recognition engine" is software or hardware that converts voice input received from a user into text data.

[0798] A "speech synthesis engine" is software or hardware that converts text data into speech data.

[0799] An "emotion engine" is software or a system that analyzes a user's tone of voice and facial expressions, recognizes the user's emotional state, and generates emotion data.

[0800] A "Natural Language Processing (NLP) engine" is software or a system that analyzes natural language text data and understands grammar and meaning.

[0801] A "generative artificial intelligence model" is an algorithm or program that generates appropriate responses based on input data.

[0802] The present invention is a system that analyzes a user's voice input and generates a response according to the user's emotion. This system is composed of a server and a terminal. Specific embodiments of the system are described in detail below.

[0803] Server configuration

[0804] The server includes the following software and hardware:

[0805] 1. Generative AI model: A pre-trained generative AI model (e.g., GPT-3) is used to generate appropriate responses based on the text data sent by the user.

[0806] 2. Natural Language Processing (NLP) engine: Syntax and semantic analysis are performed using an engine for analyzing natural language text data (e.g., spaCy or NLTK).

[0807] 3. Emotional Data Processing Engine: Adjusts the generated response based on the emotional data sent from the terminal means.

[0808] 4. Communication means: Sends and receives data to and from the terminal means via HTTP requests or WebSockets.

[0809] Terminal means configuration

[0810] The terminal includes the following hardware and software:

[0811] 1. Speech recognition engine: An engine that converts voice data from the user into text data (e.g., Google Speech-to-Text API).

[0812] 2. Speech synthesis engine: An engine that converts text data received from a server into voice data (e.g., Microsoft Azure Text-to-Speech).

[0813] 3. Emotion engine: An engine that generates emotion data by analyzing the user's voice tone and facial expressions. It uses various phonological algorithms for voice analysis and facial recognition technology (e.g., OpenCV) for facial expression analysis.

[0814] 4. Communication method: Send and receive data with the server using HTTP requests and WebSockets.

[0815] 5. Microphone and speaker: Equipped with a microphone for receiving voice from the user and a speaker for outputting generated voice data.

[0816] System operation example

[0817] A specific example of operation is shown below.

[0818] Example 1: Conversation based on user sentiment

[0819] 1. The user asks the device, "What's the weather like today?"

[0820] 2. The terminal means receives this voice using its built-in microphone and converts it into text data such as "What's the weather like today?" using its voice recognition engine.

[0821] 3. The emotion engine of the terminal means analyzes the tone of the user's voice and generates emotion data such as "tired."

[0822] 4. The terminal means transmits the text data and emotion data to the server means.

[0823] 5. The server means receives the text data and emotion data, analyzes the text data using an NLP engine, and understands the meaning of "asking about the weather today."

[0824] 6. The server means uses the generative artificial intelligence model to generate a basic response text of "Today it's sunny and the temperature is 20 degrees."

[0825] 7. The server means considers the emotion data and generates a tailored response such as, "Are you feeling tired lately? It's sunny and 20 degrees today. It's a good day to relax."

[0826] 8. The server means transmits the generated response text to the terminal means.

[0827] 9. The terminal means receives the response text from the server means and generates a voice using a speech synthesis engine saying, "Are you tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[0828] 10. The terminal means provides the generated voice to the user through a speaker.

[0829] Examples of prompt statements

[0830] Example prompt 1:

[0831] "What is the robot's response when the user says, 'What's the weather like today?' in a tired tone?"

[0832] An example response generated:

[0833] "Have you been feeling tired lately? It's sunny and 20 degrees today. It's a good day to relax."

[0834] In this way, this system can provide high-quality communication by recognizing the user's emotions and generating appropriate responses based on those emotions.

[0835] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0836] Step 1:

[0837] Receiving audio input

[0838] The user speaks to the device to input voice. The device receives the user's voice using a built-in microphone. Voice data is obtained as input in this step.

[0839] Step 2:

[0840] Speech-to-text

[0841] The device analyzes the received voice data using a voice recognition engine (e.g., Google Speech-to-Text API) and converts it into text data. Specifically, it extracts features from the voice waveform and inputs them into a language model to obtain the corresponding text. The input is voice data, and the output is text data.

[0842] Step 3:

[0843] Emotion recognition

[0844] The device uses an emotion engine to generate emotion data from text data and the user's voice tone and facial expressions. Specifically, it recognizes emotions by analyzing non-verbal features of voice waveforms and visual features from image data. The input is text data and voice / image data, and the output is emotion data.

[0845] Step 4:

[0846] Sending text and emotion data

[0847] The device sends text data and emotion data to the server via HTTP POST requests or WebSockets. The input is the generated text data and emotion data, which are then sent to the server.

[0848] Step 5:

[0849] Receiving and analyzing data

[0850] The server receives the text data and emotion data sent from the device and analyzes the text data using an NLP engine (e.g., spaCy or NLTK). Specifically, it analyzes the text grammatically and extracts meaning. The input is text data, and the output is the semantic information of the analyzed text.

[0851] Step 6:

[0852] Generating a response

[0853] The server uses a generative artificial intelligence model (e.g., GPT-3) to generate a response based on the analyzed text and emotion data. Specifically, it inputs a prompt into the generative model to obtain an appropriate response text. The input is the semantic information and emotion data of the analyzed text, and the output is the response text.

[0854] Step 7:

[0855] Regulating responses based on emotions

[0856] The server adjusts the generated response text based on the emotion data. Specifically, it modifies the text by adding phrases that match the user's emotions or by adding kind words. The input is the generated response text and emotion data, and the output is the adjusted response text.

[0857] Step 8:

[0858] Sending response text

[0859] The server sends the adjusted response text to the terminal. The communication method is HTTP response or WebSocket. The input is the adjusted response text, which is then sent to the terminal.

[0860] Step 9:

[0861] Receiving reply text

[0862] The terminal receives the response text sent from the server via the communication means, and the input is the response text sent from the server.

[0863] Step 10:

[0864] Generate audio data

[0865] The device converts the received response text into voice data using a speech synthesis engine (e.g., Microsoft Azure Text-to-Speech). Specifically, it converts text data into a voice waveform and generates it. The input is the response text data, and the output is the generated voice data.

[0866] Step 11:

[0867] Audio output

[0868] The terminal provides the generated voice data to the user through a speaker. The input is the generated voice data, which is output to the user.

[0869] (Application example 2)

[0870] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0871] In conventional customer service, it has been difficult to properly recognize the user's emotional state and provide services based on that information. Furthermore, improving customer satisfaction has been a challenge because responses that are sensitive to the user's emotions are not provided. Therefore, there is a need for a system that can analyze the user's emotions and provide responses based on those emotions.

[0872] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion recognition engine and means for analyzing the user's emotional state, means for adjusting response text based on emotion data from the robot means, and means for analyzing text data using a natural language processing (NLP) engine and generating a response using a generative artificial intelligence model. This makes it possible to provide a response adapted to the user's emotion.

[0873] A "generative artificial intelligence model" is an AI system that learns from large amounts of data in advance and generates appropriate responses to input text.

[0874] "Server means" refers to a computer system that stores data, processes data, and performs network communication.

[0875] A "robot means" is a device that has the function of acquiring voice data, communicating with a server, and providing a generated response to a user.

[0876] "Voice data" is a digital representation of user-uttered voice information.

[0877] "Text data" refers to character information obtained by converting voice data into text.

[0878] A "speech recognition engine" is a software system that analyzes voice data and converts it into a corresponding string of characters.

[0879] A "speech synthesis engine" is a software system that converts text data into voice data and outputs it as voice.

[0880] An "emotion recognition engine" is a software system that analyzes a user's tone of voice and facial expressions to estimate their emotional state.

[0881] A "Natural Language Processing (NLP) engine" is a technology for analyzing natural language data and understanding its meaning and context.

[0882] "Customer service" is a general term for the product information, support, and assistance that stores and companies provide to users.

[0883] "Response text" is text information generated by the server means as a response to a user's input.

[0884] "Emotion data" is information that represents the emotional state of the user analyzed by the emotion recognition engine.

[0885] The present invention provides a system including server means for storing a generative artificial intelligence model, robot means for converting voice data into text data and sending it to the server means, robot means for converting a response from the server means into voice and providing it to the user, and an emotion recognition engine for analyzing the emotional state of the user. This system enables the provision of a response based on the emotional state of the user in customer service.

[0886] System configuration:

[0887] Specific configurations for implementing the present invention will be described below.

[0888] Server means:

[0889] 1. Generative AI Model:

[0890] The server means stores a generative artificial intelligence model (e.g., GPT-3) that has been trained in advance from large-scale data, and generates a response based on the input text.

[0891] 2. Natural Language Processing (NLP) Engine:

[0892] The server means uses a natural language processing (NLP) engine (e.g., BERT) to analyze the transmitted text data and understand its content.

[0893] 3. Response Adjustment:

[0894] The server means adjusts the generated response text based on the emotion data transmitted by the emotion recognition engine to generate an appropriate response text.

[0895] Robotic Means:

[0896] 1. Speech Recognition Engine:

[0897] The robotic means is equipped with a microphone for receiving voice input from the user and converts the voice into text data using a speech recognition engine (e.g., Google Speech-to-Text API).

[0898] 2. Emotion Recognition Engine:

[0899] The robotic means includes an emotion recognition engine (e.g., Microsoft Azure Emotion API) that analyzes the user's tone of voice and facial expressions to generate emotion data.

[0900] 3. Speech synthesis engine:

[0901] The robot means receives the response text sent from the server means, converts it into voice data using a voice synthesis engine (e.g., Google Text-to-Speech API), and provides it to the user through a speaker.

[0902] Example of how it works:

[0903] 1. User Input:

[0904] Consider a situation where a user speaks to a robot. For example, suppose the user says, "What's your recommendation for today?"

[0905] 2. Speech Recognition:

[0906] The robot means receives the user's voice through a microphone and converts it into text data such as "What's your recommendation today?" using a voice recognition engine.

[0907] 3. Emotion analysis:

[0908] The emotion recognition engine analyzes the user's tone of voice and facial expressions to generate emotion data such as "gratitude" or "interest."

[0909] 4. Data transmission:

[0910] The robot means transmits the generated text data and emotion data to the server means.

[0911] 5. Response Generation:

[0912] The server analyzes the text data using an NLP engine and generates an appropriate response text using a generative artificial intelligence model, adjusting the response by taking into account the emotional data, and generating a response such as "We have a special product for you today. We think you'll be interested."

[0913] 6. Speech synthesis:

[0914] The robot means receives the response text sent from the server means, converts it into voice data using a voice synthesis engine, and provides it to the user through a speaker.

[0915] Examples:

[0916] An example of a prompt is:

[0917] "User sentiment is interesting. Based on recent user sentiment, respond to the following question: What's your recommendation today?"

[0918] In this way, the system of the present invention can improve the quality of customer service by understanding the user's emotions and providing responses based on them.

[0919] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0920] Step 1:

[0921] The user speaks to the robot. Specifically, the user says, "What's your recommendation for today?" The user's voice input is received by the robot's microphone.

[0922] Step 2:

[0923] The robot's speech recognition engine (e.g., Google Speech-to-Text API) converts the voice data into text data. It receives the voice data as input, performs data analysis, and generates the text data "What's your recommendation today?" as output.

[0924] Step 3:

[0925] The robot's emotion recognition engine (e.g., Microsoft Azure Emotion API) analyzes the user's voice tone and facial expressions to generate emotional data. It receives voice tone and facial expression data as input, analyzes the data, and generates emotional data such as "interest" or "gratitude" as output.

[0926] Step 4:

[0927] The robot means transmits the text data and emotion data generated in the previous step to the server means, which receives the text data and emotion data as input and transmits them to the server using the communication means.

[0928] Step 5:

[0929] The server means receives the transmitted text data and emotion data, receives the data transmitted from the robot as input, and stores it as internal data.

[0930] Step 6:

[0931] The server's Natural Language Processing (NLP) engine (e.g., BERT) analyzes and understands the text data. It receives the text data as input, performs data analysis, and generates the processed results as output.

[0932] Step 7:

[0933] The server's artificial intelligence model (e.g., GPT-3) generates an appropriate response text based on the processing results of the NLP engine and the emotional data. It receives the analyzed text data and emotional data as input, performs data generation, and generates the response text as output. For example, it generates the response text, "Today we have a special product we highly recommend. We think you'll be interested."

[0934] Step 8:

[0935] The server means transmits the generated response text to the robot means, which receives the generated response text as input and transmits it to the robot using the communication means.

[0936] Step 9:

[0937] The robot means receives the response text from the server, and receives the response text sent from the server as input and stores it as internal data.

[0938] Step 10:

[0939] The robot's speech synthesis engine (e.g., Google Text-to-Speech API) converts the response text into voice data. It receives the response text as input, performs data conversion, and generates voice data as output.

[0940] Step 11:

[0941] The robot means provides the generated voice data to the user through the speaker. The robot means receives the voice data as input and outputs it through the speaker. The user can receive a voice response such as, "Today we have a special product we highly recommend. I'm sure you'll be interested."

[0942] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0943] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0944] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0945] [Third embodiment]

[0946] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0947] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0948] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0949] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0950] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0951] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0952] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0953] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0954] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0955] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0956] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0957] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0958] The present invention is a system that includes a server that stores a generative AI model, a robot that converts voice data into text data and sends it to the server, and a robot that receives a response from the server, converts it into voice, and provides it to the user. Specific embodiments for implementing this system are described in detail below.

[0959] System Configuration

[0960] 1. Server Configuration

[0961] The server contains a generative AI model and a natural language processing (NLP) engine. The server performs the following functions:

[0962] When the voice data is transmitted as text data, the text content is analyzed.

[0963] Uses a Natural Language Processing (NLP) engine to understand the content of the text data sent to it.

[0964] An appropriate response is generated using a generative AI model, and the response is sent to the robot as text data.

[0965] 2. Robot Configuration

[0966] The robot performs the following functions:

[0967] It has a microphone for receiving voice input from the user.

[0968] A speech recognition engine is used to convert the user's speech into text data.

[0969] It has a communication means for transmitting the converted voice as text data to the server.

[0970] It has a communication means for receiving responses from the server.

[0971] A speech synthesis engine is used to convert the received text response into speech and provide it to the user through the speaker.

[0972] Program Processing

[0973] 1. Server-side processing

[0974] The server receives the text data sent by the robot and analyzes it, specifically as follows:

[0975] The server receives text data converted by a voice recognition engine from the robot.

[0976] The received text data is analyzed using a natural language processing (NLP) engine to understand its content.

[0977] Generate appropriate response text based on a generative AI model.

[0978] The generated response text is sent to the robot.

[0979] 2. Processing on the robot side

[0980] The robot converts the user's voice input into text data and sends it to the server. Specifically, it does the following:

[0981] The user speaks to the robot, which receives the voice through a microphone.

[0982] The robot's voice recognition engine converts the voice into text data.

[0983] The converted text data is sent to the server.

[0984] Receive a response from the server.

[0985] The received response text is converted into voice data by a speech synthesis engine and provided to the user through a speaker.

[0986] Specific examples

[0987] Example 1: A conversation about the weather

[0988] Consider a scenario in which a user asks a robot, "What's the weather like today?"

[0989] 1. The user says, "What's the weather like today?"

[0990] 2. The robot converts this speech into text data: "What's the weather like today?"

[0991] 3. The robot sends the text data to the server.

[0992] 4. The server receives the text data and analyzes it using the NLP engine.

[0993] 5. The server uses the generative AI model to generate a text response: "Today it's sunny and the temperature is 20 degrees."

[0994] 6. The server generates a text response and sends it to the robot.

[0995] 7. The robot converts the received text response, "Today is sunny and the temperature is 20 degrees," into voice data.

[0996] 8. The robot tells the user, "Today is sunny and the temperature is 20 degrees."

[0997] In this way, the present invention provides an effective means for seniors to overcome feelings of loneliness through natural conversation. By having the server and the robot work together, users can enjoy high-quality communication.

[0998] The processing flow will be explained below.

[0999] Step 1:

[1000] The user speaks to the robot, "What's the weather like today?"

[1001] Step 2:

[1002] The robot uses a microphone to capture the user's voice.

[1003] Step 3:

[1004] The robot's voice recognition engine converts the captured voice data into text data such as "What's the weather like today?"

[1005] Step 4:

[1006] The robot sends the converted text data to a server via the Internet.

[1007] Step 5:

[1008] The server receives the text data sent from the robot.

[1009] Step 6:

[1010] The server's Natural Language Processing (NLP) engine analyzes the received text data.

[1011] Step 7:

[1012] The server uses an NLP engine to understand that the user is looking for information about the weather.

[1013] Step 8:

[1014] The server uses a generative AI model to generate a response text such as, "Today it's sunny and the temperature is 20 degrees."

[1015] Step 9:

[1016] The server sends the generated response text to the robot.

[1017] Step 10:

[1018] The robot receives the response text sent from the server.

[1019] Step 11:

[1020] The robot's speech synthesis engine converts the received text data, "Today is sunny and the temperature is 20 degrees," into speech data.

[1021] Step 12:

[1022] The robot provides the converted voice data to the user through a speaker.

[1023] Step 13:

[1024] The user hears a voice response from the robot: "Today is sunny and the temperature is 20 degrees."

[1025] Example 1

[1026] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1027] Conventional speech recognition and response generation systems lack the processing power and response quality required to realize natural conversations with users, resulting in the inability to fully alleviate users' feelings of loneliness. This has been a particular problem for seniors and users living alone, as it has been difficult to provide sufficient psychological support through conversation.

[1028] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1029] In this invention, the server includes means for storing a generative artificial intelligence model, robot means for converting voice data into text data and transmitting it, robot means for receiving a response, converting it into voice and providing it to the user, and means for analyzing the text data using a natural language processing engine and transmitting the generated response to the robot means. This enables natural conversation with the user and provides high-quality responses, thereby alleviating the user's sense of loneliness and providing psychological support.

[1030] A "generative artificial intelligence model" is an artificial intelligence algorithm for generating appropriate responses based on input data.

[1031] A "server means" is a computing device that analyzes data and generates responses.

[1032] A "robot means" is a device that acts as an interface with a user and receives, converts, transmits, and plays back audio.

[1033] A "voice recognition engine" is software for converting voice data into text data.

[1034] A "communication module" is a device or software for sending and receiving data.

[1035] A "speech synthesis engine" is software for converting text data into voice data.

[1036] A "natural language processing engine" is software for analyzing text data and understanding its content.

[1037] "Text data" is data expressed as character information.

[1038] "Audio data" is data that represents an audio signal in digital form.

[1039] "User" refers to a person who uses this system.

[1040] A "prompt sentence" is an input sentence that generates an appropriate response based on a generative artificial intelligence model.

[1041] "Response" refers to the system's response to a user's input.

[1042] This invention relates to a speech recognition and response generation system based on a generative AI model. This system has a series of functions: converting speech data into text data, analyzing the text data to generate an appropriate response, and then converting it back into speech data to provide to the user. The following describes in detail how this system is specifically implemented.

[1043] System Configuration

[1044] 1. Server configuration

[1045] The server contains the following main components:

[1046] Generative AI models: Includes models for generating appropriate responses based on input text data.

[1047] Natural Language Processing (NLP) engine: Includes an engine for analyzing text data and understanding its content.

[1048] The server receives the text data sent from the robot, analyzes it, generates a response using a generative artificial intelligence model, and sends the response back to the robot as text data.

[1049] 2. Robot Configuration

[1050] The robot includes the following main components:

[1051] Microphone: Hardware for receiving audio input from the user.

[1052] Speech recognition engine: Software for converting received voice data into text data (e.g., Google Cloud Speech-to-Text).

[1053] Communication module: A module for sending converted text data to the server and receiving responses from the server.

[1054] Speech synthesis engine: Software (e.g., Amazon Polly) that converts received text responses into speech.

[1055] Speaker: Hardware that outputs converted audio data to the user.

[1056] Specific system processing examples

[1057] As a specific example of system processing, a scenario of a conversation about the weather is shown below.

[1058] Prompt Sentence Examples

[1059] When a user asks the robot, "What's the weather like today?" the prompt is:

[1060] "What's the weather like today?"

[1061] Weather conversation process flow

[1062] 1. Voice input: The user speaks to the robot, "What's the weather like today?"

[1063] 2. Conversion of voice data: The robot receives voice using a microphone and converts it into text data, such as "What's the weather like today?", using a voice recognition engine.

[1064] 3. Sending text data: The robot sends the converted text data to the server via the communication module.

[1065] 4. Analyzing text data: The server receives the text data and analyzes it using the NLP engine.

[1066] 5. Generate a response: The server uses the generative AI model to generate a text response: "Today it's sunny and the temperature is 20 degrees."

[1067] 6. Send Response: The server generates a text response and sends it to the robot.

[1068] 7. Generate voice data: The text response received by the robot is converted into voice data using a voice synthesis engine.

[1069] 8. Voice response: The robot speaks to the user through the speaker, "Today is sunny and the temperature is 20 degrees."

[1070] In this way, the present invention is a system that can realize natural conversations with users and provide high-quality responses. By combining a generative AI model and an NLP engine, it is possible to perform highly accurate text analysis and response generation, thereby alleviating users' feelings of loneliness.

[1071] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1072] Step 1:

[1073] The user speaks to the robot. For example, the user says, "What's the weather like today?"

[1074] Input: User's voice data

[1075] Specific operation: The user inputs voice by speaking to the robot.

[1076] Step 2:

[1077] The robot receives the user's voice using a microphone and converts it into text data using a voice recognition engine.

[1078] Input: User's voice data

[1079] Data processing: The speech recognition engine analyzes the voice signal and converts the voice data into text data.

[1080] Output: Text data "What's the weather like today?"

[1081] Specific operation: The robot's microphone receives the user's voice and the voice recognition engine converts it into text data.

[1082] Step 3:

[1083] The robot transmits the converted text data to the server via the communication module.

[1084] Input: Text data "What's the weather like today?"

[1085] Data processing: Converts text data into a format for sending to the server.

[1086] Output: Text data sent to the server

[1087] Specific operation: The robot's communication module sends text data to the server.

[1088] Step 4:

[1089] The server receives the text data and analyzes it using a natural language processing (NLP) engine.

[1090] Input: Text data sent to the server: "What's the weather like today?"

[1091] Data processing: An NLP engine analyzes the text to understand its intent and content.

[1092] Output: Analysis results (user question)

[1093] Specific operation: The server analyzes the text data received using an NLP engine and understands that the user's question is "about the weather."

[1094] Step 5:

[1095] The server uses the generative AI model to generate an appropriate response based on the analysis results.

[1096] Input: Analysis results (user question)

[1097] Data processing: The generative AI model generates an appropriate response based on the prompt.

[1098] Output: Response text "Today is sunny and the temperature is 20 degrees."

[1099] Specific behavior: The server's generative AI model generates the appropriate response: "Today is sunny and the temperature is 20 degrees."

[1100] Step 6:

[1101] The server generates a response text and sends it to the robot.

[1102] Input: Response text "It's sunny today and the temperature is 20 degrees."

[1103] Data processing: Converting response text into a format for sending to the robot.

[1104] Output: Response text sent to the robot

[1105] Specific operation: The text data generated by the server is sent to the robot via the communication module.

[1106] Step 7:

[1107] The robot converts the received response text into voice data using a speech synthesis engine.

[1108] Input: Response text "It's sunny today and the temperature is 20 degrees."

[1109] Data processing: The speech synthesis engine converts text data into speech data.

[1110] Output: Audio data

[1111] Specific operation: The robot's speech synthesis engine analyzes text data and converts it into voice data.

[1112] Step 8:

[1113] The robot provides voice data to the user through a speaker.

[1114] Input: Audio data

[1115] Output: A spoken response to the user: "It's sunny today and the temperature is 20 degrees."

[1116] Specific behavior: The robot's speaker outputs audio data to the user, providing a response.

[1117] In this way, each step works in cooperation to realize natural conversation with the user.

[1118] (Application example 1)

[1119] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1120] Conventional food delivery services require users to perform many manual operations when placing an order, which places a significant burden on users, especially elderly people and those unfamiliar with technology. Furthermore, there is a lack of systems that allow users to intuitively select from a wide variety of menus, hindering user convenience. Given this background, a new interactive ordering assistant is needed to improve user convenience.

[1121] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1122] In this invention, the server includes a means for storing a generative artificial intelligence model, an information processing device for converting voice data into text data and transmitting it, and an information processing device for receiving a response from the server, converting it into voice, and providing it to the user. This allows the user to complete a food delivery order while having a natural conversation through voice input. In addition, by using a natural language processing (NLP) engine and a voice synthesis engine, the user can proceed with the ordering process intuitively and quickly.

[1123] A "generative artificial intelligence model" is an algorithm that learns from huge data sets and performs natural language processing and data generation.

[1124] A "server means" is a computer system for processing data and communicating with other devices and resources.

[1125] "Information processing device means" refers to a device for converting voice data into text data and analyzing and processing the text data.

[1126] A "voice recognition engine" is a software system for converting input voice data into text data.

[1127] A "speech synthesis engine" is a software system for converting text data into speech data.

[1128] A "Natural Language Processing (NLP) engine" is a software system that understands and analyzes human language and generates appropriate responses.

[1129] "Food delivery service" refers to a service that allows users to order food online and have it delivered.

[1130] An "interactive order assistant" is a computer system that assists a user in the ordering process through natural conversation.

[1131] The system for realizing the "interactive order assistant" of the present invention uses the following hardware and software: An information processing device such as a smartphone acts as an interface with the user, and a server performs back-end processing.

[1132] Hardware and software used

[1133] Smartphone (information processing device)

[1134] microphone

[1135] speaker

[1136] server

[1137] Generative AI models (e.g., OpenAI GPT-3 / GPT-4)

[1138] Natural Language Processing (NLP) engines (e.g., Dialogflow)

[1139] Speech recognition engine (e.g. Google Cloud Speech-to-Text API)

[1140] Speech synthesis engine (e.g. Amazon Polly)

[1141] System Operation Overview

[1142] Smartphone side

[1143] When a user orders by voice, the smartphone's microphone captures the voice in real time. The captured voice is converted into text data using a speech recognition engine. This text data is then sent to the server.

[1144] Server side

[1145] The server receives the text data converted by the speech recognition engine. Using a natural language processing (NLP) engine, the server analyzes this text data and understands the user's intent. Based on the analyzed data, the server uses a generative AI model to generate an appropriate response text. This response text is then sent back to the smartphone.

[1146] Smartphone side

[1147] The smartphone receives the response text sent from the server. The received text is converted into voice data by a speech synthesis engine and provided to the user as voice through the smartphone speaker. Through this series of operations, the user can complete the food delivery order interactively.

[1148] Specific examples

[1149] The system is started when a user speaks into their smartphone saying, "I'd like to order a pizza." This speech is converted into text, and the text data "I'd like to order a pizza" is sent to the server. The server analyzes this text data and uses a generative AI model to generate an appropriate response. For example, it generates a response text such as "What kind of pizza would you like?" and sends it to the smartphone. The received text is converted into speech, and the user is asked aloud, "What kind of pizza would you like?" This allows the user to continue ordering by voice.

[1150] Prompt Sentence Examples

[1151] "When a user turns to a food delivery assistant and says, 'I want to order a pizza,' how do we respond?"

[1152] The prompts allow the generative AI model to generate appropriate responses to achieve natural interactions with the user, thus providing an intuitive and efficient food delivery ordering process for users.

[1153] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1154] Step 1:

[1155] A user speaks to a smartphone saying, "I'd like to order a pizza." The smartphone's microphone captures this speech in real time. The input is the user's voice data, and the output is an audio file for processing by the speech recognition engine.

[1156] Step 2:

[1157] The smartphone's speech recognition engine (e.g., Google Cloud Speech-to-Text API) converts the captured voice data into text data. Specifically, it samples the voice data, performs phonemic analysis, and generates the corresponding text data. The input is the captured voice data, and the output is the text data "I would like to order a pizza."

[1158] Step 3:

[1159] The smartphone sends the converted text data to the server. Specifically, it sends the text data to the server via an HTTP request. The input is the text data, and the output is an HTTP request to the server.

[1160] Step 4:

[1161] The server receives the text data sent from the smartphone. The input is the received text data, and the output is to pass the text data to the Natural Language Processing (NLP) system.

[1162] Step 5:

[1163] The server analyzes the text data using a natural language processing (NLP) engine (e.g., Dialogflow). Specifically, the NLP engine performs grammatical and semantic analysis of the text data to understand the user's intent. The input is the received text data, and the output is the analysis result.

[1164] Step 6:

[1165] The server uses a generative AI model (e.g., OpenAI GPT-3 / GPT-4) based on the analysis results to generate an appropriate response text. Specifically, the generative AI model receives the analysis results as a prompt and generates a natural response to the user. The input is the analysis results, and the output is a response text such as "What kind of pizza would you like?"

[1166] Step 7:

[1167] The server sends the generated response text to the smartphone. The input is the generated response text, and the output is the HTTP response to the smartphone.

[1168] Step 8:

[1169] The smartphone receives the response text sent from the server. The input is the received response text, and the output is text data for processing by the speech synthesis engine.

[1170] Step 9:

[1171] The smartphone's speech synthesis engine (e.g., Amazon Polly) converts the received text data into speech data. Specifically, it converts the text data into speech waveforms and generates an audio file. The input is the received text data, and the output is speech data.

[1172] Step 10:

[1173] Voice data is provided to the user through the smartphone speaker. Specifically, voice output is achieved by sending voice data to the speaker. The input is the generated voice data, and the output is the voice saying, "What kind of pizza would you like?"

[1174] The above processing steps enable the user to interactively place a food delivery order.

[1175] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1176] The present invention is a system including a server that stores a generative AI model, a robot that converts voice data into text data and sends it to the server, a robot that receives a response from the server, converts it into voice, and provides it to the user, and an emotion engine that recognizes the user's emotions. Specific embodiments for implementing this system are described in detail below.

[1177] System Configuration

[1178] 1. Server Configuration

[1179] The server contains a generative AI model and a natural language processing (NLP) engine. The server performs the following functions:

[1180] When the voice data is transmitted as text data, the text content is analyzed.

[1181] Uses a Natural Language Processing (NLP) engine to understand the content of the text data sent to it.

[1182] An appropriate response is generated using a generative AI model, and the response is sent to the robot as text data.

[1183] Adjust responses as needed based on emotion engine analysis.

[1184] 2. Robot Configuration

[1185] The robot performs the following functions:

[1186] It includes a microphone for receiving voice input from the user.

[1187] A speech recognition engine is used to convert the user's speech into text data.

[1188] The device includes a communication means for transmitting the converted voice data to the server as text data.

[1189] A communication means is provided for receiving a response from the server.

[1190] A speech synthesis engine is used to convert the received text response into speech and provide it to the user through the speaker.

[1191] An emotion engine is used to recognize the user's emotions from their voice tone and facial expressions, and the information is sent to the server.

[1192] Program Processing

[1193] 1. Server-side processing

[1194] The server receives the text data and emotion data sent by the robot and analyzes them. Specifically, it does the following:

[1195] The server receives from the robot the text data converted by the speech recognition engine and the emotion data analyzed by the emotion engine.

[1196] The received data is analyzed using a natural language processing (NLP) engine to understand its content.

[1197] Based on the generative AI model, an appropriate response text is generated based on the text content and emotional data.

[1198] The generated response text is sent to the robot.

[1199] 2. Processing on the robot side

[1200] The robot converts the user's voice input into text data, and also recognizes the user's emotions using its emotion engine and sends them to the server. Specifically, it works as follows:

[1201] The user speaks to the robot, which receives the voice through a microphone.

[1202] The robot's voice recognition engine converts the voice into text data.

[1203] The robot's emotion engine analyzes the user's tone of voice and facial expressions to generate emotion data.

[1204] The text data and the emotion data are transmitted to a server.

[1205] Receives the response text from the server.

[1206] The received response text is converted into voice data by a speech synthesis engine and provided to the user through a speaker.

[1207] Specific examples

[1208] Example 1: Conversation based on user sentiment

[1209] Consider a scenario where a user asks a robot, "What's the weather like today?", but the user's voice sounds tired.

[1210] 1. The user says, "What's the weather like today?"

[1211] 2. The robot converts this speech into text data: "What's the weather like today?"

[1212] 3. The robot's emotion engine analyzes the tone of the user's voice and generates emotion data such as "tired."

[1213] 4. The robot sends the text data and emotion data to the server.

[1214] 5. The server receives the text data and analyzes it using the NLP engine.

[1215] 6. The server uses the generative AI model to generate a text response: "Today it's sunny and the temperature is 20 degrees."

[1216] 7. Taking into account the emotional data, the server generates a response such as, "Are you feeling tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[1217] 8. The server sends the generated response text to the robot.

[1218] 9. The robot receives the response text and converts it into voice data using a speech synthesis engine.

[1219] 10. The robot asks the user, "Are you feeling tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[1220] In this way, the present invention helps seniors overcome feelings of loneliness through natural conversation and provides responses that take into account their emotional state. By working in tandem with the server and the robot, users can enjoy high-quality communication and emotional understanding.

[1221] The processing flow will be explained below.

[1222] Step 1:

[1223] The user speaks to the robot, "What's the weather like today?"

[1224] Step 2:

[1225] The robot uses a microphone to capture the user's voice.

[1226] Step 3:

[1227] The robot's voice recognition engine converts the captured voice data into text data such as "What's the weather like today?"

[1228] Step 4:

[1229] The robot's emotion engine analyzes the user's tone of voice and facial expressions to generate the emotion data "Tired."

[1230] Step 5:

[1231] The robot sends the converted text data and emotion data to a server via the Internet.

[1232] Step 6:

[1233] The server receives the text data and emotion data sent from the robot.

[1234] Step 7:

[1235] The server's Natural Language Processing (NLP) engine analyzes the received text data, "What's the weather like today?"

[1236] Step 8:

[1237] The server uses an NLP engine to understand that the user is looking for information about the weather.

[1238] Step 9:

[1239] The server's generative AI model generates the basic response text, "Today it's sunny and the temperature is 20 degrees."

[1240] Step 10:

[1241] The server takes into account the received emotion data "Tired" and generates a response text "Are you tired lately? It's sunny and 20 degrees today. It's a good day to relax."

[1242] Step 11:

[1243] The server sends the generated response text to the robot.

[1244] Step 12:

[1245] The robot receives the response text sent from the server.

[1246] Step 13:

[1247] The robot's speech synthesis engine converts the received text data, "Are you tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax." into speech data.

[1248] Step 14:

[1249] The robot provides the converted voice data to the user through a speaker.

[1250] Step 15:

[1251] The user hears a voice response from the robot: "Have you been feeling tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[1252] Example 2

[1253] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1254] Conventional voice assistant systems convert a user's voice into text data and generate responses, but they are unable to provide responses that take the user's emotional state into account, resulting in a problem of not being able to provide communication that fully satisfies the user. For this reason, there is a need for a system that can recognize the user's emotions and generate responses that correspond to those emotions. In addition, responses that are sensitive to the emotions of users who feel lonely, such as the elderly, are particularly important.

[1255] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1256] In this invention, the server includes means for storing a generative artificial intelligence model, means for adjusting responses based on emotional data, and means for analyzing text data using a natural language processing (NLP) engine, thereby generating responses based on the user's emotional state and enabling high-quality communication that gives the user a sense of satisfaction.

[1257] The "server means" is a device that holds a generative artificial intelligence model, analyzes and processes text data and emotional data, generates an appropriate response, and transmits it to the terminal means.

[1258] The "terminal means" is a device that receives voice input from the user, converts it into text data using a voice recognition engine, analyzes the user's emotions using an emotion engine, transmits the data to the server means, and converts the response received from the server means into voice using a voice synthesis engine to provide it to the user.

[1259] A "speech recognition engine" is software or hardware that converts voice input received from a user into text data.

[1260] A "speech synthesis engine" is software or hardware that converts text data into speech data.

[1261] An "emotion engine" is software or a system that analyzes a user's tone of voice and facial expressions, recognizes the user's emotional state, and generates emotion data.

[1262] A "Natural Language Processing (NLP) engine" is software or a system that analyzes natural language text data and understands grammar and meaning.

[1263] A "generative artificial intelligence model" is an algorithm or program that generates appropriate responses based on input data.

[1264] The present invention is a system that analyzes a user's voice input and generates a response according to the user's emotion. This system is composed of a server and a terminal. Specific embodiments of the system are described in detail below.

[1265] Server configuration

[1266] The server includes the following software and hardware:

[1267] 1. Generative AI model: A pre-trained generative AI model (e.g., GPT-3) is used to generate appropriate responses based on the text data sent by the user.

[1268] 2. Natural Language Processing (NLP) engine: Syntax and semantic analysis are performed using an engine for analyzing natural language text data (e.g., spaCy or NLTK).

[1269] 3. Emotional Data Processing Engine: Adjusts the generated response based on the emotional data sent from the terminal means.

[1270] 4. Communication means: Sends and receives data to and from the terminal means via HTTP requests or WebSockets.

[1271] Terminal means configuration

[1272] The terminal includes the following hardware and software:

[1273] 1. Speech recognition engine: An engine that converts voice data from the user into text data (e.g., Google Speech-to-Text API).

[1274] 2. Speech synthesis engine: An engine that converts text data received from a server into voice data (e.g., Microsoft Azure Text-to-Speech).

[1275] 3. Emotion engine: An engine that generates emotion data by analyzing the user's voice tone and facial expressions. It uses various phonological algorithms for voice analysis and facial recognition technology (e.g., OpenCV) for facial expression analysis.

[1276] 4. Communication method: Send and receive data with the server using HTTP requests and WebSockets.

[1277] 5. Microphone and speaker: Equipped with a microphone for receiving voice from the user and a speaker for outputting generated voice data.

[1278] System operation example

[1279] A specific example of operation is shown below.

[1280] Example 1: Conversation based on user sentiment

[1281] 1. The user asks the device, "What's the weather like today?"

[1282] 2. The terminal means receives this voice using its built-in microphone and converts it into text data such as "What's the weather like today?" using its voice recognition engine.

[1283] 3. The emotion engine of the terminal means analyzes the tone of the user's voice and generates emotion data such as "tired."

[1284] 4. The terminal means transmits the text data and emotion data to the server means.

[1285] 5. The server means receives the text data and emotion data, analyzes the text data using an NLP engine, and understands the meaning of "asking about the weather today."

[1286] 6. The server means uses the generative artificial intelligence model to generate a basic response text of "Today it's sunny and the temperature is 20 degrees."

[1287] 7. The server means considers the emotion data and generates a tailored response such as, "Are you feeling tired lately? It's sunny and 20 degrees today. It's a good day to relax."

[1288] 8. The server means transmits the generated response text to the terminal means.

[1289] 9. The terminal means receives the response text from the server means and generates a voice using a speech synthesis engine saying, "Are you tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[1290] 10. The terminal means provides the generated voice to the user through a speaker.

[1291] Examples of prompt statements

[1292] Example prompt 1:

[1293] "What is the robot's response when the user says, 'What's the weather like today?' in a tired tone?"

[1294] An example response generated:

[1295] "Have you been feeling tired lately? It's sunny and 20 degrees today. It's a good day to relax."

[1296] In this way, this system can provide high-quality communication by recognizing the user's emotions and generating appropriate responses based on those emotions.

[1297] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1298] Step 1:

[1299] Receiving audio input

[1300] The user speaks to the device to input voice. The device receives the user's voice using a built-in microphone. Voice data is obtained as input in this step.

[1301] Step 2:

[1302] Speech-to-text

[1303] The device analyzes the received voice data using a voice recognition engine (e.g., Google Speech-to-Text API) and converts it into text data. Specifically, it extracts features from the voice waveform and inputs them into a language model to obtain the corresponding text. The input is voice data, and the output is text data.

[1304] Step 3:

[1305] Emotion recognition

[1306] The device uses an emotion engine to generate emotion data from text data and the user's voice tone and facial expressions. Specifically, it recognizes emotions by analyzing non-verbal features of voice waveforms and visual features from image data. The input is text data and voice / image data, and the output is emotion data.

[1307] Step 4:

[1308] Sending text and emotion data

[1309] The device sends text data and emotion data to the server via HTTP POST requests or WebSockets. The input is the generated text data and emotion data, which are then sent to the server.

[1310] Step 5:

[1311] Receiving and analyzing data

[1312] The server receives the text data and emotion data sent from the device and analyzes the text data using an NLP engine (e.g., spaCy or NLTK). Specifically, it analyzes the text grammatically and extracts meaning. The input is text data, and the output is the semantic information of the analyzed text.

[1313] Step 6:

[1314] Generating a response

[1315] The server uses a generative artificial intelligence model (e.g., GPT-3) to generate a response based on the analyzed text and emotion data. Specifically, it inputs a prompt into the generative model to obtain an appropriate response text. The input is the semantic information and emotion data of the analyzed text, and the output is the response text.

[1316] Step 7:

[1317] Regulating responses based on emotions

[1318] The server adjusts the generated response text based on the emotion data. Specifically, it modifies the text by adding phrases that match the user's emotions or by adding kind words. The input is the generated response text and emotion data, and the output is the adjusted response text.

[1319] Step 8:

[1320] Sending response text

[1321] The server sends the adjusted response text to the terminal. The communication method is HTTP response or WebSocket. The input is the adjusted response text, which is then sent to the terminal.

[1322] Step 9:

[1323] Receiving reply text

[1324] The terminal receives the response text sent from the server via the communication means, and the input is the response text sent from the server.

[1325] Step 10:

[1326] Generate audio data

[1327] The device converts the received response text into voice data using a speech synthesis engine (e.g., Microsoft Azure Text-to-Speech). Specifically, it converts text data into a voice waveform and generates it. The input is the response text data, and the output is the generated voice data.

[1328] Step 11:

[1329] Audio output

[1330] The terminal provides the generated voice data to the user through a speaker. The input is the generated voice data, which is output to the user.

[1331] (Application example 2)

[1332] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1333] In conventional customer service, it has been difficult to properly recognize the user's emotional state and provide services based on that information. Furthermore, improving customer satisfaction has been a challenge because responses that are sensitive to the user's emotions are not provided. Therefore, there is a need for a system that can analyze the user's emotions and provide responses based on those emotions.

[1334] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion recognition engine and means for analyzing the user's emotional state, means for adjusting response text based on emotion data from the robot means, and means for analyzing text data using a natural language processing (NLP) engine and generating a response using a generative artificial intelligence model. This makes it possible to provide a response adapted to the user's emotion.

[1335] A "generative artificial intelligence model" is an AI system that learns from large amounts of data in advance and generates appropriate responses to input text.

[1336] "Server means" refers to a computer system that stores data, processes data, and performs network communication.

[1337] A "robot means" is a device that has the function of acquiring voice data, communicating with a server, and providing a generated response to a user.

[1338] "Voice data" is a digital representation of user-uttered voice information.

[1339] "Text data" refers to character information obtained by converting voice data into text.

[1340] A "speech recognition engine" is a software system that analyzes voice data and converts it into a corresponding string of characters.

[1341] A "speech synthesis engine" is a software system that converts text data into voice data and outputs it as voice.

[1342] An "emotion recognition engine" is a software system that analyzes a user's tone of voice and facial expressions to estimate their emotional state.

[1343] A "Natural Language Processing (NLP) engine" is a technology for analyzing natural language data and understanding its meaning and context.

[1344] "Customer service" is a general term for the product information, support, and assistance that stores and companies provide to users.

[1345] "Response text" is text information generated by the server means as a response to a user's input.

[1346] "Emotion data" is information that represents the emotional state of the user analyzed by the emotion recognition engine.

[1347] The present invention provides a system including server means for storing a generative artificial intelligence model, robot means for converting voice data into text data and sending it to the server means, robot means for converting a response from the server means into voice and providing it to the user, and an emotion recognition engine for analyzing the emotional state of the user. This system enables the provision of a response based on the emotional state of the user in customer service.

[1348] System configuration:

[1349] Specific configurations for implementing the present invention will be described below.

[1350] Server means:

[1351] 1. Generative AI Model:

[1352] The server means stores a generative artificial intelligence model (e.g., GPT-3) that has been trained in advance from large-scale data, and generates a response based on the input text.

[1353] 2. Natural Language Processing (NLP) Engine:

[1354] The server means uses a natural language processing (NLP) engine (e.g., BERT) to analyze the transmitted text data and understand its content.

[1355] 3. Response Adjustment:

[1356] The server means adjusts the generated response text based on the emotion data transmitted by the emotion recognition engine to generate an appropriate response text.

[1357] Robotic Means:

[1358] 1. Speech Recognition Engine:

[1359] The robotic means is equipped with a microphone for receiving voice input from the user and converts the voice into text data using a speech recognition engine (e.g., Google Speech-to-Text API).

[1360] 2. Emotion Recognition Engine:

[1361] The robotic means includes an emotion recognition engine (e.g., Microsoft Azure Emotion API) that analyzes the user's tone of voice and facial expressions to generate emotion data.

[1362] 3. Speech synthesis engine:

[1363] The robot means receives the response text sent from the server means, converts it into voice data using a voice synthesis engine (e.g., Google Text-to-Speech API), and provides it to the user through a speaker.

[1364] Example of how it works:

[1365] 1. User Input:

[1366] Consider a situation where a user speaks to a robot. For example, suppose the user says, "What's your recommendation for today?"

[1367] 2. Speech Recognition:

[1368] The robot means receives the user's voice through a microphone and converts it into text data such as "What's your recommendation today?" using a voice recognition engine.

[1369] 3. Emotion analysis:

[1370] The emotion recognition engine analyzes the user's tone of voice and facial expressions to generate emotion data such as "gratitude" or "interest."

[1371] 4. Data transmission:

[1372] The robot means transmits the generated text data and emotion data to the server means.

[1373] 5. Response Generation:

[1374] The server analyzes the text data using an NLP engine and generates an appropriate response text using a generative artificial intelligence model, adjusting the response by taking into account the emotional data, and generating a response such as "We have a special product for you today. We think you'll be interested."

[1375] 6. Speech synthesis:

[1376] The robot means receives the response text sent from the server means, converts it into voice data using a voice synthesis engine, and provides it to the user through a speaker.

[1377] Examples:

[1378] An example of a prompt is:

[1379] "User sentiment is interesting. Based on recent user sentiment, respond to the following question: What's your recommendation today?"

[1380] In this way, the system of the present invention can improve the quality of customer service by understanding the user's emotions and providing responses based on them.

[1381] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1382] Step 1:

[1383] The user speaks to the robot. Specifically, the user says, "What's your recommendation for today?" The user's voice input is received by the robot's microphone.

[1384] Step 2:

[1385] The robot's speech recognition engine (e.g., Google Speech-to-Text API) converts the voice data into text data. It receives the voice data as input, performs data analysis, and generates the text data "What's your recommendation today?" as output.

[1386] Step 3:

[1387] The robot's emotion recognition engine (e.g., Microsoft Azure Emotion API) analyzes the user's voice tone and facial expressions to generate emotional data. It receives voice tone and facial expression data as input, analyzes the data, and generates emotional data such as "interest" or "gratitude" as output.

[1388] Step 4:

[1389] The robot means transmits the text data and emotion data generated in the previous step to the server means, which receives the text data and emotion data as input and transmits them to the server using the communication means.

[1390] Step 5:

[1391] The server means receives the transmitted text data and emotion data, receives the data transmitted from the robot as input, and stores it as internal data.

[1392] Step 6:

[1393] The server's Natural Language Processing (NLP) engine (e.g., BERT) analyzes and understands the text data. It receives the text data as input, performs data analysis, and generates the processed results as output.

[1394] Step 7:

[1395] The server's artificial intelligence model (e.g., GPT-3) generates an appropriate response text based on the processing results of the NLP engine and the emotional data. It receives the analyzed text data and emotional data as input, performs data generation, and generates the response text as output. For example, it generates the response text, "Today we have a special product we highly recommend. We think you'll be interested."

[1396] Step 8:

[1397] The server means transmits the generated response text to the robot means, which receives the generated response text as input and transmits it to the robot using the communication means.

[1398] Step 9:

[1399] The robot means receives the response text from the server, and receives the response text sent from the server as input and stores it as internal data.

[1400] Step 10:

[1401] The robot's speech synthesis engine (e.g., Google Text-to-Speech API) converts the response text into voice data. It receives the response text as input, performs data conversion, and generates voice data as output.

[1402] Step 11:

[1403] The robot means provides the generated voice data to the user through the speaker. The robot means receives the voice data as input and outputs it through the speaker. The user can receive a voice response such as, "Today we have a special product we highly recommend. I'm sure you'll be interested."

[1404] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1405] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1406] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1407] [Fourth embodiment]

[1408] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1409] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1410] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1411] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1412] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1413] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1414] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1415] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1416] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1417] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1418] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1419] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1420] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1421] The present invention is a system that includes a server that stores a generative AI model, a robot that converts voice data into text data and sends it to the server, and a robot that receives a response from the server, converts it into voice, and provides it to the user. Specific embodiments for implementing this system are described in detail below.

[1422] System Configuration

[1423] 1. Server Configuration

[1424] The server contains a generative AI model and a natural language processing (NLP) engine. The server performs the following functions:

[1425] When the voice data is transmitted as text data, the text content is analyzed.

[1426] Uses a Natural Language Processing (NLP) engine to understand the content of the text data sent to it.

[1427] An appropriate response is generated using a generative AI model, and the response is sent to the robot as text data.

[1428] 2. Robot Configuration

[1429] The robot performs the following functions:

[1430] It has a microphone for receiving voice input from the user.

[1431] A speech recognition engine is used to convert the user's speech into text data.

[1432] It has a communication means for transmitting the converted voice as text data to the server.

[1433] It has a communication means for receiving responses from the server.

[1434] A speech synthesis engine is used to convert the received text response into speech and provide it to the user through the speaker.

[1435] Program Processing

[1436] 1. Server-side processing

[1437] The server receives the text data sent by the robot and analyzes it, specifically as follows:

[1438] The server receives text data converted by a voice recognition engine from the robot.

[1439] The received text data is analyzed using a natural language processing (NLP) engine to understand its content.

[1440] Generate appropriate response text based on a generative AI model.

[1441] The generated response text is sent to the robot.

[1442] 2. Processing on the robot side

[1443] The robot converts the user's voice input into text data and sends it to the server. Specifically, it does the following:

[1444] The user speaks to the robot, which receives the voice through a microphone.

[1445] The robot's voice recognition engine converts the voice into text data.

[1446] The converted text data is sent to the server.

[1447] Receive a response from the server.

[1448] The received response text is converted into voice data by a speech synthesis engine and provided to the user through a speaker.

[1449] Specific examples

[1450] Example 1: A conversation about the weather

[1451] Consider a scenario in which a user asks a robot, "What's the weather like today?"

[1452] 1. The user says, "What's the weather like today?"

[1453] 2. The robot converts this speech into text data: "What's the weather like today?"

[1454] 3. The robot sends the text data to the server.

[1455] 4. The server receives the text data and analyzes it using the NLP engine.

[1456] 5. The server uses the generative AI model to generate a text response: "Today it's sunny and the temperature is 20 degrees."

[1457] 6. The server generates a text response and sends it to the robot.

[1458] 7. The robot converts the received text response, "Today is sunny and the temperature is 20 degrees," into voice data.

[1459] 8. The robot tells the user, "Today is sunny and the temperature is 20 degrees."

[1460] In this way, the present invention provides an effective means for seniors to overcome feelings of loneliness through natural conversation. By having the server and the robot work together, users can enjoy high-quality communication.

[1461] The processing flow will be explained below.

[1462] Step 1:

[1463] The user speaks to the robot, "What's the weather like today?"

[1464] Step 2:

[1465] The robot uses a microphone to capture the user's voice.

[1466] Step 3:

[1467] The robot's voice recognition engine converts the captured voice data into text data such as "What's the weather like today?"

[1468] Step 4:

[1469] The robot sends the converted text data to a server via the Internet.

[1470] Step 5:

[1471] The server receives the text data sent from the robot.

[1472] Step 6:

[1473] The server's Natural Language Processing (NLP) engine analyzes the received text data.

[1474] Step 7:

[1475] The server uses an NLP engine to understand that the user is looking for information about the weather.

[1476] Step 8:

[1477] The server uses a generative AI model to generate a response text such as, "Today it's sunny and the temperature is 20 degrees."

[1478] Step 9:

[1479] The server sends the generated response text to the robot.

[1480] Step 10:

[1481] The robot receives the response text sent from the server.

[1482] Step 11:

[1483] The robot's speech synthesis engine converts the received text data, "Today is sunny and the temperature is 20 degrees," into speech data.

[1484] Step 12:

[1485] The robot provides the converted voice data to the user through a speaker.

[1486] Step 13:

[1487] The user hears a voice response from the robot: "Today is sunny and the temperature is 20 degrees."

[1488] Example 1

[1489] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1490] Conventional speech recognition and response generation systems lack the processing power and response quality required to realize natural conversations with users, resulting in the inability to fully alleviate users' feelings of loneliness. This has been a particular problem for seniors and users living alone, as it has been difficult to provide sufficient psychological support through conversation.

[1491] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1492] In this invention, the server includes means for storing a generative artificial intelligence model, robot means for converting voice data into text data and transmitting it, robot means for receiving a response, converting it into voice and providing it to the user, and means for analyzing the text data using a natural language processing engine and transmitting the generated response to the robot means. This enables natural conversation with the user and provides high-quality responses, thereby alleviating the user's sense of loneliness and providing psychological support.

[1493] A "generative artificial intelligence model" is an artificial intelligence algorithm for generating appropriate responses based on input data.

[1494] A "server means" is a computing device that analyzes data and generates responses.

[1495] A "robot means" is a device that acts as an interface with a user and receives, converts, transmits, and plays back audio.

[1496] A "voice recognition engine" is software for converting voice data into text data.

[1497] A "communication module" is a device or software for sending and receiving data.

[1498] A "speech synthesis engine" is software for converting text data into voice data.

[1499] A "natural language processing engine" is software for analyzing text data and understanding its content.

[1500] "Text data" is data expressed as character information.

[1501] "Audio data" is data that represents an audio signal in digital form.

[1502] "User" refers to a person who uses this system.

[1503] A "prompt sentence" is an input sentence that generates an appropriate response based on a generative artificial intelligence model.

[1504] "Response" refers to the system's response to a user's input.

[1505] This invention relates to a speech recognition and response generation system based on a generative AI model. This system has a series of functions: converting speech data into text data, analyzing the text data to generate an appropriate response, and then converting it back into speech data to provide to the user. The following describes in detail how this system is specifically implemented.

[1506] System Configuration

[1507] 1. Server configuration

[1508] The server contains the following main components:

[1509] Generative AI models: Includes models for generating appropriate responses based on input text data.

[1510] Natural Language Processing (NLP) engine: Includes an engine for analyzing text data and understanding its content.

[1511] The server receives the text data sent from the robot, analyzes it, generates a response using a generative artificial intelligence model, and sends the response back to the robot as text data.

[1512] 2. Robot Configuration

[1513] The robot includes the following main components:

[1514] Microphone: Hardware for receiving audio input from the user.

[1515] Speech recognition engine: Software for converting received voice data into text data (e.g., Google Cloud Speech-to-Text).

[1516] Communication module: A module for sending converted text data to the server and receiving responses from the server.

[1517] Speech synthesis engine: Software (e.g., Amazon Polly) that converts received text responses into speech.

[1518] Speaker: Hardware that outputs converted audio data to the user.

[1519] Specific system processing examples

[1520] As a specific example of system processing, a scenario of a conversation about the weather is shown below.

[1521] Prompt Sentence Examples

[1522] When a user asks the robot, "What's the weather like today?" the prompt is:

[1523] "What's the weather like today?"

[1524] Weather conversation process flow

[1525] 1. Voice input: The user speaks to the robot, "What's the weather like today?"

[1526] 2. Conversion of voice data: The robot receives voice using a microphone and converts it into text data, such as "What's the weather like today?", using a voice recognition engine.

[1527] 3. Sending text data: The robot sends the converted text data to the server via the communication module.

[1528] 4. Analyzing text data: The server receives the text data and analyzes it using the NLP engine.

[1529] 5. Generate a response: The server uses the generative AI model to generate a text response: "Today it's sunny and the temperature is 20 degrees."

[1530] 6. Send Response: The server generates a text response and sends it to the robot.

[1531] 7. Generate voice data: The text response received by the robot is converted into voice data using a voice synthesis engine.

[1532] 8. Voice response: The robot speaks to the user through the speaker, "Today is sunny and the temperature is 20 degrees."

[1533] In this way, the present invention is a system that can realize natural conversations with users and provide high-quality responses. By combining a generative AI model and an NLP engine, it is possible to perform highly accurate text analysis and response generation, thereby alleviating users' feelings of loneliness.

[1534] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1535] Step 1:

[1536] The user speaks to the robot. For example, the user says, "What's the weather like today?"

[1537] Input: User's voice data

[1538] Specific operation: The user inputs voice by speaking to the robot.

[1539] Step 2:

[1540] The robot receives the user's voice using a microphone and converts it into text data using a voice recognition engine.

[1541] Input: User's voice data

[1542] Data processing: The speech recognition engine analyzes the voice signal and converts the voice data into text data.

[1543] Output: Text data "What's the weather like today?"

[1544] Specific operation: The robot's microphone receives the user's voice and the voice recognition engine converts it into text data.

[1545] Step 3:

[1546] The robot transmits the converted text data to the server via the communication module.

[1547] Input: Text data "What's the weather like today?"

[1548] Data processing: Converts text data into a format for sending to the server.

[1549] Output: Text data sent to the server

[1550] Specific operation: The robot's communication module sends text data to the server.

[1551] Step 4:

[1552] The server receives the text data and analyzes it using a natural language processing (NLP) engine.

[1553] Input: Text data sent to the server: "What's the weather like today?"

[1554] Data processing: An NLP engine analyzes the text to understand its intent and content.

[1555] Output: Analysis results (user question)

[1556] Specific operation: The server analyzes the text data received using an NLP engine and understands that the user's question is "about the weather."

[1557] Step 5:

[1558] The server uses the generative AI model to generate an appropriate response based on the analysis results.

[1559] Input: Analysis results (user question)

[1560] Data processing: The generative AI model generates an appropriate response based on the prompt.

[1561] Output: Response text "Today is sunny and the temperature is 20 degrees."

[1562] Specific behavior: The server's generative AI model generates the appropriate response: "Today is sunny and the temperature is 20 degrees."

[1563] Step 6:

[1564] The server generates a response text and sends it to the robot.

[1565] Input: Response text "It's sunny today and the temperature is 20 degrees."

[1566] Data processing: Converting response text into a format for sending to the robot.

[1567] Output: Response text sent to the robot

[1568] Specific operation: The text data generated by the server is sent to the robot via the communication module.

[1569] Step 7:

[1570] The robot converts the received response text into voice data using a speech synthesis engine.

[1571] Input: Response text "It's sunny today and the temperature is 20 degrees."

[1572] Data processing: The speech synthesis engine converts text data into speech data.

[1573] Output: Audio data

[1574] Specific operation: The robot's speech synthesis engine analyzes text data and converts it into voice data.

[1575] Step 8:

[1576] The robot provides voice data to the user through a speaker.

[1577] Input: Audio data

[1578] Output: A spoken response to the user: "It's sunny today and the temperature is 20 degrees."

[1579] Specific behavior: The robot's speaker outputs audio data to the user, providing a response.

[1580] In this way, each step works in cooperation to realize natural conversation with the user.

[1581] (Application example 1)

[1582] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1583] Conventional food delivery services require users to perform many manual operations when placing an order, which places a significant burden on users, especially elderly people and those unfamiliar with technology. Furthermore, there is a lack of systems that allow users to intuitively select from a wide variety of menus, hindering user convenience. Given this background, a new interactive ordering assistant is needed to improve user convenience.

[1584] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1585] In this invention, the server includes a means for storing a generative artificial intelligence model, an information processing device for converting voice data into text data and transmitting it, and an information processing device for receiving a response from the server, converting it into voice, and providing it to the user. This allows the user to complete a food delivery order while having a natural conversation through voice input. In addition, by using a natural language processing (NLP) engine and a voice synthesis engine, the user can proceed with the ordering process intuitively and quickly.

[1586] A "generative artificial intelligence model" is an algorithm that learns from huge data sets and performs natural language processing and data generation.

[1587] A "server means" is a computer system for processing data and communicating with other devices and resources.

[1588] "Information processing device means" refers to a device for converting voice data into text data and analyzing and processing the text data.

[1589] A "voice recognition engine" is a software system for converting input voice data into text data.

[1590] A "speech synthesis engine" is a software system for converting text data into speech data.

[1591] A "Natural Language Processing (NLP) engine" is a software system that understands and analyzes human language and generates appropriate responses.

[1592] "Food delivery service" refers to a service that allows users to order food online and have it delivered.

[1593] An "interactive order assistant" is a computer system that assists a user in the ordering process through natural conversation.

[1594] The system for realizing the "interactive order assistant" of the present invention uses the following hardware and software: An information processing device such as a smartphone acts as an interface with the user, and a server performs back-end processing.

[1595] Hardware and software used

[1596] Smartphone (information processing device)

[1597] microphone

[1598] speaker

[1599] server

[1600] Generative AI models (e.g., OpenAI GPT-3 / GPT-4)

[1601] Natural Language Processing (NLP) engines (e.g., Dialogflow)

[1602] Speech recognition engine (e.g. Google Cloud Speech-to-Text API)

[1603] Speech synthesis engine (e.g. Amazon Polly)

[1604] System Operation Overview

[1605] Smartphone side

[1606] When a user orders by voice, the smartphone's microphone captures the voice in real time. The captured voice is converted into text data using a speech recognition engine. This text data is then sent to the server.

[1607] Server side

[1608] The server receives the text data converted by the speech recognition engine. Using a natural language processing (NLP) engine, the server analyzes this text data and understands the user's intent. Based on the analyzed data, the server uses a generative AI model to generate an appropriate response text. This response text is then sent back to the smartphone.

[1609] Smartphone side

[1610] The smartphone receives the response text sent from the server. The received text is converted into voice data by a speech synthesis engine and provided to the user as voice through the smartphone speaker. Through this series of operations, the user can complete the food delivery order interactively.

[1611] Specific examples

[1612] The system is started when a user speaks into their smartphone saying, "I'd like to order a pizza." This speech is converted into text, and the text data "I'd like to order a pizza" is sent to the server. The server analyzes this text data and uses a generative AI model to generate an appropriate response. For example, it generates a response text such as "What kind of pizza would you like?" and sends it to the smartphone. The received text is converted into speech, and the user is asked aloud, "What kind of pizza would you like?" This allows the user to continue ordering by voice.

[1613] Prompt Sentence Examples

[1614] "When a user turns to a food delivery assistant and says, 'I want to order a pizza,' how do we respond?"

[1615] The prompts allow the generative AI model to generate appropriate responses to achieve natural interactions with the user, thus providing an intuitive and efficient food delivery ordering process for users.

[1616] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1617] Step 1:

[1618] A user speaks to a smartphone saying, "I'd like to order a pizza." The smartphone's microphone captures this speech in real time. The input is the user's voice data, and the output is an audio file for processing by the speech recognition engine.

[1619] Step 2:

[1620] The smartphone's speech recognition engine (e.g., Google Cloud Speech-to-Text API) converts the captured voice data into text data. Specifically, it samples the voice data, performs phonemic analysis, and generates the corresponding text data. The input is the captured voice data, and the output is the text data "I would like to order a pizza."

[1621] Step 3:

[1622] The smartphone sends the converted text data to the server. Specifically, it sends the text data to the server via an HTTP request. The input is the text data, and the output is an HTTP request to the server.

[1623] Step 4:

[1624] The server receives the text data sent from the smartphone. The input is the received text data, and the output is to pass the text data to the Natural Language Processing (NLP) system.

[1625] Step 5:

[1626] The server analyzes the text data using a natural language processing (NLP) engine (e.g., Dialogflow). Specifically, the NLP engine performs grammatical and semantic analysis of the text data to understand the user's intent. The input is the received text data, and the output is the analysis result.

[1627] Step 6:

[1628] The server uses a generative AI model (e.g., OpenAI GPT-3 / GPT-4) based on the analysis results to generate an appropriate response text. Specifically, the generative AI model receives the analysis results as a prompt and generates a natural response to the user. The input is the analysis results, and the output is a response text such as "What kind of pizza would you like?"

[1629] Step 7:

[1630] The server sends the generated response text to the smartphone. The input is the generated response text, and the output is the HTTP response to the smartphone.

[1631] Step 8:

[1632] The smartphone receives the response text sent from the server. The input is the received response text, and the output is text data for processing by the speech synthesis engine.

[1633] Step 9:

[1634] The smartphone's speech synthesis engine (e.g., Amazon Polly) converts the received text data into speech data. Specifically, it converts the text data into speech waveforms and generates an audio file. The input is the received text data, and the output is speech data.

[1635] Step 10:

[1636] Voice data is provided to the user through the smartphone speaker. Specifically, voice output is achieved by sending voice data to the speaker. The input is the generated voice data, and the output is the voice saying, "What kind of pizza would you like?"

[1637] The above processing steps enable the user to interactively place a food delivery order.

[1638] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1639] The present invention is a system including a server that stores a generative AI model, a robot that converts voice data into text data and sends it to the server, a robot that receives a response from the server, converts it into voice, and provides it to the user, and an emotion engine that recognizes the user's emotions. Specific embodiments for implementing this system are described in detail below.

[1640] System Configuration

[1641] 1. Server Configuration

[1642] The server contains a generative AI model and a natural language processing (NLP) engine. The server performs the following functions:

[1643] When the voice data is transmitted as text data, the text content is analyzed.

[1644] Uses a Natural Language Processing (NLP) engine to understand the content of the text data sent to it.

[1645] An appropriate response is generated using a generative AI model, and the response is sent to the robot as text data.

[1646] Adjust responses as needed based on emotion engine analysis.

[1647] 2. Robot Configuration

[1648] The robot performs the following functions:

[1649] It includes a microphone for receiving voice input from the user.

[1650] A speech recognition engine is used to convert the user's speech into text data.

[1651] The device includes a communication means for transmitting the converted voice data to the server as text data.

[1652] A communication means is provided for receiving a response from the server.

[1653] A speech synthesis engine is used to convert the received text response into speech and provide it to the user through the speaker.

[1654] An emotion engine is used to recognize the user's emotions from their voice tone and facial expressions, and the information is sent to the server.

[1655] Program Processing

[1656] 1. Server-side processing

[1657] The server receives the text data and emotion data sent by the robot and analyzes them. Specifically, it does the following:

[1658] The server receives from the robot the text data converted by the speech recognition engine and the emotion data analyzed by the emotion engine.

[1659] The received data is analyzed using a natural language processing (NLP) engine to understand its content.

[1660] Based on the generative AI model, an appropriate response text is generated based on the text content and emotional data.

[1661] The generated response text is sent to the robot.

[1662] 2. Processing on the robot side

[1663] The robot converts the user's voice input into text data, and also recognizes the user's emotions using its emotion engine and sends them to the server. Specifically, it works as follows:

[1664] The user speaks to the robot, which receives the voice through a microphone.

[1665] The robot's voice recognition engine converts the voice into text data.

[1666] The robot's emotion engine analyzes the user's tone of voice and facial expressions to generate emotion data.

[1667] The text data and the emotion data are transmitted to a server.

[1668] Receives the response text from the server.

[1669] The received response text is converted into voice data by a speech synthesis engine and provided to the user through a speaker.

[1670] Specific examples

[1671] Example 1: Conversation based on user sentiment

[1672] Consider a scenario where a user asks a robot, "What's the weather like today?", but the user's voice sounds tired.

[1673] 1. The user says, "What's the weather like today?"

[1674] 2. The robot converts this speech into text data: "What's the weather like today?"

[1675] 3. The robot's emotion engine analyzes the tone of the user's voice and generates emotion data such as "tired."

[1676] 4. The robot sends the text data and emotion data to the server.

[1677] 5. The server receives the text data and analyzes it using the NLP engine.

[1678] 6. The server uses the generative AI model to generate a text response: "Today it's sunny and the temperature is 20 degrees."

[1679] 7. Taking into account the emotional data, the server generates a response such as, "Are you feeling tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[1680] 8. The server sends the generated response text to the robot.

[1681] 9. The robot receives the response text and converts it into voice data using a speech synthesis engine.

[1682] 10. The robot asks the user, "Are you feeling tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[1683] In this way, the present invention helps seniors overcome feelings of loneliness through natural conversation and provides responses that take into account their emotional state. By working in tandem with the server and the robot, users can enjoy high-quality communication and emotional understanding.

[1684] The processing flow will be explained below.

[1685] Step 1:

[1686] The user speaks to the robot, "What's the weather like today?"

[1687] Step 2:

[1688] The robot uses a microphone to capture the user's voice.

[1689] Step 3:

[1690] The robot's voice recognition engine converts the captured voice data into text data such as "What's the weather like today?"

[1691] Step 4:

[1692] The robot's emotion engine analyzes the user's tone of voice and facial expressions to generate the emotion data "Tired."

[1693] Step 5:

[1694] The robot sends the converted text data and emotion data to a server via the Internet.

[1695] Step 6:

[1696] The server receives the text data and emotion data sent from the robot.

[1697] Step 7:

[1698] The server's Natural Language Processing (NLP) engine analyzes the received text data, "What's the weather like today?"

[1699] Step 8:

[1700] The server uses an NLP engine to understand that the user is looking for information about the weather.

[1701] Step 9:

[1702] The server's generative AI model generates the basic response text, "Today it's sunny and the temperature is 20 degrees."

[1703] Step 10:

[1704] The server takes into account the received emotion data "Tired" and generates a response text "Are you tired lately? It's sunny and 20 degrees today. It's a good day to relax."

[1705] Step 11:

[1706] The server sends the generated response text to the robot.

[1707] Step 12:

[1708] The robot receives the response text sent from the server.

[1709] Step 13:

[1710] The robot's speech synthesis engine converts the received text data, "Are you tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax." into speech data.

[1711] Step 14:

[1712] The robot provides the converted voice data to the user through a speaker.

[1713] Step 15:

[1714] The user hears a voice response from the robot: "Have you been feeling tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[1715] Example 2

[1716] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1717] Conventional voice assistant systems convert a user's voice into text data and generate responses, but they are unable to provide responses that take the user's emotional state into account, resulting in a problem of not being able to provide communication that fully satisfies the user. For this reason, there is a need for a system that can recognize the user's emotions and generate responses that correspond to those emotions. In addition, responses that are sensitive to the emotions of users who feel lonely, such as the elderly, are particularly important.

[1718] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1719] In this invention, the server includes means for storing a generative artificial intelligence model, means for adjusting responses based on emotional data, and means for analyzing text data using a natural language processing (NLP) engine, thereby generating responses based on the user's emotional state and enabling high-quality communication that gives the user a sense of satisfaction.

[1720] The "server means" is a device that holds a generative artificial intelligence model, analyzes and processes text data and emotional data, generates an appropriate response, and transmits it to the terminal means.

[1721] The "terminal means" is a device that receives voice input from the user, converts it into text data using a voice recognition engine, analyzes the user's emotions using an emotion engine, transmits the data to the server means, and converts the response received from the server means into voice using a voice synthesis engine to provide it to the user.

[1722] A "speech recognition engine" is software or hardware that converts voice input received from a user into text data.

[1723] A "speech synthesis engine" is software or hardware that converts text data into speech data.

[1724] An "emotion engine" is software or a system that analyzes a user's tone of voice and facial expressions, recognizes the user's emotional state, and generates emotion data.

[1725] A "Natural Language Processing (NLP) engine" is software or a system that analyzes natural language text data and understands grammar and meaning.

[1726] A "generative artificial intelligence model" is an algorithm or program that generates appropriate responses based on input data.

[1727] The present invention is a system that analyzes a user's voice input and generates a response according to the user's emotion. This system is composed of a server and a terminal. Specific embodiments of the system are described in detail below.

[1728] Server configuration

[1729] The server includes the following software and hardware:

[1730] 1. Generative AI model: A pre-trained generative AI model (e.g., GPT-3) is used to generate appropriate responses based on the text data sent by the user.

[1731] 2. Natural Language Processing (NLP) engine: Syntax and semantic analysis are performed using an engine for analyzing natural language text data (e.g., spaCy or NLTK).

[1732] 3. Emotional Data Processing Engine: Adjusts the generated response based on the emotional data sent from the terminal means.

[1733] 4. Communication means: Sends and receives data to and from the terminal means via HTTP requests or WebSockets.

[1734] Terminal means configuration

[1735] The terminal includes the following hardware and software:

[1736] 1. Speech recognition engine: An engine that converts voice data from the user into text data (e.g., Google Speech-to-Text API).

[1737] 2. Speech synthesis engine: An engine that converts text data received from a server into voice data (e.g., Microsoft Azure Text-to-Speech).

[1738] 3. Emotion engine: An engine that generates emotion data by analyzing the user's voice tone and facial expressions. It uses various phonological algorithms for voice analysis and facial recognition technology (e.g., OpenCV) for facial expression analysis.

[1739] 4. Communication method: Send and receive data with the server using HTTP requests and WebSockets.

[1740] 5. Microphone and speaker: Equipped with a microphone for receiving voice from the user and a speaker for outputting generated voice data.

[1741] System operation example

[1742] A specific example of operation is shown below.

[1743] Example 1: Conversation based on user sentiment

[1744] 1. The user asks the device, "What's the weather like today?"

[1745] 2. The terminal means receives this voice using its built-in microphone and converts it into text data such as "What's the weather like today?" using its voice recognition engine.

[1746] 3. The emotion engine of the terminal means analyzes the tone of the user's voice and generates emotion data such as "tired."

[1747] 4. The terminal means transmits the text data and emotion data to the server means.

[1748] 5. The server means receives the text data and emotion data, analyzes the text data using an NLP engine, and understands the meaning of "asking about the weather today."

[1749] 6. The server means uses the generative artificial intelligence model to generate a basic response text of "Today it's sunny and the temperature is 20 degrees."

[1750] 7. The server means considers the emotion data and generates a tailored response such as, "Are you feeling tired lately? It's sunny and 20 degrees today. It's a good day to relax."

[1751] 8. The server means transmits the generated response text to the terminal means.

[1752] 9. The terminal means receives the response text from the server means and generates a voice using a speech synthesis engine saying, "Are you tired lately? It's sunny today and the temperature is 20 degrees. It's a good day to relax."

[1753] 10. The terminal means provides the generated voice to the user through a speaker.

[1754] Examples of prompt statements

[1755] Example prompt 1:

[1756] "What is the robot's response when the user says, 'What's the weather like today?' in a tired tone?"

[1757] An example response generated:

[1758] "Have you been feeling tired lately? It's sunny and 20 degrees today. It's a good day to relax."

[1759] In this way, this system can provide high-quality communication by recognizing the user's emotions and generating appropriate responses based on those emotions.

[1760] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1761] Step 1:

[1762] Receiving audio input

[1763] The user speaks to the device to input voice. The device receives the user's voice using a built-in microphone. Voice data is obtained as input in this step.

[1764] Step 2:

[1765] Speech-to-text

[1766] The device analyzes the received voice data using a voice recognition engine (e.g., Google Speech-to-Text API) and converts it into text data. Specifically, it extracts features from the voice waveform and inputs them into a language model to obtain the corresponding text. The input is voice data, and the output is text data.

[1767] Step 3:

[1768] Emotion recognition

[1769] The device uses an emotion engine to generate emotion data from text data and the user's voice tone and facial expressions. Specifically, it recognizes emotions by analyzing non-verbal features of voice waveforms and visual features from image data. The input is text data and voice / image data, and the output is emotion data.

[1770] Step 4:

[1771] Sending text and emotion data

[1772] The device sends text data and emotion data to the server via HTTP POST requests or WebSockets. The input is the generated text data and emotion data, which are then sent to the server.

[1773] Step 5:

[1774] Receiving and analyzing data

[1775] The server receives the text data and emotion data sent from the device and analyzes the text data using an NLP engine (e.g., spaCy or NLTK). Specifically, it analyzes the text grammatically and extracts meaning. The input is text data, and the output is the semantic information of the analyzed text.

[1776] Step 6:

[1777] Generating a response

[1778] The server uses a generative artificial intelligence model (e.g., GPT-3) to generate a response based on the analyzed text and emotion data. Specifically, it inputs a prompt into the generative model to obtain an appropriate response text. The input is the semantic information and emotion data of the analyzed text, and the output is the response text.

[1779] Step 7:

[1780] Regulating responses based on emotions

[1781] The server adjusts the generated response text based on the emotion data. Specifically, it modifies the text by adding phrases that match the user's emotions or by adding kind words. The input is the generated response text and emotion data, and the output is the adjusted response text.

[1782] Step 8:

[1783] Sending response text

[1784] The server sends the adjusted response text to the terminal. The communication method is HTTP response or WebSocket. The input is the adjusted response text, which is then sent to the terminal.

[1785] Step 9:

[1786] Receiving reply text

[1787] The terminal receives the response text sent from the server via the communication means, and the input is the response text sent from the server.

[1788] Step 10:

[1789] Generate audio data

[1790] The device converts the received response text into voice data using a speech synthesis engine (e.g., Microsoft Azure Text-to-Speech). Specifically, it converts text data into a voice waveform and generates it. The input is the response text data, and the output is the generated voice data.

[1791] Step 11:

[1792] Audio output

[1793] The terminal provides the generated voice data to the user through a speaker. The input is the generated voice data, which is output to the user.

[1794] (Application example 2)

[1795] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1796] In conventional customer service, it has been difficult to properly recognize the user's emotional state and provide services based on that information. Furthermore, improving customer satisfaction has been a challenge because responses that are sensitive to the user's emotions are not provided. Therefore, there is a need for a system that can analyze the user's emotions and provide responses based on those emotions.

[1797] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an emotion recognition engine and means for analyzing the user's emotional state, means for adjusting response text based on emotion data from the robot means, and means for analyzing text data using a natural language processing (NLP) engine and generating a response using a generative artificial intelligence model. This makes it possible to provide a response adapted to the user's emotion.

[1798] A "generative artificial intelligence model" is an AI system that learns from large amounts of data in advance and generates appropriate responses to input text.

[1799] "Server means" refers to a computer system that stores data, processes data, and performs network communication.

[1800] A "robot means" is a device that has the function of acquiring voice data, communicating with a server, and providing a generated response to a user.

[1801] "Voice data" is a digital representation of user-uttered voice information.

[1802] "Text data" refers to character information obtained by converting voice data into text.

[1803] A "speech recognition engine" is a software system that analyzes voice data and converts it into a corresponding string of characters.

[1804] A "speech synthesis engine" is a software system that converts text data into voice data and outputs it as voice.

[1805] An "emotion recognition engine" is a software system that analyzes a user's tone of voice and facial expressions to estimate their emotional state.

[1806] A "Natural Language Processing (NLP) engine" is a technology for analyzing natural language data and understanding its meaning and context.

[1807] "Customer service" is a general term for the product information, support, and assistance that stores and companies provide to users.

[1808] "Response text" is text information generated by the server means as a response to a user's input.

[1809] "Emotion data" is information that represents the emotional state of the user analyzed by the emotion recognition engine.

[1810] The present invention provides a system including server means for storing a generative artificial intelligence model, robot means for converting voice data into text data and sending it to the server means, robot means for converting a response from the server means into voice and providing it to the user, and an emotion recognition engine for analyzing the emotional state of the user. This system enables the provision of a response based on the emotional state of the user in customer service.

[1811] System configuration:

[1812] Specific configurations for implementing the present invention will be described below.

[1813] Server means:

[1814] 1. Generative AI Model:

[1815] The server means stores a generative artificial intelligence model (e.g., GPT-3) that has been trained in advance from large-scale data, and generates a response based on the input text.

[1816] 2. Natural Language Processing (NLP) Engine:

[1817] The server means uses a natural language processing (NLP) engine (e.g., BERT) to analyze the transmitted text data and understand its content.

[1818] 3. Response Adjustment:

[1819] The server means adjusts the generated response text based on the emotion data transmitted by the emotion recognition engine to generate an appropriate response text.

[1820] Robotic Means:

[1821] 1. Speech Recognition Engine:

[1822] The robotic means is equipped with a microphone for receiving voice input from the user and converts the voice into text data using a speech recognition engine (e.g., Google Speech-to-Text API).

[1823] 2. Emotion Recognition Engine:

[1824] The robotic means includes an emotion recognition engine (e.g., Microsoft Azure Emotion API) that analyzes the user's tone of voice and facial expressions to generate emotion data.

[1825] 3. Speech synthesis engine:

[1826] The robot means receives the response text sent from the server means, converts it into voice data using a voice synthesis engine (e.g., Google Text-to-Speech API), and provides it to the user through a speaker.

[1827] Example of how it works:

[1828] 1. User Input:

[1829] Consider a situation where a user speaks to a robot. For example, suppose the user says, "What's your recommendation for today?"

[1830] 2. Speech Recognition:

[1831] The robot means receives the user's voice through a microphone and converts it into text data such as "What's your recommendation today?" using a voice recognition engine.

[1832] 3. Emotion analysis:

[1833] The emotion recognition engine analyzes the user's tone of voice and facial expressions to generate emotion data such as "gratitude" or "interest."

[1834] 4. Data transmission:

[1835] The robot means transmits the generated text data and emotion data to the server means.

[1836] 5. Response Generation:

[1837] The server analyzes the text data using an NLP engine and generates an appropriate response text using a generative artificial intelligence model, adjusting the response by taking into account the emotional data, and generating a response such as "We have a special product for you today. We think you'll be interested."

[1838] 6. Speech synthesis:

[1839] The robot means receives the response text sent from the server means, converts it into voice data using a voice synthesis engine, and provides it to the user through a speaker.

[1840] Examples:

[1841] An example of a prompt is:

[1842] "User sentiment is interesting. Based on recent user sentiment, respond to the following question: What's your recommendation today?"

[1843] In this way, the system of the present invention can improve the quality of customer service by understanding the user's emotions and providing responses based on them.

[1844] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1845] Step 1:

[1846] The user speaks to the robot. Specifically, the user says, "What's your recommendation for today?" The user's voice input is received by the robot's microphone.

[1847] Step 2:

[1848] The robot's speech recognition engine (e.g., Google Speech-to-Text API) converts the voice data into text data. It receives the voice data as input, performs data analysis, and generates the text data "What's your recommendation today?" as output.

[1849] Step 3:

[1850] The robot's emotion recognition engine (e.g., Microsoft Azure Emotion API) analyzes the user's voice tone and facial expressions to generate emotional data. It receives voice tone and facial expression data as input, analyzes the data, and generates emotional data such as "interest" or "gratitude" as output.

[1851] Step 4:

[1852] The robot means transmits the text data and emotion data generated in the previous step to the server means, which receives the text data and emotion data as input and transmits them to the server using the communication means.

[1853] Step 5:

[1854] The server means receives the transmitted text data and emotion data, receives the data transmitted from the robot as input, and stores it as internal data.

[1855] Step 6:

[1856] The server's Natural Language Processing (NLP) engine (e.g., BERT) analyzes and understands the text data. It receives the text data as input, performs data analysis, and generates the processed results as output.

[1857] Step 7:

[1858] The server's artificial intelligence model (e.g., GPT-3) generates an appropriate response text based on the processing results of the NLP engine and the emotional data. It receives the analyzed text data and emotional data as input, performs data generation, and generates the response text as output. For example, it generates the response text, "Today we have a special product we highly recommend. We think you'll be interested."

[1859] Step 8:

[1860] The server means transmits the generated response text to the robot means, which receives the generated response text as input and transmits it to the robot using the communication means.

[1861] Step 9:

[1862] The robot means receives the response text from the server, and receives the response text sent from the server as input and stores it as internal data.

[1863] Step 10:

[1864] The robot's speech synthesis engine (e.g., Google Text-to-Speech API) converts the response text into voice data. It receives the response text as input, performs data conversion, and generates voice data as output.

[1865] Step 11:

[1866] The robot means provides the generated voice data to the user through the speaker. The robot means receives the voice data as input and outputs it through the speaker. The user can receive a voice response such as, "Today we have a special product we highly recommend. I'm sure you'll be interested."

[1867] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1868] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1869] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1870] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1871] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1872] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1873] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1874] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1875] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1876] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1877] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1878] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1879] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1880] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1881] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1882] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1883] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1884] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1885] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1886] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1887] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1888] The following is further disclosed regarding the above embodiment.

[1889] (Claim 1)

[1890] A server means for storing a generative artificial intelligence model;

[1891] robot means for converting voice data into text data and transmitting the text data to the server means;

[1892] a robot means for receiving a response from the server means, converting the response into a voice and providing the voice to the user;

[1893] the robot means includes a voice recognition engine and a voice synthesis engine;

[1894] A system including:

[1895] (Claim 2)

[1896] 2. The system according to claim 1, wherein the system aims to alleviate the user's sense of loneliness.

[1897] (Claim 3)

[1898] 10. The system of claim 1, wherein the server means uses a natural language processing (NLP) engine to analyze the text data and a generative artificial intelligence model to generate the response.

[1899]

[1900] "Example 1"

[1901] (Claim 1)

[1902] A server means for storing a generative artificial intelligence model;

[1903] robot means for converting voice data into text data and transmitting the text data to the server means;

[1904] a robot means for receiving a response from the server means, converting the response into a voice and providing the voice to the user;

[1905] the robot means including a speech recognition engine, a communication module, and a speech synthesis engine;

[1906] means for the server means to parse the text data using a natural language processing engine and send a generated response to the robot means;

[1907] A system including:

[1908] (Claim 2)

[1909] 2. The system according to claim 1, wherein the system aims to alleviate the user's sense of loneliness.

[1910] (Claim 3)

[1911] 10. The system of claim 1, wherein the server means uses a generative AI model to generate an appropriate response based on the prompt sentence.

[1912] "Application Example 1"

[1913] (Claim 1)

[1914] A server means for storing a generative artificial intelligence model;

[1915] an information processing device that converts voice data into text data and transmits the text data to the server;

[1916] an information processing device means for receiving a response from the server means, converting the response into a voice and providing the voice to the user;

[1917] the information processing device means includes a speech recognition engine and a speech synthesis engine;

[1918] the information processing device means functions as an interactive order assistant in a food delivery service;

[1919] A system including:

[1920] (Claim 2)

[1921] 2. The system according to claim 1, wherein the system aims to improve convenience for the user.

[1922] (Claim 3)

[1923] 10. The system of claim 1, wherein the server means uses a natural language processing (NLP) engine to analyze the text data and a generative artificial intelligence model to generate the response.

[1924] "Example 2: Combining Emotion Engines"

[1925] (Claim 1)

[1926] A server means for storing a generative artificial intelligence model;

[1927] a terminal means for converting voice data into text data and transmitting the text data to the server means;

[1928] a terminal means for receiving a response from the server means, converting the response into a voice and providing the voice to the user;

[1929] the terminal means includes a speech recognition engine and a speech synthesis engine;

[1930] The terminal means includes an emotion engine that analyzes the user's tone of voice and facial expressions;

[1931] means for adjusting a response based on emotion data by the server means;

[1932] A system including:

[1933] (Claim 2)

[1934] 2. The system according to claim 1, wherein the system aims to alleviate the user's sense of loneliness.

[1935] (Claim 3)

[1936] 10. The system of claim 1, wherein the server means uses a natural language processing (NLP) engine to analyze the text data and a generative artificial intelligence model to generate the response.

[1937] "Application example 2 when combining emotion engines"

[1938] (Claim 1)

[1939] A server means for storing a generative artificial intelligence model;

[1940] robot means for converting voice data into text data and transmitting the text data to the server means;

[1941] a robot means for receiving a response from the server means, converting the response into a voice and providing the voice to the user;

[1942] the robot means includes a voice recognition engine and a voice synthesis engine;

[1943] the robot means includes an emotion recognition engine and means for analyzing the user's emotional state;

[1944] server means for adjusting response text based on emotion data from said robot means;

[1945] A system including:

[1946] (Claim 2)

[1947] 10. The system of claim 1, for customer service, for providing responses based on a user's emotional state.

[1948] (Claim 3)

[1949] 10. The system of claim 1, wherein the server means uses a natural language processing (NLP) engine to analyze the text data and a generative artificial intelligence model to generate the response. [Explanation of symbols]

[1950] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A server means for storing a generative artificial intelligence model; robot means for converting voice data into text data and transmitting the text data to the server means; a robot means for receiving a response from the server means, converting the response into a voice and providing the voice to the user; the robot means includes a voice recognition engine and a voice synthesis engine; A system including:

2. The system according to claim 1 , which aims to alleviate the user's sense of loneliness.

3. 10. The system of claim 1, wherein said server means uses a natural language processing engine to parse the text data and a generative artificial intelligence model to generate responses.

Images