System

Abstract

To provide a system for smoothing communication with a user by providing a friendly interactive agent to which humor is imparted.SOLUTION: The data processing system 10 comprises a data processing apparatus 12 and a smart device 14, and includes means for receiving user input, means for analyzing the user input, means for generating humor based on the analysis, means for generating a response based on the generated humor, and means for displaying the response.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current conversational agents often respond in an impersonal manner, making it difficult for users to feel familiar with them. For this reason, there is a need to realize natural and friendly dialogue when communicating with users. In particular, there is a lack of technology that allows for dialogue that incorporates humor, making it difficult to communicate smoothly with users. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing the following means. That is, a system including means for receiving a user's input, means for analyzing the user's input, means for generating humor based on the analysis results, means for generating a response based on the generated humor, and means for displaying the response. Furthermore, by including means for generating gestures based on the analysis results, the system achieves more friendly dialogue. Furthermore, by including means for converting the user's input from voice to text, the system is also compatible with voice input. This makes it possible to provide a friendly interactive agent with a humorous touch, facilitating smooth communication with the user.

[0006] "User input" is any speech or text information provided by a user to an interactive agent.

[0007] "Means for analysis" refers to software or algorithms that analyze user input and understand its intent and content.

[0008] "Humor generation means" refers to software or algorithms that generate appropriate humor or jokes based on the analysis results.

[0009] A "means for generating a response" is software or an algorithm that assembles the generated humor into a response and presents it to the user.

[0010] The "means for displaying a response" refers to a display device or audio output device for visually or audibly presenting the generated response to the user.

[0011] The "means for generating gestures" refers to software or algorithms that create gestures according to the content of the response and control their movements using a 3D avatar or the like.

[0012] A "speech-to-text means" is speech recognition software or algorithms that converts voice input into text form. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention is a system for adding humor to an interactive agent, and for realizing friendly dialogue with the user. A specific embodiment of this system will be described below.

[0035] System Configuration

[0036] 1. A way to receive user input

[0037] The device receives voice or text input from the user. For example, if a user asks "What's the weather like today?", the device's microphone captures the voice.

[0038] 2. A means of converting input from speech to text

[0039] The voice recognition software installed on the device converts the captured voice into text, and the speech "What's the weather like today?" is converted into text data and sent to the server.

[0040] 3. Text Analysis and Intent Understanding

[0041] The server analyzes the received text data to understand the user's intent. Specifically, it uses a natural language processing engine to identify that the question is about the weather.

[0042] 4. Selecting and generating humor

[0043] Based on the analysis results, the server uses a generative AI model to generate appropriate humor. For example, for the topic "weather," it generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0044] 5. Response and Gesture Generation

[0045] The server generates a response text for the user based on the generated humor, and simultaneously generates control data for the 3D avatar, creating a response that includes actions such as opening arms, smiling, and speaking.

[0046] 6. Sending response data

[0047] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[0048] 7. Displaying the response to the user

[0049] The device displays the received data and plays an animation in which a 3D avatar holds out its arms to the user and says with a smile, "The weather's sunny today... I'm not sure, but maybe you should go outside and check!"

[0050] Specific examples

[0051] The user asks, "What's the weather like today?" The device captures the voice and converts it into text using speech recognition software. The converted text is sent to a server where it is analyzed using a natural language processing engine. The server uses a generative AI model to create weather-related humor and determines the response: "The weather is sunny today... I'm not sure, but maybe you should go outside and check!" At the same time, movement data is generated for the 3D avatar to spread its arms and speak. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[0052] In this way, the present invention can provide users with friendly, humorous interactions.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user asks the terminal by voice, "What's the weather like today?"

[0056] Step 2:

[0057] The device temporarily stores the audio received.

[0058] Step 3:

[0059] Speech recognition software installed on the device converts the saved voice data into text data.

[0060] Step 4:

[0061] The terminal transmits the converted text data to the server.

[0062] Step 5:

[0063] The server uses a natural language processing engine to analyze the text data it receives and understand the user's intent. In this case, it identifies that the question is about the weather.

[0064] Step 6:

[0065] The server uses a generative AI model based on the analysis to generate appropriate humor, in this case, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0066] Step 7:

[0067] The server generates a response text for the user based on the generated humor.

[0068] Step 8:

[0069] At the same time, the server generates control data for the 3D avatar, creating the avatar's movements of spreading its arms, smiling, and talking.

[0070] Step 9:

[0071] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[0072] Step 10:

[0073] The device displays the received data and plays an animation in which a 3D avatar opens its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[0074] This specific processing flow allows the user to receive friendly responses, including humor, from the conversational agent.

[0075] Example 1

[0076] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0077] In conversational agents, it is necessary to add humor to realize friendly dialogue with users. However, conventional systems have had difficulty generating appropriate and natural humor in response to user input. Furthermore, converting speech to text and generating and displaying actions, including gestures, have also been issues.

[0078] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0079] In this invention, the server includes means for receiving a user input, means for converting the user input from speech to text using speech recognition software, means for analyzing the user input, means for generating a prompt sentence based on the analysis result and generating humor using a generative AI model, means for generating a response text and hand gestures based on the generated humor, and means for displaying the response text and hand gestures to the user. This allows for the generation of appropriate and natural humor in response to the user input, and the generation and display of actions including speech-to-text conversion and hand gestures.

[0080] "User" refers to any individual or end user who uses this System.

[0081] "Input" refers to the provision of information in the form of speech or text by a user to a system.

[0082] "Speech recognition software" refers to a program or algorithm used to analyze captured voice data and convert it into text data.

[0083] "Text" refers to character strings of data that correspond to a user's spoken input, as converted by speech recognition software.

[0084] "Analysis" refers to the process of understanding the user's intent based on the received text data.

[0085] A "prompt" is an instruction given to a generative AI model to generate humor based on user input.

[0086] "Generative AI model" refers to an artificial intelligence model that generates humor based on prompt text.

[0087] "Humor" refers to the friendly, entertaining responses to user input generated by the system.

[0088] "Responsive Text" refers to a humorous text response to a user's input that is created by a generative AI model.

[0089] "Gestures" refers to the physical expressions, such as gestures and facial expressions, that a 3D avatar makes toward the user.

[0090] "Display" refers to projecting text and 3D avatar movements onto the device screen to provide a visual experience to the user.

[0091] "Terminal" means the electronic device through which a User accesses, enters input into, and receives responses to this System.

[0092] The present invention is a system for adding humor to an interactive agent, and for realizing friendly dialogue with the user. A specific embodiment of this system will be described below.

[0093] 1. System Configuration

[0094] The system consists of a device for receiving user input, speech recognition software for converting the user's input from speech to text, and a server for analyzing the received text data and generating appropriate humor. Specifically, the device includes a microphone and speech recognition software (e.g., Google Speech-to-Text), while the server is equipped with a natural language processing engine (e.g., SpaCy) and a generative AI model (e.g., OpenAI GPT-3).

[0095] 2. Processing Flow

[0096] The user speaks to the device, asking, "What's the weather like today?" The device's microphone captures the voice, and speech recognition software converts the voice into text data. The converted text data is sent to the server in the form of "What's the weather like today?"

[0097] The server analyzes the received text data using a natural language processing engine and identifies it as a question about the weather. Based on the analysis results, it inputs the following prompt sentence into the generative AI model: "The user is asking, 'What's the weather like today?' Please provide a humorous response to this question."

[0098] Based on this prompt, the generative AI model generates a humorous response: "I'm not sure if the weather is sunny today, but maybe I should go outside and check it out!" The server also generates motion data for controlling the 3D avatar based on this response text. The generated control data includes the motion of the 3D avatar speaking with its arms outstretched and smiling.

[0099] 3. View the response

[0100] The generated response text and action data are sent from the server to the device. The device then displays a 3D avatar on the screen based on the received data, and plays an animation of the avatar spreading its arms and smiling, saying, "I don't know if the weather is sunny today... but maybe I should go outside and check!" In this way, the user can receive humorous responses along with the 3D avatar's actions.

[0101] Specific examples

[0102] For example, if a user asks "What's the weather like today?", the device captures the speech and uses speech recognition software to convert it into text. The converted text is sent to a server, which then analyzes it using a natural language processing engine. The server then uses a generative AI model to create weather-related humor and determines a response such as "The weather is sunny today... I'm not sure, but maybe you should go outside and check!" At the same time, movement data is also generated for the 3D avatar to spread its arms and speak. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[0103] An example of the prompt above is: "The user asks, 'What's the weather like today?' Please respond to this question with humor."

[0104] This invention makes it possible to generate appropriate and natural humor in response to user input, convert speech to text, and generate and display actions including gestures.

[0105] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0106] Step 1:

[0107] The user speaks, "What's the weather like today?"

[0108] The device captures this audio with a microphone.

[0109] Input data: User's voice.

[0110] Output data: Audio data.

[0111] Step 2:

[0112] The device uses voice recognition software (e.g., Google Speech-to-Text) to convert the captured voice data into text data.

[0113] Input data: Captured audio data.

[0114] Output data: Text data "What's the weather like today?"

[0115] Step 3:

[0116] The terminal transmits the converted text data to the server.

[0117] Input data: Text data "What's the weather like today?"

[0118] Output data: Sending text data to the server.

[0119] Step 4:

[0120] The server analyzes the received text data using a natural language processing engine (e.g., SpaCy).

[0121] Input data: Text data "What's the weather like today?"

[0122] Data processing: Analyze the received text data and identify that it is a question about the weather.

[0123] Output: Parsed intent ("Question about the weather").

[0124] Step 5:

[0125] The server generates a prompt based on the analysis results and sends it to the generative AI model (e.g., OpenAI GPT-3). The specific prompt is, "The user is asking, 'What's the weather like today?' Please respond to this question with humor."

[0126] Input data: Parsed intent ("Question about the weather").

[0127] Data processing: Prompt sentence generation.

[0128] Output data: The prompt sentence to be passed to the generative AI model.

[0129] Step 6:

[0130] The generative AI model generates an appropriate humorous response based on the prompt sentence.

[0131] Input data: prompt statement.

[0132] Data operations: Operations that analyze prompt sentences and generate humorous responses.

[0133] Output: A humorous response: "The weather is sunny today... I don't know, but maybe I should go outside and check!"

[0134] Step 7:

[0135] The server generates motion data for controlling the 3D avatar based on the humorous response text, including the motion of the avatar spreading its arms and speaking.

[0136] Input data: Humor response text.

[0137] Data processing: Generation of 3D avatar movements.

[0138] Output data: response text and 3D avatar movement data.

[0139] Step 8:

[0140] The server transmits the generated response text and the motion data of the 3D avatar to the terminal.

[0141] Input data: response text and 3D avatar movement data.

[0142] Output data: Data sent to the device.

[0143] Step 9:

[0144] Based on the data received, the device displays a 3D avatar on the screen and plays an animation in which the avatar spreads its arms and says with a smile, "I don't know if the weather will be sunny today... but maybe I should go outside and check!"

[0145] Input data: response text and 3D avatar movement data.

[0146] Data calculation: 3D avatar movement playback.

[0147] Output data: Responses and avatar behaviors displayed to the user.

[0148] (Application example 1)

[0149] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0150] Current conversational agent systems face challenges in incorporating humor appropriately into dialogue with users, making it difficult to achieve friendly, natural conversations. Furthermore, when agents interact with drivers and passengers in autonomous vehicles, they are unable to generate humorous responses and provide enjoyment. There is a need for conversational agent systems that incorporate humor to solve these problems and improve the user experience.

[0151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0152] In this invention, the server includes means for receiving user input, means for converting the user input from speech to text, means for analyzing the text to understand the user's intent, means for generating humor based on the analysis results using a generative AI model, means for generating a response based on the generated humor, and means for displaying the response or playing it as an animation, thereby realizing a friendly dialogue with the user and providing a fun experience even inside an autonomous vehicle.

[0153] "Means for receiving user input" refers to devices or software for receiving instructions or questions from users, such as voice input or text input.

[0154] "Speech-to-text means" means speech recognition software or algorithms used to convert spoken input into text form.

[0155] "Means for analyzing text and understanding user intent" refers to a natural language processing engine that analyzes input text data and identifies the user's intent and purpose from its content.

[0156] "Means for generating humor using a generative AI model" refers to a method that uses a generative AI model (e.g., a GPT model) to generate relevant humor based on the analysis results.

[0157] "Means for generating responses based on generated humor" refers to software or algorithms that use the generated humor to create responses to users.

[0158] "Means for displaying a response or playing it as an animation" refers to a display device for visually conveying the generated response to the user, an engine for controlling the animation of an avatar, etc.

[0159] The present invention is a system that analyzes user input, generates humorous responses, and displays them. This system is intended to be installed in an autonomous vehicle, and specific embodiments are described below.

[0160] System Configuration

[0161] 1. A way to receive user input

[0162] The autonomous vehicle is equipped with a microphone that captures voice input from the user, and this voice data includes the user's questions and instructions.

[0163] 2. Voice-to-text conversion methods

[0164] The terminal inside the vehicle is installed with voice recognition software (e.g., Google Speech-to-Text API) that converts the captured voice data into text format.

[0165] 3. Text Analysis and Intent Understanding

[0166] The converted text is sent to a server and analyzed by a natural language processing engine (e.g., Google Cloud Natural Language API). This analysis allows the system to understand the user's intent and generate appropriate answers based on that.

[0167] 4. Selecting and generating humor

[0168] The server uses a generative AI model (such as OpenAI's GPT model) based on the analysis results to generate appropriate humor, enabling friendly dialogue.

[0169] 5. Generating Responses and Animations

[0170] Based on the humor generated, the server generates a response text for the user. At the same time, it generates avatar movement data using a 3D avatar generation engine (e.g., Unreal Engine). This movement data includes information on the avatar's facial expressions and gestures.

[0171] 6. Sending response data

[0172] The generated text response and avatar movement data are sent to the device using a communication module (e.g., WebSocket).

[0173] 7. Displaying the response to the user

[0174] The received data is displayed on the vehicle's display and an avatar animation is played, providing users with a natural and enjoyable interaction experience.

[0175] Example

[0176] For example, if a user asks, "What's the nearest restaurant?", the voice is first captured by a microphone. Speech recognition software then converts the voice into text, which is then sent to the server. The server then analyzes it with a natural language processing engine to understand the intent. Based on the results, the generative AI model generates a humorous response, such as, "The nearest restaurant is just around the corner... but the food has mixed reviews!" The response is displayed alongside an animated 3D avatar, allowing the user to experience the humorous response.

[0177] Prompt Sentence Examples

[0178] The prompt to the generative AI model would be:

[0179] Generate a humorous response about "nearest restaurant."

[0180] This embodiment allows for friendly interaction with the user, providing a fun experience even inside an autonomous vehicle.

[0181] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0182] Step 1:

[0183] A user provides voice input inside an autonomous vehicle. The user's input is captured through a microphone, and voice data is obtained as input, which is then used in subsequent processing steps.

[0184] Step 2:

[0185] The device converts the captured voice data into text data using speech recognition software (e.g., Google Speech-to-Text API). The voice data is converted into acoustic features, and the speech recognition algorithm generates a text string based on the features. The text data is obtained as output.

[0186] Step 3:

[0187] The device sends the converted text data to a server. A natural language processing engine (e.g., Google Cloud Natural Language API) connected to the server receives the text data and begins analyzing it. The input text data is analyzed, and the user's intent and topic are identified as the analysis results.

[0188] Step 4:

[0189] Based on the analysis results, the server generates humor using a generative AI model (e.g., OpenAI's GPT model). Here, the generative AI model is prompted to generate a humorous response about the "nearest restaurant." As an output, the generative AI model generates a humorous response text.

[0190] Step 5:

[0191] The server generates a response text for the user and corresponding animation data for a 3D avatar based on the generated humor text. Using a 3D avatar generation engine (e.g., Unreal Engine), motion data for the avatar to spread its arms, smile, and talk is generated. The response text and animation data are obtained as output.

[0192] Step 6:

[0193] The generated response text and avatar animation data are sent from the server to the device using a communication module (e.g., WebSocket). The device analyzes the received data and prepares it for display.

[0194] Step 7:

[0195] The terminal displays the response text on the vehicle's display and simultaneously plays an animation of the avatar, providing the user with a visual response, which allows the user to enjoy a friendly and enjoyable interaction experience. The output is the text and animation displayed on the display.

[0196] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0197] This invention relates to a system that combines humor and emotion recognition with a conversational agent, enabling friendly interactions with the user and providing appropriate responses depending on the user's emotions. Specific embodiments of this system are described below.

[0198] System Configuration

[0199] 1. A way to receive user input

[0200] The device receives voice or text input from the user. For example, if a user asks "What's the weather like today?", the device's microphone captures the voice.

[0201] 2. A means of converting input from speech to text

[0202] The voice recognition software installed on the device converts the captured voice into text, and the speech "What's the weather like today?" is converted into text data and sent to the server.

[0203] 3. Text Analysis and Intent Understanding

[0204] The server analyzes the received text data to understand the user's intent. Specifically, it uses a natural language processing engine to identify that the question is about the weather.

[0205] 4. Use of Emotion Engine

[0206] The emotion engine installed on the server recognizes the user's emotions from their voice or text data, for example, identifying whether the user is excited or depressed.

[0207] 5. Selecting and generating humor

[0208] The server uses a generative AI model to generate appropriate humor based on the analysis results and the recognition results of the emotion engine. In this case, for the topic "weather," it generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0209] 6. Response and Gesture Generation

[0210] The server generates a response text for the user based on the generated humor. At the same time, it generates control data for a 3D avatar based on the recognition results of the emotion engine, creating a response that includes actions such as spreading arms, smiling, and speaking.

[0211] 7. Sending response data

[0212] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[0213] 8. Displaying the response to the user

[0214] The device displays the received data and plays an animation in which a 3D avatar extends its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[0215] Specific examples

[0216] The user asks, "What's the weather like today?" The device receives the voice and converts it into text using speech recognition software. The converted text is sent to a server where it is analyzed by a natural language processing engine. The server uses a generative AI model to create weather-related humor and determines the response: "The weather is sunny today... I'm not sure, but maybe you should go outside and check!" At the same time, the emotion engine analyzes the user's emotions. If it determines that the user is feeling down, it generates behavior data for the 3D avatar, including more friendly and comforting actions. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[0217] In this way, the present invention can provide a user with a humorous dialogue that is friendly to the user, and can also respond appropriately according to the user's emotions.

[0218] The processing flow will be explained below.

[0219] Step 1:

[0220] The user asks the device verbally, "What's the weather like today?"

[0221] Step 2:

[0222] The device captures the user's voice and temporarily stores it.

[0223] Step 3:

[0224] Speech recognition software installed on the device converts the captured voice into text data.

[0225] Step 4:

[0226] The terminal transmits the converted text data to the server.

[0227] Step 5:

[0228] The server uses a natural language processing engine to analyze the text data it receives and understand the user's intent. In this case, it identifies that the question is about the weather.

[0229] Step 6:

[0230] The server then uses the analysis results to activate the emotion engine and analyze the user's emotional state, for example, identifying whether the user is excited or depressed.

[0231] Step 7:

[0232] Based on the analysis results and the recognition results of the emotion engine, the server uses a generative AI model to select an appropriate humor. In this case, it generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0233] Step 8:

[0234] The server generates a response text for the user based on the generated humor, and adjusts the response content according to the recognition results of the emotion engine.

[0235] Step 9:

[0236] At the same time, the server generates control data for the 3D avatar and configures it to perform movements that reflect the corresponding emotions (e.g., speaking with a smile and spreading one's arms).

[0237] Step 10:

[0238] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[0239] Step 11:

[0240] The device displays the received data and plays an animation in which a 3D avatar opens its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[0241] This specific processing flow allows the user to receive friendly responses from the conversational agent that reflect humor and emotion.

[0242] Example 2

[0243] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0244] Conventional conversational agent systems often respond to user input mechanically and without considering emotions, making interactions with users unfriendly. Furthermore, there is a lack of systems that combine humorous responses and behaviors that respond to the user's emotions, making it impossible to enrich the user experience. This has led to issues such as a decline in the quality of interaction with users and low satisfaction.

[0245] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0246] In this invention, the server includes means for receiving a user's input, means for analyzing the user's input, means for recognizing the analysis result and emotion, means for generating humor based on the analysis result and the emotion recognition, and means for generating a response and gesture based on the generated humor, thereby enabling a response including appropriate humor and action according to the user's emotion.

[0247] "Means for receiving user input" refers collectively to any interface or device that captures information when a user provides it to a system in the form of voice, text, or other information.

[0248] "Means of analysis" is a general term for algorithms and software that process input data received from users and extract specific intent or meaning.

[0249] "Emotion recognition means" is a general term for algorithms and engines that analyze and identify a user's emotional state from their voice or text data.

[0250] "Humor generation means" is a general term for algorithms and models that generate humorous content for users based on the results of analysis and emotion recognition.

[0251] "Means for generating responses and gestures" refers collectively to the algorithms and engines that control and generate response text and corresponding actions and gestures to the user based on the generated humor.

[0252] "Means for receiving" is a general term for interfaces and devices for receiving input from users and data from servers.

[0253] "Displaying means" is a general term for devices and software that visually or audibly present information to the user, such as generated response text or gestures.

[0254] This invention is a system that combines humor and emotion recognition to realize friendly dialogue with users in a conversational agent. The system receives voice or text input from the user, analyzes the input, and generates appropriate humorous responses and emotionally appropriate actions to present to the user.

[0255] System Configuration

[0256] 1. A way to receive user input

[0257] The device receives voice or text input from the user. For example, when a user asks, "What's the weather like today?", the device's microphone captures the audio. Required hardware includes a microphone, speaker, and display.

[0258] 2. A means of converting input from speech to text

[0259] Speech recognition software (e.g., Google Cloud Speech-to-Text) installed on the device converts the captured voice data into text data, which becomes "What's the weather like today?" and is sent to the server.

[0260] 3. Text Analysis and Intent Understanding

[0261] The server analyzes the received text data and uses a natural language processing engine (e.g., Google Natural Language API) to understand the user's intent, thereby identifying that the user's question is about the weather.

[0262] 4. Use of Emotion Engine

[0263] The emotion engine (e.g., IBM Watson Tone Analyzer) installed on the server recognizes emotions from the user's text data, and identifies whether the user is excited or depressed.

[0264] 5. Selecting and generating humor

[0265] Based on the analysis and emotion recognition results, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate humor, such as "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0266] 6. Response and Gesture Generation

[0267] The server generates a response text based on the generated humor, and simultaneously generates motion data for a 3D avatar based on the recognition results of the emotion engine, including, for example, a motion in which the avatar smiles and spreads its arms.

[0268] 7. Sending response data

[0269] The server transmits the generated response text and the motion data of the 3D avatar to the terminal.

[0270] 8. Displaying the response to the user

[0271] The device displays the received data and plays an animation in which a 3D avatar extends its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[0272] Specific examples

[0273] The user asks, "What's the weather like today?" The device receives the voice and converts it into text using speech recognition software. The converted text is sent to a server where it is analyzed by a natural language processing engine. The server uses a generative AI model to create a weather-related joke and determines the response, "The weather is sunny today... I don't know, but maybe you should go outside and check!" At the same time, the emotion engine analyzes the user's emotions, and if it determines that the user is feeling down, the 3D avatar will take comforting actions. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[0274] Prompt Sentence Examples

[0275] "Generate humorous responses to users asking about today's weather, and provide responses with comforting 3D avatar behavior if the user is feeling down."

[0276] By feeding this prompt into a generative AI model, the system can generate a response for the user that includes appropriate humor and actions.

[0277] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0278] Step 1:

[0279] User inputs by voice.

[0280] The user asks the device, "What's the weather like today?"

[0281] Input: User's voice data

[0282] Output: Audio data

[0283] Step 2:

[0284] The device captures the audio.

[0285] The device's microphone captures the user's voice and stores it as digital audio data.

[0286] Input: Audio data

[0287] Output: Digital audio data

[0288] Step 3:

[0289] Text conversion using voice recognition software installed on the device.

[0290] Speech recognition software (e.g., Google Cloud Speech-to-Text) converts digital voice data into text data.

[0291] Input: Digital audio data

[0292] Output: Text data "What's the weather like today?"

[0293] Step 4:

[0294] The device sends text data to the server.

[0295] The terminal sends the converted text data to the server, typically requiring an HTTP request.

[0296] Input: Text data

[0297] Output: HTTP request to the server

[0298] Step 5:

[0299] The server receives and analyzes the text data.

[0300] The server receives the HTTP request and analyzes the text data using a natural language processing engine (e.g., Google Natural Language API). This analysis helps to understand the user's intent and identifies the question as being about the weather.

[0301] Input: Text data

[0302] Output: Text analysis results

[0303] Step 6:

[0304] The server analyzes emotions using an emotion engine.

[0305] The emotion engine (e.g., IBM Watson Tone Analyzer) installed on the server recognizes the user's emotion from the text data, for example, identifying the user's emotional state, such as whether they are excited or depressed.

[0306] Input: Text data

[0307] Output: Emotion analysis results

[0308] Step 7:

[0309] Server generated humor.

[0310] Based on the results of text analysis and sentiment analysis, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate humor. It generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0311] Input: Text analysis results, sentiment analysis results

[0312] Output: Humorous response text

[0313] Step 8:

[0314] The server generates the response text and action data.

[0315] The server creates a response text based on the generated humor. At the same time, it generates motion data for the 3D avatar based on the analysis results of the emotion engine. For example, it creates a motion for the avatar to speak with its arms outstretched and smiling.

[0316] Input: Humorous response text, sentiment analysis results

[0317] Output: Response text, action data

[0318] Step 9:

[0319] The server sends the response data to the terminal.

[0320] The server sends the generated response text and the 3D avatar's movement data to the terminal.

[0321] Input: Response text, action data

[0322] Output: HTTP response to the device

[0323] Step 10:

[0324] The device responds.

[0325] The device displays the received data and plays an animation in which a 3D avatar holds out its arms to the user and says with a smile, "The weather's sunny today... I'm not sure, but maybe you should go outside and check!"

[0326] Input: Response text, action data

[0327] Output: Animation and audio response display

[0328] (Application example 2)

[0329] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0330] Conventional conversational agent systems have difficulty responding flexibly to the user's emotions and circumstances, and have been unable to sufficiently improve user satisfaction. Furthermore, generating humorous responses requires taking into account the user's current emotional state, but the lack of a function to dynamically adjust the humor accordingly can sometimes result in an inconsistent user experience.

[0331] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a user's input, means for analyzing the user's input, means for generating humor based on the analysis result, means for generating and displaying a response based on the generated humor and outputting it as voice, means for performing emotion recognition, and means for adjusting the humor in accordance with the emotion recognition result. This makes it possible to provide an appropriate and friendly response, mixed with humor, that is in accordance with the user's emotional state.

[0332] A "means for receiving user input" is a device or interface for capturing speech or text generated by a user.

[0333] "Means for analyzing user input" means software or a system that interprets captured voice or text data and analyzes its intent and content.

[0334] "Means for generating humor based on the analysis results" refers to algorithms or generative AI models that create appropriate humor based on the analyzed input data.

[0335] The "means for generating and displaying a response based on the generated humor and outputting it as audio" is a mechanism for displaying a response containing the generated humor as text and playing it back as audio using speech synthesis technology.

[0336] "Means for emotion recognition" means software or emotion engines that analyze and identify a user's emotional state from their voice or text.

[0337] The "means for adjusting humor according to the results of emotion recognition" is a mechanism for optimizing the generated humor and selecting an appropriate response based on the user's emotional state.

[0338] This invention is an interactive agent system that provides friendly dialogue with users and responds appropriately based on the user's emotions. This system consists of a device that receives the user's voice and text input, a server that analyzes the input data, and a terminal that generates and displays humorous responses and outputs them as voice.

[0339] Hardware and software used

[0340] Hardware: The microphone and speaker on your smartphone.

[0341] Software: Speech recognition library (speech_recognition), text-to-speech engine (pyttsx3), emotion recognition API, generative AI model (OpenAI GPT-3).

[0342] Process Overview

[0343] 1. Voice input acquisition:

[0344] The user inputs voice through the smartphone, and the smartphone's microphone captures the user's voice.

[0345] 2. Speech Recognition:

[0346] Smartphones use speech recognition software (speech_recognition) to convert captured speech into text, for example, Google's speech recognition API.

[0347] 3. Emotion recognition:

[0348] The converted text data is sent to the server, which then analyzes the user's emotions using an emotion recognition API to assess the user's emotional state.

[0349] 4. Humorous response generation:

[0350] The server uses a generative AI model (OpenAI GPT-3) to generate humorous responses to text data, using prompts to instruct the AI.

[0351] Example prompt sentence:

[0352] User: I hate the long traffic jams.

[0353] AI: Provide a humorous response.

[0354] 5. Audio output of response:

[0355] The server sends the generated humorous response to the smartphone, which uses the smartphone's text-to-speech engine (pyttsx3) to output the response aloud.

[0356] Specific examples

[0357] For example, if a user says "I hate being stuck in traffic jams for so long" in a self-driving vehicle, the smartphone's microphone picks up the speech. The speech recognition software then converts the speech into text, "I hate being stuck in traffic jams for so long," and sends it to the server. The server then uses an emotion recognition API to determine that the user is "angry." Based on this information, the generative AI model (OpenAI GPT-3) generates the following humorous response: "It's good to relax for a while," and the response is output via voice via the smartphone.

[0358] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0359] Step 1:

[0360] The user inputs voice using the microphone on the smartphone, and the user's voice data is captured by the microphone and sent to the device.

[0361] Step 2:

[0362] The device uses a speech recognition library (speech_recognition) to convert the captured voice data into text data. This process converts the user's voice input into text format. The input is voice data and the output is text data.

[0363] Step 3:

[0364] The converted text data is sent from the terminal to the server. The server receives this text data and proceeds to the next step. The input is text data, and the output is sending the data to the server.

[0365] Step 4:

[0366] The server uses an emotion recognition API to analyze the user's emotions from the received text data. Specifically, it extracts emotion tags (e.g., angry, sad, happy, etc.) from the text content. Through this process, the input is text data and the output is emotion tags.

[0367] Step 5:

[0368] The server uses a generative AI model (OpenAI GPT-3) to generate an appropriate humorous response based on the emotion tag and text data. Here, a prompt is used to instruct the AI. Specifically, a prompt such as "User: I'm tired of the long traffic jam. AI: Provide a humorous response" is created and passed to the AI ​​model. This process results in emotion tag and text data as input, and a humorous response as output.

[0369] Step 6:

[0370] Based on the generated humorous response and emotion tag, the response is adjusted as needed. For example, if the emotion tag is "angry," an additional phrase such as "It's good to relax for a while" is inserted into the humorous response. With this process, the input is a humorous response and emotion tag, and the output is the adjusted humorous response.

[0371] Step 7:

[0372] The server sends the adjusted humor response to the terminal. The input is the adjusted humor response, and the output is the transmission of data to the terminal.

[0373] Step 8:

[0374] The device converts the received tailored humor response into speech using a text-to-speech engine (pyttsx3) and outputs it to the user through the speaker. As a result of this process, the input is the tailored humor response and the output is the audio response.

[0375] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0376] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0377] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0378] [Second embodiment]

[0379] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0380] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0381] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0382] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0383] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0384] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0385] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0386] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0387] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0388] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0389] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0390] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0391] The present invention is a system for adding humor to an interactive agent, and for realizing friendly dialogue with the user. A specific embodiment of this system will be described below.

[0392] System Configuration

[0393] 1. A way to receive user input

[0394] The device receives voice or text input from the user. For example, if a user asks "What's the weather like today?", the device's microphone captures the voice.

[0395] 2. A means of converting input from speech to text

[0396] The voice recognition software installed on the device converts the captured voice into text, and the speech "What's the weather like today?" is converted into text data and sent to the server.

[0397] 3. Text Analysis and Intent Understanding

[0398] The server analyzes the received text data to understand the user's intent. Specifically, it uses a natural language processing engine to identify that the question is about the weather.

[0399] 4. Selecting and generating humor

[0400] Based on the analysis results, the server uses a generative AI model to generate appropriate humor. For example, for the topic "weather," it generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0401] 5. Response and Gesture Generation

[0402] The server generates a response text for the user based on the generated humor, and simultaneously generates control data for the 3D avatar, creating a response that includes actions such as opening arms, smiling, and speaking.

[0403] 6. Sending response data

[0404] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[0405] 7. Displaying the response to the user

[0406] The device displays the received data and plays an animation in which a 3D avatar holds out its arms to the user and says with a smile, "The weather's sunny today... I'm not sure, but maybe you should go outside and check!"

[0407] Specific examples

[0408] The user asks, "What's the weather like today?" The device captures the voice and converts it into text using speech recognition software. The converted text is sent to a server where it is analyzed using a natural language processing engine. The server uses a generative AI model to create weather-related humor and determines the response: "The weather is sunny today... I'm not sure, but maybe you should go outside and check!" At the same time, movement data is generated for the 3D avatar to spread its arms and speak. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[0409] In this way, the present invention can provide users with friendly, humorous interactions.

[0410] The processing flow will be explained below.

[0411] Step 1:

[0412] The user asks the terminal by voice, "What's the weather like today?"

[0413] Step 2:

[0414] The device temporarily stores the audio received.

[0415] Step 3:

[0416] Speech recognition software installed on the device converts the saved voice data into text data.

[0417] Step 4:

[0418] The terminal transmits the converted text data to the server.

[0419] Step 5:

[0420] The server uses a natural language processing engine to analyze the text data it receives and understand the user's intent. In this case, it identifies that the question is about the weather.

[0421] Step 6:

[0422] The server uses a generative AI model based on the analysis to generate appropriate humor, in this case, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0423] Step 7:

[0424] The server generates a response text for the user based on the generated humor.

[0425] Step 8:

[0426] At the same time, the server generates control data for the 3D avatar, creating the avatar's movements of spreading its arms, smiling, and talking.

[0427] Step 9:

[0428] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[0429] Step 10:

[0430] The device displays the received data and plays an animation in which a 3D avatar opens its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[0431] This specific processing flow allows the user to receive friendly responses, including humor, from the conversational agent.

[0432] Example 1

[0433] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0434] In conversational agents, it is necessary to add humor to realize friendly dialogue with users. However, conventional systems have had difficulty generating appropriate and natural humor in response to user input. Furthermore, converting speech to text and generating and displaying actions, including gestures, have also been issues.

[0435] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0436] In this invention, the server includes means for receiving a user input, means for converting the user input from speech to text using speech recognition software, means for analyzing the user input, means for generating a prompt sentence based on the analysis result and generating humor using a generative AI model, means for generating a response text and hand gestures based on the generated humor, and means for displaying the response text and hand gestures to the user. This allows for the generation of appropriate and natural humor in response to the user input, and the generation and display of actions including speech-to-text conversion and hand gestures.

[0437] "User" refers to any individual or end user who uses this System.

[0438] "Input" refers to the provision of information in the form of speech or text by a user to a system.

[0439] "Speech recognition software" refers to a program or algorithm used to analyze captured voice data and convert it into text data.

[0440] "Text" refers to character strings of data that correspond to a user's spoken input, as converted by speech recognition software.

[0441] "Analysis" refers to the process of understanding the user's intent based on the received text data.

[0442] A "prompt" is an instruction given to a generative AI model to generate humor based on user input.

[0443] "Generative AI model" refers to an artificial intelligence model that generates humor based on prompt text.

[0444] "Humor" refers to the friendly, entertaining responses to user input generated by the system.

[0445] "Responsive Text" refers to a humorous text response to a user's input that is created by a generative AI model.

[0446] "Gestures" refers to the physical expressions, such as gestures and facial expressions, that a 3D avatar makes toward the user.

[0447] "Display" refers to projecting text and 3D avatar movements onto the device screen to provide a visual experience to the user.

[0448] "Terminal" means the electronic device through which a User accesses, enters input into, and receives responses to this System.

[0449] The present invention is a system for adding humor to an interactive agent, and for realizing friendly dialogue with the user. A specific embodiment of this system will be described below.

[0450] 1. System Configuration

[0451] The system consists of a device for receiving user input, speech recognition software for converting the user's input from speech to text, and a server for analyzing the received text data and generating appropriate humor. Specifically, the device includes a microphone and speech recognition software (e.g., Google Speech-to-Text), while the server is equipped with a natural language processing engine (e.g., SpaCy) and a generative AI model (e.g., OpenAI GPT-3).

[0452] 2. Processing Flow

[0453] The user speaks to the device, asking, "What's the weather like today?" The device's microphone captures the voice, and speech recognition software converts the voice into text data. The converted text data is sent to the server in the form of "What's the weather like today?"

[0454] The server analyzes the received text data using a natural language processing engine and identifies it as a question about the weather. Based on the analysis results, it inputs the following prompt sentence into the generative AI model: "The user is asking, 'What's the weather like today?' Please provide a humorous response to this question."

[0455] Based on this prompt, the generative AI model generates a humorous response: "I'm not sure if the weather is sunny today, but maybe I should go outside and check it out!" The server also generates motion data for controlling the 3D avatar based on this response text. The generated control data includes the motion of the 3D avatar speaking with its arms outstretched and smiling.

[0456] 3. View the response

[0457] The generated response text and action data are sent from the server to the device. The device then displays a 3D avatar on the screen based on the received data, and plays an animation of the avatar spreading its arms and smiling, saying, "I don't know if the weather is sunny today... but maybe I should go outside and check!" In this way, the user can receive humorous responses along with the 3D avatar's actions.

[0458] Specific examples

[0459] For example, if a user asks "What's the weather like today?", the device captures the speech and uses speech recognition software to convert it into text. The converted text is sent to a server, which then analyzes it using a natural language processing engine. The server then uses a generative AI model to create weather-related humor and determines a response such as "The weather is sunny today... I'm not sure, but maybe you should go outside and check!" At the same time, movement data is also generated for the 3D avatar to spread its arms and speak. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[0460] An example of the prompt above is: "The user asks, 'What's the weather like today?' Please respond to this question with humor."

[0461] This invention makes it possible to generate appropriate and natural humor in response to user input, convert speech to text, and generate and display actions including gestures.

[0462] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0463] Step 1:

[0464] The user speaks, "What's the weather like today?"

[0465] The device captures this audio with a microphone.

[0466] Input data: User's voice.

[0467] Output data: Audio data.

[0468] Step 2:

[0469] The device uses voice recognition software (e.g., Google Speech-to-Text) to convert the captured voice data into text data.

[0470] Input data: Captured audio data.

[0471] Output data: Text data "What's the weather like today?"

[0472] Step 3:

[0473] The terminal transmits the converted text data to the server.

[0474] Input data: Text data "What's the weather like today?"

[0475] Output data: Sending text data to the server.

[0476] Step 4:

[0477] The server analyzes the received text data using a natural language processing engine (e.g., SpaCy).

[0478] Input data: Text data "What's the weather like today?"

[0479] Data processing: Analyze the received text data and identify that it is a question about the weather.

[0480] Output: Parsed intent ("Question about the weather").

[0481] Step 5:

[0482] The server generates a prompt based on the analysis results and sends it to the generative AI model (e.g., OpenAI GPT-3). The specific prompt is, "The user is asking, 'What's the weather like today?' Please respond to this question with humor."

[0483] Input data: Parsed intent ("Question about the weather").

[0484] Data processing: Prompt sentence generation.

[0485] Output data: The prompt sentence to be passed to the generative AI model.

[0486] Step 6:

[0487] The generative AI model generates an appropriate humorous response based on the prompt sentence.

[0488] Input data: prompt statement.

[0489] Data operations: Operations that analyze prompt sentences and generate humorous responses.

[0490] Output: A humorous response: "The weather is sunny today... I don't know, but maybe I should go outside and check!"

[0491] Step 7:

[0492] The server generates motion data for controlling the 3D avatar based on the humorous response text, including the motion of the avatar spreading its arms and speaking.

[0493] Input data: Humor response text.

[0494] Data processing: Generation of 3D avatar movements.

[0495] Output data: response text and 3D avatar movement data.

[0496] Step 8:

[0497] The server transmits the generated response text and the motion data of the 3D avatar to the terminal.

[0498] Input data: response text and 3D avatar movement data.

[0499] Output data: Data sent to the device.

[0500] Step 9:

[0501] Based on the data received, the device displays a 3D avatar on the screen and plays an animation in which the avatar spreads its arms and says with a smile, "I don't know if the weather will be sunny today... but maybe I should go outside and check!"

[0502] Input data: response text and 3D avatar movement data.

[0503] Data calculation: 3D avatar movement playback.

[0504] Output data: Responses and avatar behaviors displayed to the user.

[0505] (Application example 1)

[0506] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0507] Current conversational agent systems face challenges in incorporating humor appropriately into dialogue with users, making it difficult to achieve friendly, natural conversations. Furthermore, when agents interact with drivers and passengers in autonomous vehicles, they are unable to generate humorous responses and provide enjoyment. There is a need for conversational agent systems that incorporate humor to solve these problems and improve the user experience.

[0508] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0509] In this invention, the server includes means for receiving user input, means for converting the user input from speech to text, means for analyzing the text to understand the user's intent, means for generating humor based on the analysis results using a generative AI model, means for generating a response based on the generated humor, and means for displaying the response or playing it as an animation, thereby realizing a friendly dialogue with the user and providing a fun experience even inside an autonomous vehicle.

[0510] "Means for receiving user input" refers to devices or software for receiving instructions or questions from users, such as voice input or text input.

[0511] "Speech-to-text means" means speech recognition software or algorithms used to convert spoken input into text form.

[0512] "Means for analyzing text and understanding user intent" refers to a natural language processing engine that analyzes input text data and identifies the user's intent and purpose from its content.

[0513] "Means for generating humor using a generative AI model" refers to a method that uses a generative AI model (e.g., a GPT model) to generate relevant humor based on the analysis results.

[0514] "Means for generating responses based on generated humor" refers to software or algorithms that use the generated humor to create responses to users.

[0515] "Means for displaying a response or playing it as an animation" refers to a display device for visually conveying the generated response to the user, an engine for controlling the animation of an avatar, etc.

[0516] The present invention is a system that analyzes user input, generates humorous responses, and displays them. This system is intended to be installed in an autonomous vehicle, and specific embodiments are described below.

[0517] System Configuration

[0518] 1. A way to receive user input

[0519] The autonomous vehicle is equipped with a microphone that captures voice input from the user, and this voice data includes the user's questions and instructions.

[0520] 2. Voice-to-text conversion methods

[0521] The terminal inside the vehicle is installed with voice recognition software (e.g., Google Speech-to-Text API) that converts the captured voice data into text format.

[0522] 3. Text Analysis and Intent Understanding

[0523] The converted text is sent to a server and analyzed by a natural language processing engine (e.g., Google Cloud Natural Language API). This analysis allows the system to understand the user's intent and generate appropriate answers based on that.

[0524] 4. Selecting and generating humor

[0525] The server uses a generative AI model (such as OpenAI's GPT model) based on the analysis results to generate appropriate humor, enabling friendly dialogue.

[0526] 5. Generating Responses and Animations

[0527] Based on the humor generated, the server generates a response text for the user. At the same time, it generates avatar movement data using a 3D avatar generation engine (e.g., Unreal Engine). This movement data includes information on the avatar's facial expressions and gestures.

[0528] 6. Sending response data

[0529] The generated text response and avatar movement data are sent to the device using a communication module (e.g., WebSocket).

[0530] 7. Displaying the response to the user

[0531] The received data is displayed on the vehicle's display and an avatar animation is played, providing users with a natural and enjoyable interaction experience.

[0532] Example

[0533] For example, if a user asks, "What's the nearest restaurant?", the voice is first captured by a microphone. Speech recognition software then converts the voice into text, which is then sent to the server. The server then analyzes it with a natural language processing engine to understand the intent. Based on the results, the generative AI model generates a humorous response, such as, "The nearest restaurant is just around the corner... but the food has mixed reviews!" The response is displayed alongside an animated 3D avatar, allowing the user to experience the humorous response.

[0534] Prompt Sentence Examples

[0535] The prompt to the generative AI model would be:

[0536] Generate a humorous response about "nearest restaurant."

[0537] This embodiment allows for friendly interaction with the user, providing a fun experience even inside an autonomous vehicle.

[0538] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0539] Step 1:

[0540] A user provides voice input inside an autonomous vehicle. The user's input is captured through a microphone, and voice data is obtained as input, which is then used in subsequent processing steps.

[0541] Step 2:

[0542] The device converts the captured voice data into text data using speech recognition software (e.g., Google Speech-to-Text API). The voice data is converted into acoustic features, and the speech recognition algorithm generates a text string based on the features. The text data is obtained as output.

[0543] Step 3:

[0544] The device sends the converted text data to a server. A natural language processing engine (e.g., Google Cloud Natural Language API) connected to the server receives the text data and begins analyzing it. The input text data is analyzed, and the user's intent and topic are identified as the analysis results.

[0545] Step 4:

[0546] Based on the analysis results, the server generates humor using a generative AI model (e.g., OpenAI's GPT model). Here, the generative AI model is prompted to generate a humorous response about the "nearest restaurant." As an output, the generative AI model generates a humorous response text.

[0547] Step 5:

[0548] The server generates a response text for the user and corresponding animation data for a 3D avatar based on the generated humor text. Using a 3D avatar generation engine (e.g., Unreal Engine), motion data for the avatar to spread its arms, smile, and talk is generated. The response text and animation data are obtained as output.

[0549] Step 6:

[0550] The generated response text and avatar animation data are sent from the server to the device using a communication module (e.g., WebSocket). The device analyzes the received data and prepares it for display.

[0551] Step 7:

[0552] The terminal displays the response text on the vehicle's display and simultaneously plays an animation of the avatar, providing the user with a visual response, which allows the user to enjoy a friendly and enjoyable interaction experience. The output is the text and animation displayed on the display.

[0553] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0554] This invention relates to a system that combines humor and emotion recognition with a conversational agent, enabling friendly interactions with the user and providing appropriate responses depending on the user's emotions. Specific embodiments of this system are described below.

[0555] System Configuration

[0556] 1. A way to receive user input

[0557] The device receives voice or text input from the user. For example, if a user asks "What's the weather like today?", the device's microphone captures the voice.

[0558] 2. A means of converting input from speech to text

[0559] The voice recognition software installed on the device converts the captured voice into text, and the speech "What's the weather like today?" is converted into text data and sent to the server.

[0560] 3. Text Analysis and Intent Understanding

[0561] The server analyzes the received text data to understand the user's intent. Specifically, it uses a natural language processing engine to identify that the question is about the weather.

[0562] 4. Use of Emotion Engine

[0563] The emotion engine installed on the server recognizes the user's emotions from their voice or text data, for example, identifying whether the user is excited or depressed.

[0564] 5. Selecting and generating humor

[0565] The server uses a generative AI model to generate appropriate humor based on the analysis results and the recognition results of the emotion engine. In this case, for the topic "weather," it generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0566] 6. Response and Gesture Generation

[0567] The server generates a response text for the user based on the generated humor. At the same time, it generates control data for a 3D avatar based on the recognition results of the emotion engine, creating a response that includes actions such as spreading arms, smiling, and speaking.

[0568] 7. Sending response data

[0569] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[0570] 8. Displaying the response to the user

[0571] The device displays the received data and plays an animation in which a 3D avatar extends its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[0572] Specific examples

[0573] The user asks, "What's the weather like today?" The device receives the voice and converts it into text using speech recognition software. The converted text is sent to a server where it is analyzed by a natural language processing engine. The server uses a generative AI model to create weather-related humor and determines the response: "The weather is sunny today... I'm not sure, but maybe you should go outside and check!" At the same time, the emotion engine analyzes the user's emotions. If it determines that the user is feeling down, it generates behavior data for the 3D avatar, including more friendly and comforting actions. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[0574] In this way, the present invention can provide a user with a humorous dialogue that is friendly to the user, and can also respond appropriately according to the user's emotions.

[0575] The processing flow will be explained below.

[0576] Step 1:

[0577] The user asks the device verbally, "What's the weather like today?"

[0578] Step 2:

[0579] The device captures the user's voice and temporarily stores it.

[0580] Step 3:

[0581] Speech recognition software installed on the device converts the captured voice into text data.

[0582] Step 4:

[0583] The terminal transmits the converted text data to the server.

[0584] Step 5:

[0585] The server uses a natural language processing engine to analyze the text data it receives and understand the user's intent. In this case, it identifies that the question is about the weather.

[0586] Step 6:

[0587] The server then uses the analysis results to activate the emotion engine and analyze the user's emotional state, for example, identifying whether the user is excited or depressed.

[0588] Step 7:

[0589] Based on the analysis results and the recognition results of the emotion engine, the server uses a generative AI model to select an appropriate humor. In this case, it generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0590] Step 8:

[0591] The server generates a response text for the user based on the generated humor, and adjusts the response content according to the recognition results of the emotion engine.

[0592] Step 9:

[0593] At the same time, the server generates control data for the 3D avatar and configures it to perform movements that reflect the corresponding emotions (e.g., speaking with a smile and spreading one's arms).

[0594] Step 10:

[0595] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[0596] Step 11:

[0597] The device displays the received data and plays an animation in which a 3D avatar opens its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[0598] This specific processing flow allows the user to receive friendly responses from the conversational agent that reflect humor and emotion.

[0599] Example 2

[0600] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0601] Conventional conversational agent systems often respond to user input mechanically and without considering emotions, making interactions with users unfriendly. Furthermore, there is a lack of systems that combine humorous responses and behaviors that respond to the user's emotions, making it impossible to enrich the user experience. This has led to issues such as a decline in the quality of interaction with users and low satisfaction.

[0602] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0603] In this invention, the server includes means for receiving a user's input, means for analyzing the user's input, means for recognizing the analysis result and emotion, means for generating humor based on the analysis result and the emotion recognition, and means for generating a response and gesture based on the generated humor, thereby enabling a response including appropriate humor and action according to the user's emotion.

[0604] "Means for receiving user input" refers collectively to any interface or device that captures information when a user provides it to a system in the form of voice, text, or other information.

[0605] "Means of analysis" is a general term for algorithms and software that process input data received from users and extract specific intent or meaning.

[0606] "Emotion recognition means" is a general term for algorithms and engines that analyze and identify a user's emotional state from their voice or text data.

[0607] "Humor generation means" is a general term for algorithms and models that generate humorous content for users based on the results of analysis and emotion recognition.

[0608] "Means for generating responses and gestures" refers collectively to the algorithms and engines that control and generate response text and corresponding actions and gestures to the user based on the generated humor.

[0609] "Means for receiving" is a general term for interfaces and devices for receiving input from users and data from servers.

[0610] "Displaying means" is a general term for devices and software that visually or audibly present information to the user, such as generated response text or gestures.

[0611] This invention is a system that combines humor and emotion recognition to realize friendly dialogue with users in a conversational agent. The system receives voice or text input from the user, analyzes the input, and generates appropriate humorous responses and emotionally appropriate actions to present to the user.

[0612] System Configuration

[0613] 1. A way to receive user input

[0614] The device receives voice or text input from the user. For example, when a user asks, "What's the weather like today?", the device's microphone captures the audio. Required hardware includes a microphone, speaker, and display.

[0615] 2. A means of converting input from speech to text

[0616] Speech recognition software (e.g., Google Cloud Speech-to-Text) installed on the device converts the captured voice data into text data, which becomes "What's the weather like today?" and is sent to the server.

[0617] 3. Text Analysis and Intent Understanding

[0618] The server analyzes the received text data and uses a natural language processing engine (e.g., Google Natural Language API) to understand the user's intent, thereby identifying that the user's question is about the weather.

[0619] 4. Use of Emotion Engine

[0620] The emotion engine (e.g., IBM Watson Tone Analyzer) installed on the server recognizes emotions from the user's text data, and identifies whether the user is excited or depressed.

[0621] 5. Selecting and generating humor

[0622] Based on the analysis and emotion recognition results, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate humor, such as "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0623] 6. Response and Gesture Generation

[0624] The server generates a response text based on the generated humor, and simultaneously generates motion data for a 3D avatar based on the recognition results of the emotion engine, including, for example, a motion in which the avatar smiles and spreads its arms.

[0625] 7. Sending response data

[0626] The server transmits the generated response text and the motion data of the 3D avatar to the terminal.

[0627] 8. Displaying the response to the user

[0628] The device displays the received data and plays an animation in which a 3D avatar extends its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[0629] Specific examples

[0630] The user asks, "What's the weather like today?" The device receives the voice and converts it into text using speech recognition software. The converted text is sent to a server where it is analyzed by a natural language processing engine. The server uses a generative AI model to create a weather-related joke and determines the response, "The weather is sunny today... I don't know, but maybe you should go outside and check!" At the same time, the emotion engine analyzes the user's emotions, and if it determines that the user is feeling down, the 3D avatar will take comforting actions. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[0631] Prompt Sentence Examples

[0632] "Generate humorous responses to users asking about today's weather, and provide responses with comforting 3D avatar behavior if the user is feeling down."

[0633] By feeding this prompt into a generative AI model, the system can generate a response for the user that includes appropriate humor and actions.

[0634] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0635] Step 1:

[0636] User inputs by voice.

[0637] The user asks the device, "What's the weather like today?"

[0638] Input: User's voice data

[0639] Output: Audio data

[0640] Step 2:

[0641] The device captures the audio.

[0642] The device's microphone captures the user's voice and stores it as digital audio data.

[0643] Input: Audio data

[0644] Output: Digital audio data

[0645] Step 3:

[0646] Text conversion using voice recognition software installed on the device.

[0647] Speech recognition software (e.g., Google Cloud Speech-to-Text) converts digital voice data into text data.

[0648] Input: Digital audio data

[0649] Output: Text data "What's the weather like today?"

[0650] Step 4:

[0651] The device sends text data to the server.

[0652] The terminal sends the converted text data to the server, typically requiring an HTTP request.

[0653] Input: Text data

[0654] Output: HTTP request to the server

[0655] Step 5:

[0656] The server receives and analyzes the text data.

[0657] The server receives the HTTP request and analyzes the text data using a natural language processing engine (e.g., Google Natural Language API). This analysis helps to understand the user's intent and identifies the question as being about the weather.

[0658] Input: Text data

[0659] Output: Text analysis results

[0660] Step 6:

[0661] The server analyzes emotions using an emotion engine.

[0662] The emotion engine (e.g., IBM Watson Tone Analyzer) installed on the server recognizes the user's emotion from the text data, for example, identifying the user's emotional state, such as whether they are excited or depressed.

[0663] Input: Text data

[0664] Output: Emotion analysis results

[0665] Step 7:

[0666] Server generated humor.

[0667] Based on the results of text analysis and sentiment analysis, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate humor. It generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0668] Input: Text analysis results, sentiment analysis results

[0669] Output: Humorous response text

[0670] Step 8:

[0671] The server generates the response text and action data.

[0672] The server creates a response text based on the generated humor. At the same time, it generates motion data for the 3D avatar based on the analysis results of the emotion engine. For example, it creates a motion for the avatar to speak with its arms outstretched and smiling.

[0673] Input: Humorous response text, sentiment analysis results

[0674] Output: Response text, action data

[0675] Step 9:

[0676] The server sends the response data to the terminal.

[0677] The server sends the generated response text and the 3D avatar's movement data to the terminal.

[0678] Input: Response text, action data

[0679] Output: HTTP response to the device

[0680] Step 10:

[0681] The device responds.

[0682] The device displays the received data and plays an animation in which a 3D avatar holds out its arms to the user and says with a smile, "The weather's sunny today... I'm not sure, but maybe you should go outside and check!"

[0683] Input: Response text, action data

[0684] Output: Animation and audio response display

[0685] (Application example 2)

[0686] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0687] Conventional conversational agent systems have difficulty responding flexibly to the user's emotions and circumstances, and have been unable to sufficiently improve user satisfaction. Furthermore, generating humorous responses requires taking into account the user's current emotional state, but the lack of a function to dynamically adjust the humor accordingly can sometimes result in an inconsistent user experience.

[0688] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a user's input, means for analyzing the user's input, means for generating humor based on the analysis result, means for generating and displaying a response based on the generated humor and outputting it as voice, means for performing emotion recognition, and means for adjusting the humor in accordance with the emotion recognition result. This makes it possible to provide an appropriate and friendly response, mixed with humor, that is in accordance with the user's emotional state.

[0689] A "means for receiving user input" is a device or interface for capturing speech or text generated by a user.

[0690] "Means for analyzing user input" means software or a system that interprets captured voice or text data and analyzes its intent and content.

[0691] "Means for generating humor based on the analysis results" refers to algorithms or generative AI models that create appropriate humor based on the analyzed input data.

[0692] The "means for generating and displaying a response based on the generated humor and outputting it as audio" is a mechanism for displaying a response containing the generated humor as text and playing it back as audio using speech synthesis technology.

[0693] "Means for emotion recognition" means software or emotion engines that analyze and identify a user's emotional state from their voice or text.

[0694] The "means for adjusting humor according to the results of emotion recognition" is a mechanism for optimizing the generated humor and selecting an appropriate response based on the user's emotional state.

[0695] This invention is an interactive agent system that provides friendly dialogue with users and responds appropriately based on the user's emotions. This system consists of a device that receives the user's voice and text input, a server that analyzes the input data, and a terminal that generates and displays humorous responses and outputs them as voice.

[0696] Hardware and software used

[0697] Hardware: The microphone and speaker on your smartphone.

[0698] Software: Speech recognition library (speech_recognition), text-to-speech engine (pyttsx3), emotion recognition API, generative AI model (OpenAI GPT-3).

[0699] Process Overview

[0700] 1. Voice input acquisition:

[0701] The user inputs voice through the smartphone, and the smartphone's microphone captures the user's voice.

[0702] 2. Speech Recognition:

[0703] Smartphones use speech recognition software (speech_recognition) to convert captured speech into text, for example, Google's speech recognition API.

[0704] 3. Emotion recognition:

[0705] The converted text data is sent to the server, which then analyzes the user's emotions using an emotion recognition API to assess the user's emotional state.

[0706] 4. Humorous response generation:

[0707] The server uses a generative AI model (OpenAI GPT-3) to generate humorous responses to text data, using prompts to instruct the AI.

[0708] Example prompt sentence:

[0709] User: I hate the long traffic jams.

[0710] AI: Provide a humorous response.

[0711] 5. Audio output of response:

[0712] The server sends the generated humorous response to the smartphone, which uses the smartphone's text-to-speech engine (pyttsx3) to output the response aloud.

[0713] Specific examples

[0714] For example, if a user says "I hate being stuck in traffic jams for so long" in a self-driving vehicle, the smartphone's microphone picks up the speech. The speech recognition software then converts the speech into text, "I hate being stuck in traffic jams for so long," and sends it to the server. The server then uses an emotion recognition API to determine that the user is "angry." Based on this information, the generative AI model (OpenAI GPT-3) generates the following humorous response: "It's good to relax for a while," and the response is output via voice via the smartphone.

[0715] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0716] Step 1:

[0717] The user inputs voice using the microphone on the smartphone, and the user's voice data is captured by the microphone and sent to the device.

[0718] Step 2:

[0719] The device uses a speech recognition library (speech_recognition) to convert the captured voice data into text data. This process converts the user's voice input into text format. The input is voice data and the output is text data.

[0720] Step 3:

[0721] The converted text data is sent from the terminal to the server. The server receives this text data and proceeds to the next step. The input is text data, and the output is sending the data to the server.

[0722] Step 4:

[0723] The server uses an emotion recognition API to analyze the user's emotions from the received text data. Specifically, it extracts emotion tags (e.g., angry, sad, happy, etc.) from the text content. Through this process, the input is text data and the output is emotion tags.

[0724] Step 5:

[0725] The server uses a generative AI model (OpenAI GPT-3) to generate an appropriate humorous response based on the emotion tag and text data. Here, a prompt is used to instruct the AI. Specifically, a prompt such as "User: I'm tired of the long traffic jam. AI: Provide a humorous response" is created and passed to the AI ​​model. This process results in emotion tag and text data as input, and a humorous response as output.

[0726] Step 6:

[0727] Based on the generated humorous response and emotion tag, the response is adjusted as needed. For example, if the emotion tag is "angry," an additional phrase such as "It's good to relax for a while" is inserted into the humorous response. With this process, the input is a humorous response and emotion tag, and the output is the adjusted humorous response.

[0728] Step 7:

[0729] The server sends the adjusted humor response to the terminal. The input is the adjusted humor response, and the output is the transmission of data to the terminal.

[0730] Step 8:

[0731] The device converts the received tailored humor response into speech using a text-to-speech engine (pyttsx3) and outputs it to the user through the speaker. As a result of this process, the input is the tailored humor response and the output is the audio response.

[0732] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0733] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0734] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0735] [Third embodiment]

[0736] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0737] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0738] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0739] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0740] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0741] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0742] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0743] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0744] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0745] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0746] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0747] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0748] The present invention is a system for adding humor to an interactive agent, and for realizing friendly dialogue with the user. A specific embodiment of this system will be described below.

[0749] System Configuration

[0750] 1. A way to receive user input

[0751] The device receives voice or text input from the user. For example, if a user asks "What's the weather like today?", the device's microphone captures the voice.

[0752] 2. A means of converting input from speech to text

[0753] The voice recognition software installed on the device converts the captured voice into text, and the speech "What's the weather like today?" is converted into text data and sent to the server.

[0754] 3. Text Analysis and Intent Understanding

[0755] The server analyzes the received text data to understand the user's intent. Specifically, it uses a natural language processing engine to identify that the question is about the weather.

[0756] 4. Selecting and generating humor

[0757] Based on the analysis results, the server uses a generative AI model to generate appropriate humor. For example, for the topic "weather," it generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0758] 5. Response and Gesture Generation

[0759] The server generates a response text for the user based on the generated humor, and simultaneously generates control data for the 3D avatar, creating a response that includes actions such as opening arms, smiling, and speaking.

[0760] 6. Sending response data

[0761] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[0762] 7. Displaying the response to the user

[0763] The device displays the received data and plays an animation in which a 3D avatar holds out its arms to the user and says with a smile, "The weather's sunny today... I'm not sure, but maybe you should go outside and check!"

[0764] Specific examples

[0765] The user asks, "What's the weather like today?" The device captures the voice and converts it into text using speech recognition software. The converted text is sent to a server where it is analyzed using a natural language processing engine. The server uses a generative AI model to create weather-related humor and determines the response: "The weather is sunny today... I'm not sure, but maybe you should go outside and check!" At the same time, movement data is generated for the 3D avatar to spread its arms and speak. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[0766] In this way, the present invention can provide users with friendly, humorous interactions.

[0767] The processing flow will be explained below.

[0768] Step 1:

[0769] The user asks the terminal by voice, "What's the weather like today?"

[0770] Step 2:

[0771] The device temporarily stores the audio received.

[0772] Step 3:

[0773] Speech recognition software installed on the device converts the saved voice data into text data.

[0774] Step 4:

[0775] The terminal transmits the converted text data to the server.

[0776] Step 5:

[0777] The server uses a natural language processing engine to analyze the text data it receives and understand the user's intent. In this case, it identifies that the question is about the weather.

[0778] Step 6:

[0779] The server uses a generative AI model based on the analysis to generate appropriate humor, in this case, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0780] Step 7:

[0781] The server generates a response text for the user based on the generated humor.

[0782] Step 8:

[0783] At the same time, the server generates control data for the 3D avatar, creating the avatar's movements of spreading its arms, smiling, and talking.

[0784] Step 9:

[0785] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[0786] Step 10:

[0787] The device displays the received data and plays an animation in which a 3D avatar opens its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[0788] This specific processing flow allows the user to receive friendly responses, including humor, from the conversational agent.

[0789] Example 1

[0790] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0791] In conversational agents, it is necessary to add humor to realize friendly dialogue with users. However, conventional systems have had difficulty generating appropriate and natural humor in response to user input. Furthermore, converting speech to text and generating and displaying actions, including gestures, have also been issues.

[0792] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0793] In this invention, the server includes means for receiving a user input, means for converting the user input from speech to text using speech recognition software, means for analyzing the user input, means for generating a prompt sentence based on the analysis result and generating humor using a generative AI model, means for generating a response text and hand gestures based on the generated humor, and means for displaying the response text and hand gestures to the user. This allows for the generation of appropriate and natural humor in response to the user input, and the generation and display of actions including speech-to-text conversion and hand gestures.

[0794] "User" refers to any individual or end user who uses this System.

[0795] "Input" refers to the provision of information in the form of speech or text by a user to a system.

[0796] "Speech recognition software" refers to a program or algorithm used to analyze captured voice data and convert it into text data.

[0797] "Text" refers to character strings of data that correspond to a user's spoken input, as converted by speech recognition software.

[0798] "Analysis" refers to the process of understanding the user's intent based on the received text data.

[0799] A "prompt" is an instruction given to a generative AI model to generate humor based on user input.

[0800] "Generative AI model" refers to an artificial intelligence model that generates humor based on prompt text.

[0801] "Humor" refers to the friendly, entertaining responses to user input generated by the system.

[0802] "Responsive Text" refers to a humorous text response to a user's input that is created by a generative AI model.

[0803] "Gestures" refers to the physical expressions, such as gestures and facial expressions, that a 3D avatar makes toward the user.

[0804] "Display" refers to projecting text and 3D avatar movements onto the device screen to provide a visual experience to the user.

[0805] "Terminal" means the electronic device through which a User accesses, enters input into, and receives responses to this System.

[0806] The present invention is a system for adding humor to an interactive agent, and for realizing friendly dialogue with the user. A specific embodiment of this system will be described below.

[0807] 1. System Configuration

[0808] The system consists of a device for receiving user input, speech recognition software for converting the user's input from speech to text, and a server for analyzing the received text data and generating appropriate humor. Specifically, the device includes a microphone and speech recognition software (e.g., Google Speech-to-Text), while the server is equipped with a natural language processing engine (e.g., SpaCy) and a generative AI model (e.g., OpenAI GPT-3).

[0809] 2. Processing Flow

[0810] The user speaks to the device, asking, "What's the weather like today?" The device's microphone captures the voice, and speech recognition software converts the voice into text data. The converted text data is sent to the server in the form of "What's the weather like today?"

[0811] The server analyzes the received text data using a natural language processing engine and identifies it as a question about the weather. Based on the analysis results, it inputs the following prompt sentence into the generative AI model: "The user is asking, 'What's the weather like today?' Please provide a humorous response to this question."

[0812] Based on this prompt, the generative AI model generates a humorous response: "I'm not sure if the weather is sunny today, but maybe I should go outside and check it out!" The server also generates motion data for controlling the 3D avatar based on this response text. The generated control data includes the motion of the 3D avatar speaking with its arms outstretched and smiling.

[0813] 3. View the response

[0814] The generated response text and action data are sent from the server to the device. The device then displays a 3D avatar on the screen based on the received data, and plays an animation of the avatar spreading its arms and smiling, saying, "I don't know if the weather is sunny today... but maybe I should go outside and check!" In this way, the user can receive humorous responses along with the 3D avatar's actions.

[0815] Specific examples

[0816] For example, if a user asks "What's the weather like today?", the device captures the speech and uses speech recognition software to convert it into text. The converted text is sent to a server, which then analyzes it using a natural language processing engine. The server then uses a generative AI model to create weather-related humor and determines a response such as "The weather is sunny today... I'm not sure, but maybe you should go outside and check!" At the same time, movement data is also generated for the 3D avatar to spread its arms and speak. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[0817] An example of the prompt above is: "The user asks, 'What's the weather like today?' Please respond to this question with humor."

[0818] This invention makes it possible to generate appropriate and natural humor in response to user input, convert speech to text, and generate and display actions including gestures.

[0819] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0820] Step 1:

[0821] The user speaks, "What's the weather like today?"

[0822] The device captures this audio with a microphone.

[0823] Input data: User's voice.

[0824] Output data: Audio data.

[0825] Step 2:

[0826] The device uses voice recognition software (e.g., Google Speech-to-Text) to convert the captured voice data into text data.

[0827] Input data: Captured audio data.

[0828] Output data: Text data "What's the weather like today?"

[0829] Step 3:

[0830] The terminal transmits the converted text data to the server.

[0831] Input data: Text data "What's the weather like today?"

[0832] Output data: Sending text data to the server.

[0833] Step 4:

[0834] The server analyzes the received text data using a natural language processing engine (e.g., SpaCy).

[0835] Input data: Text data "What's the weather like today?"

[0836] Data processing: Analyze the received text data and identify that it is a question about the weather.

[0837] Output: Parsed intent ("Question about the weather").

[0838] Step 5:

[0839] The server generates a prompt based on the analysis results and sends it to the generative AI model (e.g., OpenAI GPT-3). The specific prompt is, "The user is asking, 'What's the weather like today?' Please respond to this question with humor."

[0840] Input data: Parsed intent ("Question about the weather").

[0841] Data processing: Prompt sentence generation.

[0842] Output data: The prompt sentence to be passed to the generative AI model.

[0843] Step 6:

[0844] The generative AI model generates an appropriate humorous response based on the prompt sentence.

[0845] Input data: prompt statement.

[0846] Data operations: Operations that analyze prompt sentences and generate humorous responses.

[0847] Output: A humorous response: "The weather is sunny today... I don't know, but maybe I should go outside and check!"

[0848] Step 7:

[0849] The server generates motion data for controlling the 3D avatar based on the humorous response text, including the motion of the avatar spreading its arms and speaking.

[0850] Input data: Humor response text.

[0851] Data processing: Generation of 3D avatar movements.

[0852] Output data: response text and 3D avatar movement data.

[0853] Step 8:

[0854] The server transmits the generated response text and the motion data of the 3D avatar to the terminal.

[0855] Input data: response text and 3D avatar movement data.

[0856] Output data: Data sent to the device.

[0857] Step 9:

[0858] Based on the data received, the device displays a 3D avatar on the screen and plays an animation in which the avatar spreads its arms and says with a smile, "I don't know if the weather will be sunny today... but maybe I should go outside and check!"

[0859] Input data: response text and 3D avatar movement data.

[0860] Data calculation: 3D avatar movement playback.

[0861] Output data: Responses and avatar behaviors displayed to the user.

[0862] (Application example 1)

[0863] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0864] Current conversational agent systems face challenges in incorporating humor appropriately into dialogue with users, making it difficult to achieve friendly, natural conversations. Furthermore, when agents interact with drivers and passengers in autonomous vehicles, they are unable to generate humorous responses and provide enjoyment. There is a need for conversational agent systems that incorporate humor to solve these problems and improve the user experience.

[0865] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0866] In this invention, the server includes means for receiving user input, means for converting the user input from speech to text, means for analyzing the text to understand the user's intent, means for generating humor based on the analysis results using a generative AI model, means for generating a response based on the generated humor, and means for displaying the response or playing it as an animation, thereby realizing a friendly dialogue with the user and providing a fun experience even inside an autonomous vehicle.

[0867] "Means for receiving user input" refers to devices or software for receiving instructions or questions from users, such as voice input or text input.

[0868] "Speech-to-text means" means speech recognition software or algorithms used to convert spoken input into text form.

[0869] "Means for analyzing text and understanding user intent" refers to a natural language processing engine that analyzes input text data and identifies the user's intent and purpose from its content.

[0870] "Means for generating humor using a generative AI model" refers to a method that uses a generative AI model (e.g., a GPT model) to generate relevant humor based on the analysis results.

[0871] "Means for generating responses based on generated humor" refers to software or algorithms that use the generated humor to create responses to users.

[0872] "Means for displaying a response or playing it as an animation" refers to a display device for visually conveying the generated response to the user, an engine for controlling the animation of an avatar, etc.

[0873] The present invention is a system that analyzes user input, generates humorous responses, and displays them. This system is intended to be installed in an autonomous vehicle, and specific embodiments are described below.

[0874] System Configuration

[0875] 1. A way to receive user input

[0876] The autonomous vehicle is equipped with a microphone that captures voice input from the user, and this voice data includes the user's questions and instructions.

[0877] 2. Voice-to-text conversion methods

[0878] The terminal inside the vehicle is installed with voice recognition software (e.g., Google Speech-to-Text API) that converts the captured voice data into text format.

[0879] 3. Text Analysis and Intent Understanding

[0880] The converted text is sent to a server and analyzed by a natural language processing engine (e.g., Google Cloud Natural Language API). This analysis allows the system to understand the user's intent and generate appropriate answers based on that.

[0881] 4. Selecting and generating humor

[0882] The server uses a generative AI model (such as OpenAI's GPT model) based on the analysis results to generate appropriate humor, enabling friendly dialogue.

[0883] 5. Generating Responses and Animations

[0884] Based on the humor generated, the server generates a response text for the user. At the same time, it generates avatar movement data using a 3D avatar generation engine (e.g., Unreal Engine). This movement data includes information on the avatar's facial expressions and gestures.

[0885] 6. Sending response data

[0886] The generated text response and avatar movement data are sent to the device using a communication module (e.g., WebSocket).

[0887] 7. Displaying the response to the user

[0888] The received data is displayed on the vehicle's display and an avatar animation is played, providing users with a natural and enjoyable interaction experience.

[0889] Example

[0890] For example, if a user asks, "What's the nearest restaurant?", the voice is first captured by a microphone. Speech recognition software then converts the voice into text, which is then sent to the server. The server then analyzes it with a natural language processing engine to understand the intent. Based on the results, the generative AI model generates a humorous response, such as, "The nearest restaurant is just around the corner... but the food has mixed reviews!" The response is displayed alongside an animated 3D avatar, allowing the user to experience the humorous response.

[0891] Prompt Sentence Examples

[0892] The prompt to the generative AI model would be:

[0893] Generate a humorous response about "nearest restaurant."

[0894] This embodiment allows for friendly interaction with the user, providing a fun experience even inside an autonomous vehicle.

[0895] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0896] Step 1:

[0897] A user provides voice input inside an autonomous vehicle. The user's input is captured through a microphone, and voice data is obtained as input, which is then used in subsequent processing steps.

[0898] Step 2:

[0899] The device converts the captured voice data into text data using speech recognition software (e.g., Google Speech-to-Text API). The voice data is converted into acoustic features, and the speech recognition algorithm generates a text string based on the features. The text data is obtained as output.

[0900] Step 3:

[0901] The device sends the converted text data to a server. A natural language processing engine (e.g., Google Cloud Natural Language API) connected to the server receives the text data and begins analyzing it. The input text data is analyzed, and the user's intent and topic are identified as the analysis results.

[0902] Step 4:

[0903] Based on the analysis results, the server generates humor using a generative AI model (e.g., OpenAI's GPT model). Here, the generative AI model is prompted to generate a humorous response about the "nearest restaurant." As an output, the generative AI model generates a humorous response text.

[0904] Step 5:

[0905] The server generates a response text for the user and corresponding animation data for a 3D avatar based on the generated humor text. Using a 3D avatar generation engine (e.g., Unreal Engine), motion data for the avatar to spread its arms, smile, and talk is generated. The response text and animation data are obtained as output.

[0906] Step 6:

[0907] The generated response text and avatar animation data are sent from the server to the device using a communication module (e.g., WebSocket). The device analyzes the received data and prepares it for display.

[0908] Step 7:

[0909] The terminal displays the response text on the vehicle's display and simultaneously plays an animation of the avatar, providing the user with a visual response, which allows the user to enjoy a friendly and enjoyable interaction experience. The output is the text and animation displayed on the display.

[0910] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0911] This invention relates to a system that combines humor and emotion recognition with a conversational agent, enabling friendly interactions with the user and providing appropriate responses depending on the user's emotions. Specific embodiments of this system are described below.

[0912] System Configuration

[0913] 1. A way to receive user input

[0914] The device receives voice or text input from the user. For example, if a user asks "What's the weather like today?", the device's microphone captures the voice.

[0915] 2. A means of converting input from speech to text

[0916] The voice recognition software installed on the device converts the captured voice into text, and the speech "What's the weather like today?" is converted into text data and sent to the server.

[0917] 3. Text Analysis and Intent Understanding

[0918] The server analyzes the received text data to understand the user's intent. Specifically, it uses a natural language processing engine to identify that the question is about the weather.

[0919] 4. Use of Emotion Engine

[0920] The emotion engine installed on the server recognizes the user's emotions from their voice or text data, for example, identifying whether the user is excited or depressed.

[0921] 5. Selecting and generating humor

[0922] The server uses a generative AI model to generate appropriate humor based on the analysis results and the recognition results of the emotion engine. In this case, for the topic "weather," it generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0923] 6. Response and Gesture Generation

[0924] The server generates a response text for the user based on the generated humor. At the same time, it generates control data for a 3D avatar based on the recognition results of the emotion engine, creating a response that includes actions such as spreading arms, smiling, and speaking.

[0925] 7. Sending response data

[0926] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[0927] 8. Displaying the response to the user

[0928] The device displays the received data and plays an animation in which a 3D avatar extends its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[0929] Specific examples

[0930] The user asks, "What's the weather like today?" The device receives the voice and converts it into text using speech recognition software. The converted text is sent to a server where it is analyzed by a natural language processing engine. The server uses a generative AI model to create weather-related humor and determines the response: "The weather is sunny today... I'm not sure, but maybe you should go outside and check!" At the same time, the emotion engine analyzes the user's emotions. If it determines that the user is feeling down, it generates behavior data for the 3D avatar, including more friendly and comforting actions. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[0931] In this way, the present invention can provide a user with a humorous dialogue that is friendly to the user, and can also respond appropriately according to the user's emotions.

[0932] The processing flow will be explained below.

[0933] Step 1:

[0934] The user asks the device verbally, "What's the weather like today?"

[0935] Step 2:

[0936] The device captures the user's voice and temporarily stores it.

[0937] Step 3:

[0938] Speech recognition software installed on the device converts the captured voice into text data.

[0939] Step 4:

[0940] The terminal transmits the converted text data to the server.

[0941] Step 5:

[0942] The server uses a natural language processing engine to analyze the text data it receives and understand the user's intent. In this case, it identifies that the question is about the weather.

[0943] Step 6:

[0944] The server then uses the analysis results to activate the emotion engine and analyze the user's emotional state, for example, identifying whether the user is excited or depressed.

[0945] Step 7:

[0946] Based on the analysis results and the recognition results of the emotion engine, the server uses a generative AI model to select an appropriate humor. In this case, it generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0947] Step 8:

[0948] The server generates a response text for the user based on the generated humor, and adjusts the response content according to the recognition results of the emotion engine.

[0949] Step 9:

[0950] At the same time, the server generates control data for the 3D avatar and configures it to perform movements that reflect the corresponding emotions (e.g., speaking with a smile and spreading one's arms).

[0951] Step 10:

[0952] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[0953] Step 11:

[0954] The device displays the received data and plays an animation in which a 3D avatar opens its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[0955] This specific processing flow allows the user to receive friendly responses from the conversational agent that reflect humor and emotion.

[0956] Example 2

[0957] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0958] Conventional conversational agent systems often respond to user input mechanically and without considering emotions, making interactions with users unfriendly. Furthermore, there is a lack of systems that combine humorous responses and behaviors that respond to the user's emotions, making it impossible to enrich the user experience. This has led to issues such as a decline in the quality of interaction with users and low satisfaction.

[0959] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0960] In this invention, the server includes means for receiving a user's input, means for analyzing the user's input, means for recognizing the analysis result and emotion, means for generating humor based on the analysis result and the emotion recognition, and means for generating a response and gesture based on the generated humor, thereby enabling a response including appropriate humor and action according to the user's emotion.

[0961] "Means for receiving user input" refers collectively to any interface or device that captures information when a user provides it to a system in the form of voice, text, or other information.

[0962] "Means of analysis" is a general term for algorithms and software that process input data received from users and extract specific intent or meaning.

[0963] "Emotion recognition means" is a general term for algorithms and engines that analyze and identify a user's emotional state from their voice or text data.

[0964] "Humor generation means" is a general term for algorithms and models that generate humorous content for users based on the results of analysis and emotion recognition.

[0965] "Means for generating responses and gestures" refers collectively to the algorithms and engines that control and generate response text and corresponding actions and gestures to the user based on the generated humor.

[0966] "Means for receiving" is a general term for interfaces and devices for receiving input from users and data from servers.

[0967] "Displaying means" is a general term for devices and software that visually or audibly present information to the user, such as generated response text or gestures.

[0968] This invention is a system that combines humor and emotion recognition to realize friendly dialogue with users in a conversational agent. The system receives voice or text input from the user, analyzes the input, and generates appropriate humorous responses and emotionally appropriate actions to present to the user.

[0969] System Configuration

[0970] 1. A way to receive user input

[0971] The device receives voice or text input from the user. For example, when a user asks, "What's the weather like today?", the device's microphone captures the audio. Required hardware includes a microphone, speaker, and display.

[0972] 2. A means of converting input from speech to text

[0973] Speech recognition software (e.g., Google Cloud Speech-to-Text) installed on the device converts the captured voice data into text data, which becomes "What's the weather like today?" and is sent to the server.

[0974] 3. Text Analysis and Intent Understanding

[0975] The server analyzes the received text data and uses a natural language processing engine (e.g., Google Natural Language API) to understand the user's intent, thereby identifying that the user's question is about the weather.

[0976] 4. Use of Emotion Engine

[0977] The emotion engine (e.g., IBM Watson Tone Analyzer) installed on the server recognizes emotions from the user's text data, and identifies whether the user is excited or depressed.

[0978] 5. Selecting and generating humor

[0979] Based on the analysis and emotion recognition results, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate humor, such as "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[0980] 6. Response and Gesture Generation

[0981] The server generates a response text based on the generated humor, and simultaneously generates motion data for a 3D avatar based on the recognition results of the emotion engine, including, for example, a motion in which the avatar smiles and spreads its arms.

[0982] 7. Sending response data

[0983] The server transmits the generated response text and the motion data of the 3D avatar to the terminal.

[0984] 8. Displaying the response to the user

[0985] The device displays the received data and plays an animation in which a 3D avatar extends its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[0986] Specific examples

[0987] The user asks, "What's the weather like today?" The device receives the voice and converts it into text using speech recognition software. The converted text is sent to a server where it is analyzed by a natural language processing engine. The server uses a generative AI model to create a weather-related joke and determines the response, "The weather is sunny today... I don't know, but maybe you should go outside and check!" At the same time, the emotion engine analyzes the user's emotions, and if it determines that the user is feeling down, the 3D avatar will take comforting actions. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[0988] Prompt Sentence Examples

[0989] "Generate humorous responses to users asking about today's weather, and provide responses with comforting 3D avatar behavior if the user is feeling down."

[0990] By feeding this prompt into a generative AI model, the system can generate a response for the user that includes appropriate humor and actions.

[0991] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0992] Step 1:

[0993] User inputs by voice.

[0994] The user asks the device, "What's the weather like today?"

[0995] Input: User's voice data

[0996] Output: Audio data

[0997] Step 2:

[0998] The device captures the audio.

[0999] The device's microphone captures the user's voice and stores it as digital audio data.

[1000] Input: Audio data

[1001] Output: Digital audio data

[1002] Step 3:

[1003] Text conversion using voice recognition software installed on the device.

[1004] Speech recognition software (e.g., Google Cloud Speech-to-Text) converts digital voice data into text data.

[1005] Input: Digital audio data

[1006] Output: Text data "What's the weather like today?"

[1007] Step 4:

[1008] The device sends text data to the server.

[1009] The terminal sends the converted text data to the server, typically requiring an HTTP request.

[1010] Input: Text data

[1011] Output: HTTP request to the server

[1012] Step 5:

[1013] The server receives and analyzes the text data.

[1014] The server receives the HTTP request and analyzes the text data using a natural language processing engine (e.g., Google Natural Language API). This analysis helps to understand the user's intent and identifies the question as being about the weather.

[1015] Input: Text data

[1016] Output: Text analysis results

[1017] Step 6:

[1018] The server analyzes emotions using an emotion engine.

[1019] The emotion engine (e.g., IBM Watson Tone Analyzer) installed on the server recognizes the user's emotion from the text data, for example, identifying the user's emotional state, such as whether they are excited or depressed.

[1020] Input: Text data

[1021] Output: Emotion analysis results

[1022] Step 7:

[1023] Server generated humor.

[1024] Based on the results of text analysis and sentiment analysis, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate humor. It generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[1025] Input: Text analysis results, sentiment analysis results

[1026] Output: Humorous response text

[1027] Step 8:

[1028] The server generates the response text and action data.

[1029] The server creates a response text based on the generated humor. At the same time, it generates motion data for the 3D avatar based on the analysis results of the emotion engine. For example, it creates a motion for the avatar to speak with its arms outstretched and smiling.

[1030] Input: Humorous response text, sentiment analysis results

[1031] Output: Response text, action data

[1032] Step 9:

[1033] The server sends the response data to the terminal.

[1034] The server sends the generated response text and the 3D avatar's movement data to the terminal.

[1035] Input: Response text, action data

[1036] Output: HTTP response to the device

[1037] Step 10:

[1038] The device responds.

[1039] The device displays the received data and plays an animation in which a 3D avatar holds out its arms to the user and says with a smile, "The weather's sunny today... I'm not sure, but maybe you should go outside and check!"

[1040] Input: Response text, action data

[1041] Output: Animation and audio response display

[1042] (Application example 2)

[1043] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1044] Conventional conversational agent systems have difficulty responding flexibly to the user's emotions and circumstances, and have been unable to sufficiently improve user satisfaction. Furthermore, generating humorous responses requires taking into account the user's current emotional state, but the lack of a function to dynamically adjust the humor accordingly can sometimes result in an inconsistent user experience.

[1045] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a user's input, means for analyzing the user's input, means for generating humor based on the analysis result, means for generating and displaying a response based on the generated humor and outputting it as voice, means for performing emotion recognition, and means for adjusting the humor in accordance with the emotion recognition result. This makes it possible to provide an appropriate and friendly response, mixed with humor, that is in accordance with the user's emotional state.

[1046] A "means for receiving user input" is a device or interface for capturing speech or text generated by a user.

[1047] "Means for analyzing user input" means software or a system that interprets captured voice or text data and analyzes its intent and content.

[1048] "Means for generating humor based on the analysis results" refers to algorithms or generative AI models that create appropriate humor based on the analyzed input data.

[1049] The "means for generating and displaying a response based on the generated humor and outputting it as audio" is a mechanism for displaying a response containing the generated humor as text and playing it back as audio using speech synthesis technology.

[1050] "Means for emotion recognition" means software or emotion engines that analyze and identify a user's emotional state from their voice or text.

[1051] The "means for adjusting humor according to the results of emotion recognition" is a mechanism for optimizing the generated humor and selecting an appropriate response based on the user's emotional state.

[1052] This invention is an interactive agent system that provides friendly dialogue with users and responds appropriately based on the user's emotions. This system consists of a device that receives the user's voice and text input, a server that analyzes the input data, and a terminal that generates and displays humorous responses and outputs them as voice.

[1053] Hardware and software used

[1054] Hardware: The microphone and speaker on your smartphone.

[1055] Software: Speech recognition library (speech_recognition), text-to-speech engine (pyttsx3), emotion recognition API, generative AI model (OpenAI GPT-3).

[1056] Process Overview

[1057] 1. Voice input acquisition:

[1058] The user inputs voice through the smartphone, and the smartphone's microphone captures the user's voice.

[1059] 2. Speech Recognition:

[1060] Smartphones use speech recognition software (speech_recognition) to convert captured speech into text, for example, Google's speech recognition API.

[1061] 3. Emotion recognition:

[1062] The converted text data is sent to the server, which then analyzes the user's emotions using an emotion recognition API to assess the user's emotional state.

[1063] 4. Humorous response generation:

[1064] The server uses a generative AI model (OpenAI GPT-3) to generate humorous responses to text data, using prompts to instruct the AI.

[1065] Example prompt sentence:

[1066] User: I hate the long traffic jams.

[1067] AI: Provide a humorous response.

[1068] 5. Audio output of response:

[1069] The server sends the generated humorous response to the smartphone, which uses the smartphone's text-to-speech engine (pyttsx3) to output the response aloud.

[1070] Specific examples

[1071] For example, if a user says "I hate being stuck in traffic jams for so long" in a self-driving vehicle, the smartphone's microphone picks up the speech. The speech recognition software then converts the speech into text, "I hate being stuck in traffic jams for so long," and sends it to the server. The server then uses an emotion recognition API to determine that the user is "angry." Based on this information, the generative AI model (OpenAI GPT-3) generates the following humorous response: "It's good to relax for a while," and the response is output via voice via the smartphone.

[1072] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1073] Step 1:

[1074] The user inputs voice using the microphone on the smartphone, and the user's voice data is captured by the microphone and sent to the device.

[1075] Step 2:

[1076] The device uses a speech recognition library (speech_recognition) to convert the captured voice data into text data. This process converts the user's voice input into text format. The input is voice data and the output is text data.

[1077] Step 3:

[1078] The converted text data is sent from the terminal to the server. The server receives this text data and proceeds to the next step. The input is text data, and the output is sending the data to the server.

[1079] Step 4:

[1080] The server uses an emotion recognition API to analyze the user's emotions from the received text data. Specifically, it extracts emotion tags (e.g., angry, sad, happy, etc.) from the text content. Through this process, the input is text data and the output is emotion tags.

[1081] Step 5:

[1082] The server uses a generative AI model (OpenAI GPT-3) to generate an appropriate humorous response based on the emotion tag and text data. Here, a prompt is used to instruct the AI. Specifically, a prompt such as "User: I'm tired of the long traffic jam. AI: Provide a humorous response" is created and passed to the AI ​​model. This process results in emotion tag and text data as input, and a humorous response as output.

[1083] Step 6:

[1084] Based on the generated humorous response and emotion tag, the response is adjusted as needed. For example, if the emotion tag is "angry," an additional phrase such as "It's good to relax for a while" is inserted into the humorous response. With this process, the input is a humorous response and emotion tag, and the output is the adjusted humorous response.

[1085] Step 7:

[1086] The server sends the adjusted humor response to the terminal. The input is the adjusted humor response, and the output is the transmission of data to the terminal.

[1087] Step 8:

[1088] The device converts the received tailored humor response into speech using a text-to-speech engine (pyttsx3) and outputs it to the user through the speaker. As a result of this process, the input is the tailored humor response and the output is the audio response.

[1089] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1090] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1091] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1092] [Fourth embodiment]

[1093] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1094] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1095] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1096] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1097] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1098] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1099] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1100] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1101] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1102] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1103] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1104] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1105] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1106] The present invention is a system for adding humor to an interactive agent, and for realizing friendly dialogue with the user. A specific embodiment of this system will be described below.

[1107] System Configuration

[1108] 1. A way to receive user input

[1109] The device receives voice or text input from the user. For example, if a user asks "What's the weather like today?", the device's microphone captures the voice.

[1110] 2. A means of converting input from speech to text

[1111] The voice recognition software installed on the device converts the captured voice into text, and the speech "What's the weather like today?" is converted into text data and sent to the server.

[1112] 3. Text Analysis and Intent Understanding

[1113] The server analyzes the received text data to understand the user's intent. Specifically, it uses a natural language processing engine to identify that the question is about the weather.

[1114] 4. Selecting and generating humor

[1115] Based on the analysis results, the server uses a generative AI model to generate appropriate humor. For example, for the topic "weather," it generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[1116] 5. Response and Gesture Generation

[1117] The server generates a response text for the user based on the generated humor, and simultaneously generates control data for the 3D avatar, creating a response that includes actions such as opening arms, smiling, and speaking.

[1118] 6. Sending response data

[1119] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[1120] 7. Displaying the response to the user

[1121] The device displays the received data and plays an animation in which a 3D avatar holds out its arms to the user and says with a smile, "The weather's sunny today... I'm not sure, but maybe you should go outside and check!"

[1122] Specific examples

[1123] The user asks, "What's the weather like today?" The device captures the voice and converts it into text using speech recognition software. The converted text is sent to a server where it is analyzed using a natural language processing engine. The server uses a generative AI model to create weather-related humor and determines the response: "The weather is sunny today... I'm not sure, but maybe you should go outside and check!" At the same time, movement data is generated for the 3D avatar to spread its arms and speak. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[1124] In this way, the present invention can provide users with friendly, humorous interactions.

[1125] The processing flow will be explained below.

[1126] Step 1:

[1127] The user asks the terminal by voice, "What's the weather like today?"

[1128] Step 2:

[1129] The device temporarily stores the audio received.

[1130] Step 3:

[1131] Speech recognition software installed on the device converts the saved voice data into text data.

[1132] Step 4:

[1133] The terminal transmits the converted text data to the server.

[1134] Step 5:

[1135] The server uses a natural language processing engine to analyze the text data it receives and understand the user's intent. In this case, it identifies that the question is about the weather.

[1136] Step 6:

[1137] The server uses a generative AI model based on the analysis to generate appropriate humor, in this case, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[1138] Step 7:

[1139] The server generates a response text for the user based on the generated humor.

[1140] Step 8:

[1141] At the same time, the server generates control data for the 3D avatar, creating the avatar's movements of spreading its arms, smiling, and talking.

[1142] Step 9:

[1143] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[1144] Step 10:

[1145] The device displays the received data and plays an animation in which a 3D avatar opens its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[1146] This specific processing flow allows the user to receive friendly responses, including humor, from the conversational agent.

[1147] Example 1

[1148] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1149] In conversational agents, it is necessary to add humor to realize friendly dialogue with users. However, conventional systems have had difficulty generating appropriate and natural humor in response to user input. Furthermore, converting speech to text and generating and displaying actions, including gestures, have also been issues.

[1150] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1151] In this invention, the server includes means for receiving a user input, means for converting the user input from speech to text using speech recognition software, means for analyzing the user input, means for generating a prompt sentence based on the analysis result and generating humor using a generative AI model, means for generating a response text and hand gestures based on the generated humor, and means for displaying the response text and hand gestures to the user. This allows for the generation of appropriate and natural humor in response to the user input, and the generation and display of actions including speech-to-text conversion and hand gestures.

[1152] "User" refers to any individual or end user who uses this System.

[1153] "Input" refers to the provision of information in the form of speech or text by a user to a system.

[1154] "Speech recognition software" refers to a program or algorithm used to analyze captured voice data and convert it into text data.

[1155] "Text" refers to character strings of data that correspond to a user's spoken input, as converted by speech recognition software.

[1156] "Analysis" refers to the process of understanding the user's intent based on the received text data.

[1157] A "prompt" is an instruction given to a generative AI model to generate humor based on user input.

[1158] "Generative AI model" refers to an artificial intelligence model that generates humor based on prompt text.

[1159] "Humor" refers to the friendly, entertaining responses to user input generated by the system.

[1160] "Responsive Text" refers to a humorous text response to a user's input that is created by a generative AI model.

[1161] "Gestures" refers to the physical expressions, such as gestures and facial expressions, that a 3D avatar makes toward the user.

[1162] "Display" refers to projecting text and 3D avatar movements onto the device screen to provide a visual experience to the user.

[1163] "Terminal" means the electronic device through which a User accesses, enters input into, and receives responses to this System.

[1164] The present invention is a system for adding humor to an interactive agent, and for realizing friendly dialogue with the user. A specific embodiment of this system will be described below.

[1165] 1. System Configuration

[1166] The system consists of a device for receiving user input, speech recognition software for converting the user's input from speech to text, and a server for analyzing the received text data and generating appropriate humor. Specifically, the device includes a microphone and speech recognition software (e.g., Google Speech-to-Text), while the server is equipped with a natural language processing engine (e.g., SpaCy) and a generative AI model (e.g., OpenAI GPT-3).

[1167] 2. Processing Flow

[1168] The user speaks to the device, asking, "What's the weather like today?" The device's microphone captures the voice, and speech recognition software converts the voice into text data. The converted text data is sent to the server in the form of "What's the weather like today?"

[1169] The server analyzes the received text data using a natural language processing engine and identifies it as a question about the weather. Based on the analysis results, it inputs the following prompt sentence into the generative AI model: "The user is asking, 'What's the weather like today?' Please provide a humorous response to this question."

[1170] Based on this prompt, the generative AI model generates a humorous response: "I'm not sure if the weather is sunny today, but maybe I should go outside and check it out!" The server also generates motion data for controlling the 3D avatar based on this response text. The generated control data includes the motion of the 3D avatar speaking with its arms outstretched and smiling.

[1171] 3. View the response

[1172] The generated response text and action data are sent from the server to the device. The device then displays a 3D avatar on the screen based on the received data, and plays an animation of the avatar spreading its arms and smiling, saying, "I don't know if the weather is sunny today... but maybe I should go outside and check!" In this way, the user can receive humorous responses along with the 3D avatar's actions.

[1173] Specific examples

[1174] For example, if a user asks "What's the weather like today?", the device captures the speech and uses speech recognition software to convert it into text. The converted text is sent to a server, which then analyzes it using a natural language processing engine. The server then uses a generative AI model to create weather-related humor and determines a response such as "The weather is sunny today... I'm not sure, but maybe you should go outside and check!" At the same time, movement data is also generated for the 3D avatar to spread its arms and speak. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[1175] An example of the prompt above is: "The user asks, 'What's the weather like today?' Please respond to this question with humor."

[1176] This invention makes it possible to generate appropriate and natural humor in response to user input, convert speech to text, and generate and display actions including gestures.

[1177] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1178] Step 1:

[1179] The user speaks, "What's the weather like today?"

[1180] The device captures this audio with a microphone.

[1181] Input data: User's voice.

[1182] Output data: Audio data.

[1183] Step 2:

[1184] The device uses voice recognition software (e.g., Google Speech-to-Text) to convert the captured voice data into text data.

[1185] Input data: Captured audio data.

[1186] Output data: Text data "What's the weather like today?"

[1187] Step 3:

[1188] The terminal transmits the converted text data to the server.

[1189] Input data: Text data "What's the weather like today?"

[1190] Output data: Sending text data to the server.

[1191] Step 4:

[1192] The server analyzes the received text data using a natural language processing engine (e.g., SpaCy).

[1193] Input data: Text data "What's the weather like today?"

[1194] Data processing: Analyze the received text data and identify that it is a question about the weather.

[1195] Output: Parsed intent ("Question about the weather").

[1196] Step 5:

[1197] The server generates a prompt based on the analysis results and sends it to the generative AI model (e.g., OpenAI GPT-3). The specific prompt is, "The user is asking, 'What's the weather like today?' Please respond to this question with humor."

[1198] Input data: Parsed intent ("Question about the weather").

[1199] Data processing: Prompt sentence generation.

[1200] Output data: The prompt sentence to be passed to the generative AI model.

[1201] Step 6:

[1202] The generative AI model generates an appropriate humorous response based on the prompt sentence.

[1203] Input data: prompt statement.

[1204] Data operations: Operations that analyze prompt sentences and generate humorous responses.

[1205] Output: A humorous response: "The weather is sunny today... I don't know, but maybe I should go outside and check!"

[1206] Step 7:

[1207] The server generates motion data for controlling the 3D avatar based on the humorous response text, including the motion of the avatar spreading its arms and speaking.

[1208] Input data: Humor response text.

[1209] Data processing: Generation of 3D avatar movements.

[1210] Output data: response text and 3D avatar movement data.

[1211] Step 8:

[1212] The server transmits the generated response text and the motion data of the 3D avatar to the terminal.

[1213] Input data: response text and 3D avatar movement data.

[1214] Output data: Data sent to the device.

[1215] Step 9:

[1216] Based on the data received, the device displays a 3D avatar on the screen and plays an animation in which the avatar spreads its arms and says with a smile, "I don't know if the weather will be sunny today... but maybe I should go outside and check!"

[1217] Input data: response text and 3D avatar movement data.

[1218] Data calculation: 3D avatar movement playback.

[1219] Output data: Responses and avatar behaviors displayed to the user.

[1220] (Application example 1)

[1221] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1222] Current conversational agent systems face challenges in incorporating humor appropriately into dialogue with users, making it difficult to achieve friendly, natural conversations. Furthermore, when agents interact with drivers and passengers in autonomous vehicles, they are unable to generate humorous responses and provide enjoyment. There is a need for conversational agent systems that incorporate humor to solve these problems and improve the user experience.

[1223] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1224] In this invention, the server includes means for receiving user input, means for converting the user input from speech to text, means for analyzing the text to understand the user's intent, means for generating humor based on the analysis results using a generative AI model, means for generating a response based on the generated humor, and means for displaying the response or playing it as an animation, thereby realizing a friendly dialogue with the user and providing a fun experience even inside an autonomous vehicle.

[1225] "Means for receiving user input" refers to devices or software for receiving instructions or questions from users, such as voice input or text input.

[1226] "Speech-to-text means" means speech recognition software or algorithms used to convert spoken input into text form.

[1227] "Means for analyzing text and understanding user intent" refers to a natural language processing engine that analyzes input text data and identifies the user's intent and purpose from its content.

[1228] "Means for generating humor using a generative AI model" refers to a method that uses a generative AI model (e.g., a GPT model) to generate relevant humor based on the analysis results.

[1229] "Means for generating responses based on generated humor" refers to software or algorithms that use the generated humor to create responses to users.

[1230] "Means for displaying a response or playing it as an animation" refers to a display device for visually conveying the generated response to the user, an engine for controlling the animation of an avatar, etc.

[1231] The present invention is a system that analyzes user input, generates humorous responses, and displays them. This system is intended to be installed in an autonomous vehicle, and specific embodiments are described below.

[1232] System Configuration

[1233] 1. A way to receive user input

[1234] The autonomous vehicle is equipped with a microphone that captures voice input from the user, and this voice data includes the user's questions and instructions.

[1235] 2. Voice-to-text conversion methods

[1236] The terminal inside the vehicle is installed with voice recognition software (e.g., Google Speech-to-Text API) that converts the captured voice data into text format.

[1237] 3. Text Analysis and Intent Understanding

[1238] The converted text is sent to a server and analyzed by a natural language processing engine (e.g., Google Cloud Natural Language API). This analysis allows the system to understand the user's intent and generate appropriate answers based on that.

[1239] 4. Selecting and generating humor

[1240] The server uses a generative AI model (such as OpenAI's GPT model) based on the analysis results to generate appropriate humor, enabling friendly dialogue.

[1241] 5. Generating Responses and Animations

[1242] Based on the humor generated, the server generates a response text for the user. At the same time, it generates avatar movement data using a 3D avatar generation engine (e.g., Unreal Engine). This movement data includes information on the avatar's facial expressions and gestures.

[1243] 6. Sending response data

[1244] The generated text response and avatar movement data are sent to the device using a communication module (e.g., WebSocket).

[1245] 7. Displaying the response to the user

[1246] The received data is displayed on the vehicle's display and an avatar animation is played, providing users with a natural and enjoyable interaction experience.

[1247] Example

[1248] For example, if a user asks, "What's the nearest restaurant?", the voice is first captured by a microphone. Speech recognition software then converts the voice into text, which is then sent to the server. The server then analyzes it with a natural language processing engine to understand the intent. Based on the results, the generative AI model generates a humorous response, such as, "The nearest restaurant is just around the corner... but the food has mixed reviews!" The response is displayed alongside an animated 3D avatar, allowing the user to experience the humorous response.

[1249] Prompt Sentence Examples

[1250] The prompt to the generative AI model would be:

[1251] Generate a humorous response about "nearest restaurant."

[1252] This embodiment allows for friendly interaction with the user, providing a fun experience even inside an autonomous vehicle.

[1253] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1254] Step 1:

[1255] A user provides voice input inside an autonomous vehicle. The user's input is captured through a microphone, and voice data is obtained as input, which is then used in subsequent processing steps.

[1256] Step 2:

[1257] The device converts the captured voice data into text data using speech recognition software (e.g., Google Speech-to-Text API). The voice data is converted into acoustic features, and the speech recognition algorithm generates a text string based on the features. The text data is obtained as output.

[1258] Step 3:

[1259] The device sends the converted text data to a server. A natural language processing engine (e.g., Google Cloud Natural Language API) connected to the server receives the text data and begins analyzing it. The input text data is analyzed, and the user's intent and topic are identified as the analysis results.

[1260] Step 4:

[1261] Based on the analysis results, the server generates humor using a generative AI model (e.g., OpenAI's GPT model). Here, the generative AI model is prompted to generate a humorous response about the "nearest restaurant." As an output, the generative AI model generates a humorous response text.

[1262] Step 5:

[1263] The server generates a response text for the user and corresponding animation data for a 3D avatar based on the generated humor text. Using a 3D avatar generation engine (e.g., Unreal Engine), motion data for the avatar to spread its arms, smile, and talk is generated. The response text and animation data are obtained as output.

[1264] Step 6:

[1265] The generated response text and avatar animation data are sent from the server to the device using a communication module (e.g., WebSocket). The device analyzes the received data and prepares it for display.

[1266] Step 7:

[1267] The terminal displays the response text on the vehicle's display and simultaneously plays an animation of the avatar, providing the user with a visual response, which allows the user to enjoy a friendly and enjoyable interaction experience. The output is the text and animation displayed on the display.

[1268] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1269] This invention relates to a system that combines humor and emotion recognition with a conversational agent, enabling friendly interactions with the user and providing appropriate responses depending on the user's emotions. Specific embodiments of this system are described below.

[1270] System Configuration

[1271] 1. A way to receive user input

[1272] The device receives voice or text input from the user. For example, if a user asks "What's the weather like today?", the device's microphone captures the voice.

[1273] 2. A means of converting input from speech to text

[1274] The voice recognition software installed on the device converts the captured voice into text, and the speech "What's the weather like today?" is converted into text data and sent to the server.

[1275] 3. Text Analysis and Intent Understanding

[1276] The server analyzes the received text data to understand the user's intent. Specifically, it uses a natural language processing engine to identify that the question is about the weather.

[1277] 4. Use of Emotion Engine

[1278] The emotion engine installed on the server recognizes the user's emotions from their voice or text data, for example, identifying whether the user is excited or depressed.

[1279] 5. Selecting and generating humor

[1280] The server uses a generative AI model to generate appropriate humor based on the analysis results and the recognition results of the emotion engine. In this case, for the topic "weather," it generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[1281] 6. Response and Gesture Generation

[1282] The server generates a response text for the user based on the generated humor. At the same time, it generates control data for a 3D avatar based on the recognition results of the emotion engine, creating a response that includes actions such as spreading arms, smiling, and speaking.

[1283] 7. Sending response data

[1284] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[1285] 8. Displaying the response to the user

[1286] The device displays the received data and plays an animation in which a 3D avatar extends its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[1287] Specific examples

[1288] The user asks, "What's the weather like today?" The device receives the voice and converts it into text using speech recognition software. The converted text is sent to a server where it is analyzed by a natural language processing engine. The server uses a generative AI model to create weather-related humor and determines the response: "The weather is sunny today... I'm not sure, but maybe you should go outside and check!" At the same time, the emotion engine analyzes the user's emotions. If it determines that the user is feeling down, it generates behavior data for the 3D avatar, including more friendly and comforting actions. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[1289] In this way, the present invention can provide a user with a humorous dialogue that is friendly to the user, and can also respond appropriately according to the user's emotions.

[1290] The processing flow will be explained below.

[1291] Step 1:

[1292] The user asks the device verbally, "What's the weather like today?"

[1293] Step 2:

[1294] The device captures the user's voice and temporarily stores it.

[1295] Step 3:

[1296] Speech recognition software installed on the device converts the captured voice into text data.

[1297] Step 4:

[1298] The terminal transmits the converted text data to the server.

[1299] Step 5:

[1300] The server uses a natural language processing engine to analyze the text data it receives and understand the user's intent. In this case, it identifies that the question is about the weather.

[1301] Step 6:

[1302] The server then uses the analysis results to activate the emotion engine and analyze the user's emotional state, for example, identifying whether the user is excited or depressed.

[1303] Step 7:

[1304] Based on the analysis results and the recognition results of the emotion engine, the server uses a generative AI model to select an appropriate humor. In this case, it generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[1305] Step 8:

[1306] The server generates a response text for the user based on the generated humor, and adjusts the response content according to the recognition results of the emotion engine.

[1307] Step 9:

[1308] At the same time, the server generates control data for the 3D avatar and configures it to perform movements that reflect the corresponding emotions (e.g., speaking with a smile and spreading one's arms).

[1309] Step 10:

[1310] The server transmits the generated text response and the motion data of the 3D avatar to the terminal.

[1311] Step 11:

[1312] The device displays the received data and plays an animation in which a 3D avatar opens its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[1313] This specific processing flow allows the user to receive friendly responses from the conversational agent that reflect humor and emotion.

[1314] Example 2

[1315] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1316] Conventional conversational agent systems often respond to user input mechanically and without considering emotions, making interactions with users unfriendly. Furthermore, there is a lack of systems that combine humorous responses and behaviors that respond to the user's emotions, making it impossible to enrich the user experience. This has led to issues such as a decline in the quality of interaction with users and low satisfaction.

[1317] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1318] In this invention, the server includes means for receiving a user's input, means for analyzing the user's input, means for recognizing the analysis result and emotion, means for generating humor based on the analysis result and the emotion recognition, and means for generating a response and gesture based on the generated humor, thereby enabling a response including appropriate humor and action according to the user's emotion.

[1319] "Means for receiving user input" refers collectively to any interface or device that captures information when a user provides it to a system in the form of voice, text, or other information.

[1320] "Means of analysis" is a general term for algorithms and software that process input data received from users and extract specific intent or meaning.

[1321] "Emotion recognition means" is a general term for algorithms and engines that analyze and identify a user's emotional state from their voice or text data.

[1322] "Humor generation means" is a general term for algorithms and models that generate humorous content for users based on the results of analysis and emotion recognition.

[1323] "Means for generating responses and gestures" refers collectively to the algorithms and engines that control and generate response text and corresponding actions and gestures to the user based on the generated humor.

[1324] "Means for receiving" is a general term for interfaces and devices for receiving input from users and data from servers.

[1325] "Displaying means" is a general term for devices and software that visually or audibly present information to the user, such as generated response text or gestures.

[1326] This invention is a system that combines humor and emotion recognition to realize friendly dialogue with users in a conversational agent. The system receives voice or text input from the user, analyzes the input, and generates appropriate humorous responses and emotionally appropriate actions to present to the user.

[1327] System Configuration

[1328] 1. A way to receive user input

[1329] The device receives voice or text input from the user. For example, when a user asks, "What's the weather like today?", the device's microphone captures the audio. Required hardware includes a microphone, speaker, and display.

[1330] 2. A means of converting input from speech to text

[1331] Speech recognition software (e.g., Google Cloud Speech-to-Text) installed on the device converts the captured voice data into text data, which becomes "What's the weather like today?" and is sent to the server.

[1332] 3. Text Analysis and Intent Understanding

[1333] The server analyzes the received text data and uses a natural language processing engine (e.g., Google Natural Language API) to understand the user's intent, thereby identifying that the user's question is about the weather.

[1334] 4. Use of Emotion Engine

[1335] The emotion engine (e.g., IBM Watson Tone Analyzer) installed on the server recognizes emotions from the user's text data, and identifies whether the user is excited or depressed.

[1336] 5. Selecting and generating humor

[1337] Based on the analysis and emotion recognition results, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate humor, such as "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[1338] 6. Response and Gesture Generation

[1339] The server generates a response text based on the generated humor, and simultaneously generates motion data for a 3D avatar based on the recognition results of the emotion engine, including, for example, a motion in which the avatar smiles and spreads its arms.

[1340] 7. Sending response data

[1341] The server transmits the generated response text and the motion data of the 3D avatar to the terminal.

[1342] 8. Displaying the response to the user

[1343] The device displays the received data and plays an animation in which a 3D avatar extends its arms to the user and says with a smile, "The weather's sunny today... I don't know, but maybe you should go outside and check!"

[1344] Specific examples

[1345] The user asks, "What's the weather like today?" The device receives the voice and converts it into text using speech recognition software. The converted text is sent to a server where it is analyzed by a natural language processing engine. The server uses a generative AI model to create a weather-related joke and determines the response, "The weather is sunny today... I don't know, but maybe you should go outside and check!" At the same time, the emotion engine analyzes the user's emotions, and if it determines that the user is feeling down, the 3D avatar will take comforting actions. This response data is sent to the device, and the user receives the response along with the avatar's movements.

[1346] Prompt Sentence Examples

[1347] "Generate humorous responses to users asking about today's weather, and provide responses with comforting 3D avatar behavior if the user is feeling down."

[1348] By feeding this prompt into a generative AI model, the system can generate a response for the user that includes appropriate humor and actions.

[1349] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1350] Step 1:

[1351] User inputs by voice.

[1352] The user asks the device, "What's the weather like today?"

[1353] Input: User's voice data

[1354] Output: Audio data

[1355] Step 2:

[1356] The device captures the audio.

[1357] The device's microphone captures the user's voice and stores it as digital audio data.

[1358] Input: Audio data

[1359] Output: Digital audio data

[1360] Step 3:

[1361] Text conversion using voice recognition software installed on the device.

[1362] Speech recognition software (e.g., Google Cloud Speech-to-Text) converts digital voice data into text data.

[1363] Input: Digital audio data

[1364] Output: Text data "What's the weather like today?"

[1365] Step 4:

[1366] The device sends text data to the server.

[1367] The terminal sends the converted text data to the server, typically requiring an HTTP request.

[1368] Input: Text data

[1369] Output: HTTP request to the server

[1370] Step 5:

[1371] The server receives and analyzes the text data.

[1372] The server receives the HTTP request and analyzes the text data using a natural language processing engine (e.g., Google Natural Language API). This analysis helps to understand the user's intent and identifies the question as being about the weather.

[1373] Input: Text data

[1374] Output: Text analysis results

[1375] Step 6:

[1376] The server analyzes emotions using an emotion engine.

[1377] The emotion engine (e.g., IBM Watson Tone Analyzer) installed on the server recognizes the user's emotion from the text data, for example, identifying the user's emotional state, such as whether they are excited or depressed.

[1378] Input: Text data

[1379] Output: Emotion analysis results

[1380] Step 7:

[1381] Server generated humor.

[1382] Based on the results of text analysis and sentiment analysis, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate humor. It generates a joke like, "The weather is sunny today... I'm not sure, but maybe you should go outside and check!"

[1383] Input: Text analysis results, sentiment analysis results

[1384] Output: Humorous response text

[1385] Step 8:

[1386] The server generates the response text and action data.

[1387] The server creates a response text based on the generated humor. At the same time, it generates motion data for the 3D avatar based on the analysis results of the emotion engine. For example, it creates a motion for the avatar to speak with its arms outstretched and smiling.

[1388] Input: Humorous response text, sentiment analysis results

[1389] Output: Response text, action data

[1390] Step 9:

[1391] The server sends the response data to the terminal.

[1392] The server sends the generated response text and the 3D avatar's movement data to the terminal.

[1393] Input: Response text, action data

[1394] Output: HTTP response to the device

[1395] Step 10:

[1396] The device responds.

[1397] The device displays the received data and plays an animation in which a 3D avatar holds out its arms to the user and says with a smile, "The weather's sunny today... I'm not sure, but maybe you should go outside and check!"

[1398] Input: Response text, action data

[1399] Output: Animation and audio response display

[1400] (Application example 2)

[1401] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1402] Conventional conversational agent systems have difficulty responding flexibly to the user's emotions and circumstances, and have been unable to sufficiently improve user satisfaction. Furthermore, generating humorous responses requires taking into account the user's current emotional state, but the lack of a function to dynamically adjust the humor accordingly can sometimes result in an inconsistent user experience.

[1403] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a user's input, means for analyzing the user's input, means for generating humor based on the analysis result, means for generating and displaying a response based on the generated humor and outputting it as voice, means for performing emotion recognition, and means for adjusting the humor in accordance with the emotion recognition result. This makes it possible to provide an appropriate and friendly response, mixed with humor, that is in accordance with the user's emotional state.

[1404] A "means for receiving user input" is a device or interface for capturing speech or text generated by a user.

[1405] "Means for analyzing user input" means software or a system that interprets captured voice or text data and analyzes its intent and content.

[1406] "Means for generating humor based on the analysis results" refers to algorithms or generative AI models that create appropriate humor based on the analyzed input data.

[1407] The "means for generating and displaying a response based on the generated humor and outputting it as audio" is a mechanism for displaying a response containing the generated humor as text and playing it back as audio using speech synthesis technology.

[1408] "Means for emotion recognition" means software or emotion engines that analyze and identify a user's emotional state from their voice or text.

[1409] The "means for adjusting humor according to the results of emotion recognition" is a mechanism for optimizing the generated humor and selecting an appropriate response based on the user's emotional state.

[1410] This invention is an interactive agent system that provides friendly dialogue with users and responds appropriately based on the user's emotions. This system consists of a device that receives the user's voice and text input, a server that analyzes the input data, and a terminal that generates and displays humorous responses and outputs them as voice.

[1411] Hardware and software used

[1412] Hardware: The microphone and speaker on your smartphone.

[1413] Software: Speech recognition library (speech_recognition), text-to-speech engine (pyttsx3), emotion recognition API, generative AI model (OpenAI GPT-3).

[1414] Process Overview

[1415] 1. Voice input acquisition:

[1416] The user inputs voice through the smartphone, and the smartphone's microphone captures the user's voice.

[1417] 2. Speech Recognition:

[1418] Smartphones use speech recognition software (speech_recognition) to convert captured speech into text, for example, Google's speech recognition API.

[1419] 3. Emotion recognition:

[1420] The converted text data is sent to the server, which then analyzes the user's emotions using an emotion recognition API to assess the user's emotional state.

[1421] 4. Humorous response generation:

[1422] The server uses a generative AI model (OpenAI GPT-3) to generate humorous responses to text data, using prompts to instruct the AI.

[1423] Example prompt sentence:

[1424] User: I hate the long traffic jams.

[1425] AI: Provide a humorous response.

[1426] 5. Audio output of response:

[1427] The server sends the generated humorous response to the smartphone, which uses the smartphone's text-to-speech engine (pyttsx3) to output the response aloud.

[1428] Specific examples

[1429] For example, if a user says "I hate being stuck in traffic jams for so long" in a self-driving vehicle, the smartphone's microphone picks up the speech. The speech recognition software then converts the speech into text, "I hate being stuck in traffic jams for so long," and sends it to the server. The server then uses an emotion recognition API to determine that the user is "angry." Based on this information, the generative AI model (OpenAI GPT-3) generates the following humorous response: "It's good to relax for a while," and the response is output via voice via the smartphone.

[1430] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1431] Step 1:

[1432] The user inputs voice using the microphone on the smartphone, and the user's voice data is captured by the microphone and sent to the device.

[1433] Step 2:

[1434] The device uses a speech recognition library (speech_recognition) to convert the captured voice data into text data. This process converts the user's voice input into text format. The input is voice data and the output is text data.

[1435] Step 3:

[1436] The converted text data is sent from the terminal to the server. The server receives this text data and proceeds to the next step. The input is text data, and the output is sending the data to the server.

[1437] Step 4:

[1438] The server uses an emotion recognition API to analyze the user's emotions from the received text data. Specifically, it extracts emotion tags (e.g., angry, sad, happy, etc.) from the text content. Through this process, the input is text data and the output is emotion tags.

[1439] Step 5:

[1440] The server uses a generative AI model (OpenAI GPT-3) to generate an appropriate humorous response based on the emotion tag and text data. Here, a prompt is used to instruct the AI. Specifically, a prompt such as "User: I'm tired of the long traffic jam. AI: Provide a humorous response" is created and passed to the AI ​​model. This process results in emotion tag and text data as input, and a humorous response as output.

[1441] Step 6:

[1442] Based on the generated humorous response and emotion tag, the response is adjusted as needed. For example, if the emotion tag is "angry," an additional phrase such as "It's good to relax for a while" is inserted into the humorous response. With this process, the input is a humorous response and emotion tag, and the output is the adjusted humorous response.

[1443] Step 7:

[1444] The server sends the adjusted humor response to the terminal. The input is the adjusted humor response, and the output is the transmission of data to the terminal.

[1445] Step 8:

[1446] The device converts the received tailored humor response into speech using a text-to-speech engine (pyttsx3) and outputs it to the user through the speaker. As a result of this process, the input is the tailored humor response and the output is the audio response.

[1447] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1448] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1449] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1450] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1451] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1452] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1453] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1454] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1455] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1456] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1457] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1458] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1459] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1460] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1461] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1462] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1463] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1464] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1465] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1466] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1467] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1468] The following is further disclosed regarding the above embodiment.

[1469] (Claim 1)

[1470] a means for receiving user input;

[1471] means for analyzing said user input;

[1472] means for generating humor based on the analysis results;

[1473] means for generating a response based on the generated humor;

[1474] means for displaying the response;

[1475] A system including:

[1476] (Claim 2)

[1477] 2. The system according to claim 1, further comprising means for generating a gesture based on the analysis result.

[1478] (Claim 3)

[1479] 10. The system of claim 1, further comprising means for converting the user's input from speech to text.

[1480] "Example 1"

[1481] (Claim 1)

[1482] a means for receiving user input;

[1483] means for converting said user input from speech to text using speech recognition software;

[1484] means for analyzing said user input;

[1485] a means for generating prompt sentences based on the analysis results and generating humor using a generative AI model;

[1486] means for generating a response text and a hand gesture based on the generated humor;

[1487] means for displaying the response text and hand gesture to a user;

[1488] A system including:

[1489] (Claim 2)

[1490] 2. The system according to claim 1, further comprising means for transmitting the generated response text and hand gesture to a terminal and displaying the received data.

[1491] (Claim 3)

[1492] 10. The system of claim 1, further comprising a natural language processing engine for parsing the user's input.

[1493] "Application Example 1"

[1494] (Claim 1)

[1495] a means for receiving user input;

[1496] means for converting said user input from speech to text;

[1497] means for analyzing said text to understand user intent;

[1498] A means for generating humor based on the analysis results using an AI model;

[1499] means for generating a response based on the generated humor;

[1500] means for displaying or replaying the response as an animation;

[1501] A system including:

[1502] (Claim 2)

[1503] 2. The system according to claim 1, further comprising means for generating a gesture based on the analysis result.

[1504] (Claim 3)

[1505] The system according to claim 1, characterized in that the information is displayed on a display mounted on an autonomous vehicle.

[1506] "Example 2: Combining Emotion Engines"

[1507] (Claim 1)

[1508] a means for receiving user input;

[1509] means for analyzing said user input;

[1510] means for recognizing the analysis result and emotion;

[1511] means for generating humor based on the analysis results and emotion recognition;

[1512] means for generating responses and gestures based on the generated humor;

[1513] means for displaying said responses and gestures;

[1514] A system including:

[1515] (Claim 2)

[1516] 10. The system of claim 1, further comprising means for converting the user's input from speech to text.

[1517] (Claim 3)

[1518] 2. The system of claim 1, wherein the emotion recognition means identifies emotions from the user's voice or text data.

[1519] "Application example 2 when combining emotion engines"

[1520] (Claim 1)

[1521] a means for receiving user input;

[1522] means for analyzing said user input;

[1523] means for generating humor based on the analysis results;

[1524] means for generating a response based on the generated humor;

[1525] means for displaying and audibly outputting said response;

[1526] a means for performing emotion recognition;

[1527] means for adjusting the humor in response to the emotion recognition results;

[1528] A system including:

[1529] (Claim 2)

[1530] 2. The system according to claim 1, further comprising means for generating a gesture based on the analysis result.

[1531] (Claim 3)

[1532] 10. The system of claim 1, further comprising means for converting the user's input from speech to text. [Explanation of symbols]

[1533] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for receiving user input; means for analyzing said user input; means for generating humor based on the analysis results; means for generating a response based on the generated humor; means for displaying the response; A system including:

2. The system of claim 1 further comprising means for generating a gesture based on the analysis result.

3. 2. The system of claim 1, further comprising means for converting said user input from speech to text.

Images