System

Abstract

A system is provided.SOLUTION: A system comprising: means for recording conversation data of a user in life; means for analyzing the recorded conversation data and interpreting hobbies, preferences, and characteristics of the user; means for automatically generating response candidates based on the hobbies, preferences, and characteristics of the user by analyzing a speech of another person in a scene in which it is difficult for the user to speak; means for presenting the generated response candidates to the user and causing the user to select a response candidate; and means for outputting the selected response candidate by a voice of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Users with selective mutism are limited in the places where they can speak freely. This makes communication difficult in certain environments, making it difficult to build social relationships. Conventional technologies assume that users will converse directly, and there are no systems that allow users with selective mutism to communicate smoothly. Therefore, there is a need for a method that provides appropriate responses and supports smooth communication even in situations where it is difficult for users to speak. [Means for solving the problem]

[0005] The present invention is a system that includes a means for recording conversation data from a user's daily life, a means for analyzing the recorded conversation data and interpreting the user's hobbies, preferences, and characteristics, a means for automatically generating reply candidates based on the user's hobbies, preferences, and characteristics by analyzing what others say in situations where it is difficult for the user to speak, a means for presenting the generated reply candidates to the user and allowing the user to select one, and a means for outputting the selected reply candidate in the user's voice. Specifically, the system converts the recorded conversation data into text data using speech recognition, analyzes the data using natural language processing, and further generates reply candidates using a generative AI model. The reply selected by the user is output in the user's voice using speech synthesis technology, allowing the user to communicate smoothly even in situations where it is difficult to speak.

[0006] "Conversation data" refers to voice information of daily conversations that users have and the textual content of such conversations.

[0007] "Speech recognition" refers to the technology of analyzing recorded voice data and converting it into corresponding text data.

[0008] "Natural language processing" refers to technology for analyzing and understanding text data and interpreting the meaning and context of language.

[0009] A "generative AI model" refers to an artificial intelligence model that generates appropriate response candidates based on a user's preferences and characteristics.

[0010] "Candidate responses" refer to possible answers or reactions that a user may have generated to facilitate smooth communication with others.

[0011] "Speech synthesis" refers to the technology of converting text data into voice data that resembles the user's voice.

[0012] A "user profile" refers to a data set that compiles information such as a user's hobbies, preferences, and characteristics.

[0013] "Terminal" refers to a device used by a user that has the functions of recording voice, converting voice into text, presenting response candidates, and playing back voice.

[0014] A "server" refers to a remote computer system that analyzes data sent from a terminal, generates response candidates, and synthesizes voice. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system to help resolve selective mutism, an anxiety disorder that makes it difficult or impossible to speak in certain social environments or situations. In situations where it is difficult for the user to speak for themselves, a generative AI model is used to generate response candidates based on the user's preferences, and a mechanism is built in which the content selected by the user is spoken in the user's voice instead.

[0037] System configuration

[0038] 1. Conversation data collection

[0039] Device: Record conversations in scenes where the user can interact. For example, if a user is talking to a friend about a game at home, record this conversation.

[0040] 2. Data Analysis

[0041] Device: The recorded voice data is converted into text data using a speech recognition API. This text data will be in a format such as "What games are you playing these days?"

[0042] Terminal: Sends text data to the server.

[0043] 3. Create a user profile

[0044] Server: Analyzes the received text data using a natural language processing engine. From the analyzed data, the user's hobbies, preferences, and characteristics are extracted to create a user profile.

[0045] Server: Store information such as "I like games" or "I'm interested in RPGs" in the user profile.

[0046] 4. Real-time conversation processing

[0047] Terminal: Records a teacher's response to a question in a classroom where the user has difficulty speaking. The question is, "Please explain why you are working on the following assignment."

[0048] Device: The recorded speech is converted into text data using a speech recognition API. This text data is also sent to the server.

[0049] 5. Response candidate generation

[0050] Server: Based on the received text data and user profile, the server uses a generative AI model to automatically generate appropriate response candidates, such as "This assignment seems interesting because it will allow me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[0051] Server: Sends multiple response candidates to the device.

[0052] 6. User Selection and Speech

[0053] Terminal: Presents the received response candidates to the user. The user selects an option such as "This assignment seems interesting because it allows me to learn a new perspective."

[0054] Terminal: Sends the user-selected response to the server.

[0055] Server: Converts the selected response into audio data in the user's voice using speech synthesis technology.

[0056] Server: Sends the generated audio data to the device.

[0057] Terminal: Plays back the received voice data and speaks on behalf of the user.

[0058] Specific examples

[0059] Example 1: Collecting everyday conversation data

[0060] Device: At home, the user speaks to a friend, "What games have you been playing recently?" The device records this conversation, converts it into text using voice recognition, and sends it to the server.

[0061] Server: Analyzes the received data and determines that "the user is interested in the latest RPG."

[0062] Example 2: Real-time conversation at school

[0063] Device: During a class, the teacher asks, "Please explain why you are going to tackle the next assignment." The device records this question, converts it into text using voice recognition, and sends it to the server.

[0064] Server: Based on the question, it generates response candidates such as "This assignment seems interesting because it will allow me to learn a new perspective" and "Because the content of this assignment will be useful to me in the future" and sends them to the device.

[0065] Terminal: Sends the user-selected response to the server.

[0066] Server: Synthesizes speech using the user's voice and sends the generated voice data to the device.

[0067] Device: Plays back the received audio data and speaks on behalf of the user, "This assignment seems interesting because it will allow me to learn a new perspective."

[0068] In this way, the present invention realizes smooth communication by uttering a response appropriate to the situation in the user's voice, even in situations where it is difficult for the user to speak.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] The user initiates a conversation in a scene where they can interact, for example, a user is talking with a friend about a game at home.

[0072] Step 2:

[0073] The device records the conversation. The device's microphone captures what you and your friends say.

[0074] Step 3:

[0075] The voice data recorded by the device is converted into text data using a speech recognition API. This conversion results in the text data "What games have you been playing recently?"

[0076] Step 4:

[0077] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[0078] Step 5:

[0079] The server receives the text data and stores it in a database, which stores past conversation data.

[0080] Step 6:

[0081] The server analyzes the stored text data using a natural language processing engine, extracting keywords such as "game" or "RPG."

[0082] Step 7:

[0083] Based on the analysis results, the server reflects the user's hobbies, preferences, and characteristics in the profile. Information such as "I like games" and "I'm interested in RPGs" is added to the user profile.

[0084] Step 8:

[0085] In situations where it is difficult for the user to speak (e.g., a school class), the teacher may ask a question during the class, such as, "Please explain why you are working on the next assignment."

[0086] Step 9:

[0087] The device records questions asked during class, and the teacher's comments are captured through the device's microphone.

[0088] Step 10:

[0089] The device converts the recorded voice data into text data using a speech recognition API. After conversion, the resulting text data reads, "Please explain why you want to tackle the following assignment."

[0090] Step 11:

[0091] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[0092] Step 12:

[0093] The server receives the text data and compares it with the user's profile. It then creates response candidates based on the teacher's question and the user's hobbies and preferences.

[0094] Step 13:

[0095] The server uses a generative AI model to generate multiple response candidates, such as "This assignment seems interesting because it allows me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[0096] Step 14:

[0097] The server sends multiple response options to the device. The data is sent securely using the HTTPS protocol.

[0098] Step 15:

[0099] The terminal presents the received response candidates to the user through a graphical user interface (GUI), displaying options such as "This assignment seems interesting because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[0100] Step 16:

[0101] The user selects one of the suggested responses, for example, "This assignment seems interesting because it will allow me to learn a new perspective."

[0102] Step 17:

[0103] The device sends the user-selected response to the server, transmitting data securely using the HTTPS protocol.

[0104] Step 18:

[0105] The server converts the selected response into audio data in the user's voice using a speech synthesis technology, such as a speech synthesis technology (e.g., WaveNet).

[0106] Step 19:

[0107] The server sends the audio data to the device. The data is sent securely using the HTTPS protocol.

[0108] Step 20:

[0109] The device plays back the received audio data, saying on behalf of the user, "This assignment seems interesting because it allows me to learn a new perspective."

[0110] The above are the specific program processing steps for implementing this system, which enables users to communicate smoothly even in situations where speaking is difficult.

[0111] Example 1

[0112] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0113] In the past, when users with anxiety disorders such as selective mutism had difficulty communicating, there was a lack of means for them to utter appropriate responses themselves, which led to difficulties and stress in social interactions.

[0114] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0115] In this invention, the server includes means for acquiring voice data from the user's daily life, means for converting the acquired voice data into text data using a voice recognition system, means for analyzing the converted text data and extracting the user's interests and characteristics, means for converting speech by others into text data using a voice recognition system and transmitting the text data to the server in situations where it is difficult for the user to speak, and automatically generating candidate responses using a generative AI model based on the received text data, means for presenting the generated candidate responses to the user for selection, means for converting the candidate response selected by the user into voice data in the user's voice using voice synthesis technology, and means for playing back the converted voice data. This makes it possible for the user to produce appropriate responses in their own voice even in situations where it is difficult for the user to speak.

[0116] "Voice data" is digital audio information that is a collection of speech from the user and others.

[0117] A "voice recognition system" is a technology or device that analyzes voice data and converts the content into text data.

[0118] "Text data" refers to character information of voice data converted by a voice recognition system.

[0119] "User interests" refers to themes or fields in which a user shows interest or concern.

[0120] "User characteristics" refers to personal characteristics such as a user's preferences, behavioral patterns, and personality.

[0121] A "server" is a computing device that receives and analyzes voice and text data and generates candidate responses.

[0122] A "generative AI model" refers to an artificial intelligence algorithm that performs natural language processing based on received data and automatically generates response candidates.

[0123] "Response candidates" are utterance options presented to the user that are automatically generated by a generative AI model.

[0124] "Speech synthesis technology" is a technology that converts text data into voice data and enables it to be played in a specific voice.

[0125] "Means for reproducing audio data" refers to a device that outputs the generated audio data as sound through speakers, headphones, etc.

[0126] The present invention is a system for helping to resolve selective mutism, an anxiety disorder that makes it difficult or impossible to speak in certain social environments or situations. Below, we will explain in detail how to implement this system.

[0127] Conversation data collection

[0128] Device: A device that records a user's everyday conversations. For example, it uses hardware such as a smartphone or a dedicated voice recorder. This allows a user to record a conversation, such as when talking with a friend about a game at home.

[0129] Data analysis

[0130] Device: The recorded voice data is converted into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text API). Through this process, the recorded conversation is obtained as text data such as "What games have you been playing recently?"

[0131] Terminal: Sends the converted text data to the server.

[0132] Creating a User Profile

[0133] Server: The received text data is analyzed using a natural language processing engine (e.g., Google Cloud Natural Language API). This analysis extracts the user's interests and characteristics. For example, the analysis result may be information such as "The user is interested in the latest RPGs." This information is saved in a database as a user profile.

[0134] Real-time conversation processing

[0135] Device: Records situations where it is difficult for the user to speak, such as questions asked in class. If a teacher asks a question like, "Please explain why you are going to tackle the next assignment," the device will record the content.

[0136] Terminal: The recorded speech is converted into text data using a voice recognition system, and this text data is sent to the server.

[0137] Response candidate generation

[0138] Server: Based on the received text data and user profile, a generative AI model (e.g., OpenAI's GPT-3) is used to generate appropriate response candidates, such as "This assignment seems interesting because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[0139] User selection and speech

[0140] Terminal: Presents the generated response candidates to the user, showing options on a screen, etc. The user selects the appropriate response from the presented candidates.

[0141] Terminal: Sends the selected response to the server.

[0142] Server: The selected response is converted into voice data that resembles the user's voice using voice synthesis technology (e.g., Google Cloud Text-to-Speech API), and this voice data is sent to the device.

[0143] Terminal: Plays back the received voice data and speaks on behalf of the user.

[0144] Specific examples

[0145] Example 1: Collecting everyday conversation data

[0146] Device: At home, a user talks to a friend, "What games have you been playing recently?" This conversation is recorded, converted into text using voice recognition, and sent to the server.

[0147] Server: Analyzes the received data and determines that "the user is interested in the latest RPGs" and reflects this in the profile.

[0148] Example 2: Real-time conversation at school

[0149] Device: During a class, a teacher asks, "Please explain why you are going to tackle the following assignment." This question is recorded, converted into text using voice recognition, and sent to a server.

[0150] Server: Based on the question, it generates response candidates such as "This assignment seems interesting because it will allow me to learn a new perspective" and "Because the content of this assignment will be useful to me in the future." It then sends these to the device.

[0151] Terminal: The user selects "This assignment seems interesting because it allows me to learn a new perspective" from the presented response candidates. After the selection, the terminal sends this content to the server.

[0152] Server: The selected content is synthesized using the user's voice and the generated voice data is sent to the terminal.

[0153] Terminal: Plays back audio data and speaks on behalf of the user.

[0154] Example prompts to input to the generative AI model

[0155] "When a teacher asks a user why they are taking on the next assignment, generate appropriate response suggestions based on the user profile: The user likes games and is interested in RPGs."

[0156] As a result, the present invention can achieve smooth communication even in situations where it is difficult for the user to speak by uttering a response appropriate to the situation in the user's voice.

[0157] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0158] Step 1: Record the conversation

[0159] Device: Activates a function to record the user's everyday conversations and dialogue scenes. For example, by using a smartphone application, a user can start recording when talking to a friend at home, asking, "What games have you been playing recently?" The input is environmental sounds and human speech, and the output is digital audio data.

[0160] Step 2: Convert the audio data

[0161] Device: The recorded voice data is converted into text data using a speech recognition system such as the Google Cloud Speech-to-Text API. During the conversion process, the voice signal is analyzed and converted into text data. The input is digital voice data, and the output is text data such as "What games have you been playing lately?"

[0162] Step 3: Sending data

[0163] Terminal: Transmits the converted text data to the server. The input is text data, and the output is a digital message sent to the server via the Internet.

[0164] Step 4: Analyzing the text data

[0165] Server: The received text data is analyzed using a natural language processing engine such as the Google Cloud Natural Language API. During the analysis process, the user's interests and characteristics are extracted. The input is text data, and the output is user profile information such as "I like games" or "I'm interested in RPGs."

[0166] Step 5: Creating a User Profile

[0167] Server: Creates a user profile based on the analyzed interests and characteristics and stores it in a database. The input is the user's interests and characteristics data, and the output is a user profile stored in the database.

[0168] Step 6: Record real-time conversations

[0169] Terminal: Records speech in situations where it is difficult for the user to speak, such as when a teacher asks a question in a school class, such as "Please explain why you are going to work on the next assignment." The input is real-time speech, and the output is recorded audio data.

[0170] Step 7: Re-encode the audio data

[0171] Terminal: The recorded voice data is converted into text data using a voice recognition system and sent to the server. The input is the recorded voice data, and the output is the text data and its transmission.

[0172] Step 8: Generate candidate responses

[0173] Server: Based on the received text data and user profile, a generative AI model (e.g., OpenAI's GPT-3) is used to generate appropriate response candidates. The input is the text data and the user profile, and the output is response candidates such as "This assignment seems interesting because it allows me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[0174] Step 9: Suggesting possible responses

[0175] Terminal: A screen display that presents the generated response candidates to the user. The input is the response candidate data, and the output is the on-screen options that the user can see.

[0176] Step 10: Selecting the User

[0177] User: Select the appropriate response from the presented response candidates. The input is the response candidates on the screen, and the output is the user's selected data.

[0178] Step 11: Submit Selected Data

[0179] Terminal: Sends the user-selected response to the server. The input is the user-selected data, and the output is the selected data sent to the server.

[0180] Step 12: Speech synthesis

[0181] Server: The selected response is converted into audio data that resembles the user's voice using speech synthesis technology such as the Google Cloud Text-to-Speech API. The input is the selected text data, and the output is audio data.

[0182] Step 13: Sending audio data

[0183] Server: Sends the generated voice data to the terminal. The input is the voice data, and the output is the data sent to the terminal.

[0184] Step 14: Playing Audio

[0185] Terminal: The terminal plays the received voice data through a device such as a speaker or headphones and speaks on behalf of the user. The input is the voice data, and the output is the user's response, which is played as sound.

[0186] In this way, the system of the present invention makes it possible for the user to utter an appropriate response in his or her own voice even in situations where it is difficult for the user to speak.

[0187] (Application example 1)

[0188] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0189] Many people today use food delivery services, but people with anxiety disorders such as selective mutism have difficulty ordering by voice. This makes it difficult to use food delivery services smoothly. In particular, when selecting order contents based on the user's preferences or when confirming the contents by voice, problems arise in which the system is unable to respond appropriately. A system that can solve these problems is needed.

[0190] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0191] In this invention, the server includes means for recording conversation data from the user's daily life, means for analyzing the recorded conversation data and interpreting the user's hobbies, preferences, and characteristics, means for generating candidate responses based on the user's profile and input keywords, means for outputting the generated candidate responses as the user's voice using speech recognition and speech synthesis technology, and food delivery order support system means for automatically transmitting order details based on the user's selection and confirmation, thereby enabling even users who have difficulty speaking to order and confirm appropriate food delivery.

[0192] "Conversation data" refers to audio information of utterances and conversations that occur in the user's daily life.

[0193] "Hobbies and preferences" refers to the interests and concerns that a user has in their daily life, or the things that the user tends to avoid.

[0194] "Characteristics" refers to the user's personality, behavioral characteristics, habits, etc.

[0195] "Response candidates" refer to response options automatically generated by a generative AI model based on the user's preferences and characteristics.

[0196] "Speech recognition" refers to the technology of converting voice data into text data.

[0197] "Speech synthesis technology" refers to technology that converts text data into voice data and generates natural speaking voices.

[0198] "User profile" refers to individual information based on a user's hobbies, preferences, characteristics, and past behavioral history.

[0199] "Keywords" refer to specific words or phrases that users enter when placing an order.

[0200] A "food delivery order support system" refers to a system that automatically generates response candidates and outputs them in the user's voice using speech recognition and speech synthesis technology to assist users in ordering.

[0201] A "generative AI model" refers to an artificial intelligence model that generates appropriate responses to user input based on large amounts of data.

[0202] A "prompt sentence" refers to a sentence that serves as an instruction for generating response candidates based on a sentence input into a generative AI model.

[0203] MODE FOR CARRYING OUT THE INVENTION

[0204] The present invention provides a system that enables users to smoothly place orders when using a food delivery service even if they have difficulty speaking. An embodiment of this system will be described in detail below.

[0205] System configuration

[0206] 1. Conversation data recording

[0207] The device records and stores conversation data from the user's daily life, using devices such as smartphones and smart speakers. This voice data is stored in a database for later analysis.

[0208] 2. Data Analysis

[0209] The device converts the saved conversation data into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text), and the converted text data is sent to the server.

[0210] 3. Create a user profile

[0211] The server analyzes the received text data using a natural language processing engine (e.g., OpenAI GPT). Based on the analyzed data, the user's hobbies, preferences, and characteristics are extracted, and a user profile is created. The created profile includes information such as the user's favorite foods, ingredients they avoid, and their past ordering history.

[0212] 4. Real-time order support

[0213] If a user has difficulty speaking when using food delivery, they can enable the "order assistance" mode on the device, which allows the user to input the order details in keyword format without speaking.

[0214] 5. Response candidate generation

[0215] The server uses a generative AI model to generate appropriate response candidates based on the user profile and the entered keywords, and provides instructions to the AI ​​model using prompts. Examples of prompts include:

[0216] "User's favorite food: pizza. User's favorite ingredient: cheese. User's current restaurant: Pizza Hut. Generate menu response suggestions."

[0217] The generated response candidates are, for example, "Vegetable pizza (no cheese)" and "Hawaiian pizza (no cheese)."

[0218] 6. Presentation and Selection

[0219] The generated response candidates are displayed on the terminal and presented to the user, who then selects the appropriate answer from the presented options. The selected answer is then sent back to the server.

[0220] 7. Audio output and confirmation

[0221] The server uses voice synthesis technology (e.g., Google Cloud Text-to-Speech) to output the selected response as the user's voice. The generated voice data is sent back to the terminal and played as a final confirmation of the order details. The confirmation is something like, "Are you sure you want to order with this?"

[0222] 8. Completing your order

[0223] After the user confirms, the server automatically sends the order to the food delivery partner, allowing the user to place an accurate order without speaking.

[0224] Specific examples

[0225] For example, if a user wants to order a pizza, the system operates as follows:

[0226] The user selects the "order assistance" mode and inputs the text "I would like to order a pizza."

[0227] The server generates response candidates such as "vegetable pizza (no cheese)" or "Hawaiian pizza (no cheese)" based on the user's preferences and past history.

[0228] After the user selects from the displayed options, a final voice confirmation is given and the order is finally sent to the food delivery service.

[0229] In this way, the present invention can reduce the burden on users and provide a smoother ordering process through ordering support for food delivery services.

[0230] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0231] Step 1:

[0232] The device records and saves conversation data from the user's daily life. Specifically, it records the user's speech through devices such as smartphones and smart speakers. The conversations input as voice data are temporarily stored in the device's local storage.

[0233] Step 2:

[0234] The device converts the saved conversation data into text data using a speech recognition API. Specifically, it analyzes the voice data using the Google Cloud Speech-to-Text API and outputs it as text data. This text data is then sent to the server.

[0235] Step 3:

[0236] The server analyzes the received text data using a natural language processing engine. Specifically, it uses the OpenAI GPT model to extract the user's hobbies, preferences, and characteristics from the text data. A user profile is generated as a result of the analysis and stored in this profile database.

[0237] Step 4:

[0238] If a user has difficulty speaking when using food delivery, the device will activate an "order assistance" mode. The user enters the "keyword" they want to order in text format on the device. The keyword entered by the user is received as the initial input by the device.

[0239] Step 5:

[0240] The server generates candidate responses using a generative AI model based on the user profile and input keywords. The prompt is set as follows: "User's favorite food: pizza, ingredients the user wants to avoid: cheese, the restaurant the user is currently using: Pizza Hut. Please generate candidate responses for the menu." The server uses the OpenAI GPT model to generate appropriate candidate responses based on this prompt.

[0241] Step 6:

[0242] The generated response candidates are sent from the server to the terminal. The terminal displays the received response candidates on the screen and presents them to the user. The user then selects their order from the displayed options.

[0243] Step 7:

[0244] The order details selected by the user are sent from the terminal to the server again. The server then uses voice synthesis technology to convert the received order details into audio data in the user's voice. Specifically, it uses the Google Cloud Text-to-Speech API to generate natural audio data from text data.

[0245] Step 8:

[0246] The generated voice data is sent from the server to the terminal, and the terminal plays the voice data. The user makes a final confirmation by voice, asking, "Are you sure you want to order with this content?"

[0247] Step 9:

[0248] After the user confirms, the server automatically sends the order to the food delivery partner, which then sends the order in digital format to the food delivery service's API, where the order is finally fulfilled.

[0249] In this way, the system helps users smoothly order food delivery even in situations where they have difficulty speaking.

[0250] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0251] This invention is a system to help resolve selective mutism, an anxiety disorder that makes it difficult or impossible to speak in certain social environments or situations. In situations where it is difficult for the user to speak for themselves, the system combines a generative AI model and an emotion engine to generate response candidates based on the user's preferences and emotions, allowing the system to speak the content selected by the user in their own voice.

[0252] System configuration

[0253] 1. Conversation data collection

[0254] Device: Record conversations in scenes where the user can interact. For example, if a user is talking with a friend about a game at home, record this conversation.

[0255] 2. Data Analysis and Emotion Recognition

[0256] Device: The recorded voice data is converted into text data using a speech recognition API. This text data will be in the format of "What games have you been playing recently?"

[0257] Terminal: Sends text data to the server.

[0258] Server: Analyzes the received text data using a natural language processing engine to extract the user's hobbies, preferences, and characteristics.

[0259] Server: Analyzes the user's emotions from recorded voice and text data using an emotion recognition engine. For example, it detects "happiness" or "excitement" from voice tone and specific keywords.

[0260] 3. Create a user profile

[0261] Server: Creates a user profile based on the analyzed hobbies, preferences, characteristics, and emotional information. The user profile may include such things as "I like games," "I'm interested in RPGs," and "I talk with a happy attitude."

[0262] 4. Real-time conversation processing

[0263] User: In situations where it is difficult to speak (e.g., a school class), a question is asked during class. For example, the teacher might ask, "Please explain why you are working on the next assignment."

[0264] Device: The question is recorded and converted into text data using a speech recognition API. This text data is also sent to the server.

[0265] 5. Response candidate generation and emotion adaptation

[0266] Server: Based on the received text data and user profile, the generative AI model creates multiple response candidates, such as "This assignment seems interesting because it will allow me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[0267] Server: Based on the emotion recognition results, the generated response candidates are adapted to the user's current emotions. For example, if the user is feeling happy, the server can adjust the response to something like, "This task seems fun because it will allow me to learn a new perspective."

[0268] Server: Sends multiple coordinated response candidates to the device.

[0269] 6. User Selection and Speech

[0270] Terminal: Presents the received response candidates to the user through a graphical user interface (GUI), displaying responses such as "This assignment seems fun because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[0271] User: Choose the response that they feel is most appropriate from the suggested responses. For example, "This assignment seems fun because it will help me learn new perspectives."

[0272] Terminal: Sends the user-selected response to the server.

[0273] Server: Converts the selected response into audio data in the user's voice using speech synthesis technology, for example, using speech synthesis technology (e.g., WaveNet).

[0274] Server: Sends the generated audio data to the device.

[0275] Device: Plays back the received audio data and speaks on behalf of the user. For example, it might say in the user's voice, "This assignment seems fun because it allows me to learn a new perspective."

[0276] Specific examples

[0277] Example 1: Daily conversation data collection and emotion recognition

[0278] Device: At home, the user speaks to a friend, "What games have you been playing recently?" The device records this conversation, converts it into text using voice recognition, and sends it to the server.

[0279] Server: Analyzes the received data and determines that "the user is interested in the latest RPG." At the same time, the emotion engine analyzes it and recognizes that the user is "enjoying the conversation."

[0280] Example 2: Real-time conversations and emotional adaptation in schools

[0281] Device: During a class, the teacher asks, "Please explain why you are going to tackle the next assignment." The device records this question, converts it into text using voice recognition, and sends it to the server.

[0282] Server: Based on the question, it generates response candidates such as "This assignment seems interesting because it will allow me to learn a new perspective" and "Because the content of this assignment will be useful to me in the future." At the same time, the emotion engine analyzes the user's current emotions and recognizes that they are "interested."

[0283] Server: Adapts the generated response candidates to the user's emotions and generates an adjusted response such as, "This task seems fun because it will allow me to learn a new perspective."

[0284] Terminal: Sends the user-selected response to the server.

[0285] Server: Synthesizes speech using the user's voice and sends the generated voice data to the device.

[0286] Device: Plays back the received audio data and speaks on behalf of the user, "This assignment seems fun because it allows me to learn a new perspective."

[0287] In this way, by combining emotion engines, more natural responses that reflect the user's emotional state become possible, resulting in even smoother communication.

[0288] The processing flow will be explained below.

[0289] Step 1:

[0290] The user begins a conversation in an interactive scene, for example, a user talking about gaming with a friend at home.

[0291] Step 2:

[0292] The device records the conversation. The device's microphone captures what you and your friends say.

[0293] Step 3:

[0294] The voice data recorded by the device is converted into text data using a speech recognition API. This conversion results in the text data "What games have you been playing recently?"

[0295] Step 4:

[0296] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[0297] Step 5:

[0298] The server receives the text data and stores it in a database, which stores past conversation data.

[0299] Step 6:

[0300] The server analyzes the stored text data using a natural language processing engine, extracting keywords such as "game" or "RPG."

[0301] Step 7:

[0302] Based on the analysis results, the server extracts the user's hobbies, preferences, and characteristics and reflects them in the user's profile. Information such as "I like games" and "I'm interested in RPGs" is added to the user profile.

[0303] Step 8:

[0304] The server sends the voice and text data to an emotion recognition engine, which analyzes emotions from voice tone and specific keywords.

[0305] Step 9:

[0306] The server analyzes the emotion recognition results and adds the user's emotional state to the profile, for example, storing emotional information such as "the person is talking happily."

[0307] Step 10:

[0308] In situations where it is difficult for the user to speak (e.g., a school class), the teacher may ask a question during the class, such as, "Please explain why you are working on the next assignment."

[0309] Step 11:

[0310] The device records questions asked during class, and the teacher's comments are captured through the device's microphone.

[0311] Step 12:

[0312] The device converts the recorded voice data into text data using a speech recognition API. After conversion, the resulting text data reads, "Please explain why you want to tackle the following assignment."

[0313] Step 13:

[0314] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[0315] Step 14:

[0316] The server receives the text data and compares it with the user's profile. It then creates response candidates based on the teacher's question, the user's hobbies, preferences, and emotional state.

[0317] Step 15:

[0318] The server uses a generative AI model to generate multiple response candidates, such as "This assignment seems interesting because it allows me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[0319] Step 16:

[0320] The server then uses the emotion recognition results to tailor the generated response candidates to the user's current emotions. For example, if the user is interested, the server might tailor the response to something like, "This task seems fun because it will allow me to learn a new perspective."

[0321] Step 17:

[0322] The server sends multiple coordinated response candidates to the device, and transmits the data securely using the HTTPS protocol.

[0323] Step 18:

[0324] The terminal presents the received response candidates to the user through a graphical user interface (GUI), displaying options such as "This assignment seems fun because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[0325] Step 19:

[0326] The user selects one of the suggested responses, for example, "This assignment seems fun because it allows me to learn a new perspective."

[0327] Step 20:

[0328] The device sends the user-selected response to the server, transmitting data securely using the HTTPS protocol.

[0329] Step 21:

[0330] The server converts the selected response into audio data in the user's voice using a speech synthesis technology, such as a speech synthesis technology (e.g., WaveNet).

[0331] Step 22:

[0332] The server sends the generated audio data to the device, securely using the HTTPS protocol.

[0333] Step 23:

[0334] The device plays back the received audio data, saying on behalf of the user, "This assignment seems fun because it allows me to learn a new perspective."

[0335] These are the specific program processing steps of this system. This system allows users to communicate smoothly even in situations where speaking is difficult, and by combining it with an emotion engine, it becomes possible to respond more naturally and appropriately.

[0336] Example 2

[0337] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0338] The challenge is to provide a means to support smooth communication for users with anxiety disorders, such as selective mutism, who find it difficult or impossible to speak in certain social environments or situations. In particular, it is necessary to build a system that can generate appropriate responses based on the user's own preferences and emotions and speak them in the user's own voice, even in situations where it is difficult for the user to speak.

[0339] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0340] In this invention, the server includes a means for creating a user profile based on the user's hobbies, preferences, and emotional information, a means for automatically generating candidate responses based on the utterances of others using a generative AI model, and a means for vocalizing the selected candidate responses in the user's voice using speech synthesis technology, thereby enabling the user to smoothly utter natural responses that reflect their emotions, hobbies, preferences, and emotional information in appropriate situations.

[0341] "User's daily conversation data" refers to all voice data uttered by the user in daily life.

[0342] "Recorded conversation data" refers to voice data recorded during the user's daily life.

[0343] "Hobbies and interests" refers to activities and areas of interest that a user is particularly interested in and likes to do.

[0344] "Characteristics" refers to a user's individual characteristics, personality, habits, etc.

[0345] "Speech recognition technology" refers to the technology that converts voice data into text data. Specifically, it is a technology that utilizes a speech recognition API.

[0346] "Text data" refers to text information converted using voice recognition technology.

[0347] A "server" refers to a computer system that provides functions such as data processing and storage over a network.

[0348] A "natural language processing engine" refers to technology for analyzing text data and understanding its meaning and context.

[0349] An "emotion recognition engine" refers to technology for analyzing emotions from voice or text data.

[0350] "User profile" refers to a user's individual data set created based on analyzed hobbies, preferences, characteristics, and emotional information.

[0351] A "generative AI model" refers to a model that uses artificial intelligence (AI) to generate candidate answers to questions.

[0352] "Candidate answers" refer to multiple options that indicate how a user should answer a question from another person.

[0353] "Speech synthesis technology" refers to the technology that converts text data into voice data. Specifically, it is a technology that reproduces the user's voice.

[0354] "Audio data" refers to audio files or audio signals generated using speech synthesis technology.

[0355] "Speech" refers to the act of outputting voice on behalf of the user by playing back voice data.

[0356] The present invention is a system for supporting users with anxiety disorders such as selective mutism, which makes it difficult or impossible for them to speak in certain social environments or situations. This system generates response candidates based on the user's hobbies, preferences, and emotions in situations where it is difficult for the user to speak, and speaks on the user's behalf.

[0357] The system is configured using the following hardware and software:

[0358] 1. Hardware:

[0359] Device: A device such as a smartphone or tablet used by a user.

[0360] Server: A remote server that analyzes data and synthesizes speech.

[0361] 2. Software:

[0362] Speech Recognition API: Speech recognition technologies such as Google Cloud Speech-to-Text.

[0363] Natural language processing engine: An engine for analyzing text data, such as spaCy.

[0364] Emotion recognition engine: An engine for analyzing emotions, such as IBM Watson Tone Analyzer.

[0365] Generative AI models: Generative AI models such as GPT-4.

[0366] Speech synthesis technology: Speech synthesis technology such as WaveNet.

[0367] The detailed process of the system is as follows:

[0368] Conversation data collection

[0369] The device records conversations that users have in their daily lives. For example, if a user is talking with a friend about a game at home, the device records this conversation.

[0370] Speech-to-text conversion

[0371] The device sends the recorded data to a speech recognition API and converts the speech into text. For example, the device converts the utterance, "What games are you playing these days?" into text.

[0372] Data transmission and analysis

[0373] The terminal transmits the converted text data to the server.

[0374] The server analyzes the received text data using a natural language processing engine to extract the user's hobbies, preferences, and characteristics, and then uses an emotion recognition engine to analyze the user's emotions from the voice tone and text content.

[0375] Creating a User Profile

[0376] The server creates a user profile based on the analysis results, which includes information such as "likes games," "interested in RPGs," and "is happy to talk."

[0377] Receiving and converting questions

[0378] During a class, the user is asked by a teacher, "Please explain why you are taking on the following assignment."

[0379] The device records the questions it receives and converts them into text data using a voice recognition API.

[0380] Generate and refine response candidates

[0381] The server uses a generative AI model to generate multiple response candidates based on the question text data and the user profile, such as "This assignment seems interesting because it will allow me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[0382] Based on the emotion recognition results, the generated response candidates are adapted to the user's current emotion, for example, to expressions such as "It looks like you're having fun."

[0383] Presentation and selection of response candidates

[0384] The terminal presents the adjusted response candidates to the user, displaying multiple response candidates through a GUI for the user to select from.

[0385] The user selects the most appropriate response from the presented options, for example, "This assignment seems fun because it allows me to learn a new perspective."

[0386] Speech synthesis and speech

[0387] The terminal sends the response selected by the user to the server.

[0388] The server converts the selected response into audio data in the user's voice using voice synthesis technology.

[0389] The device plays back the received voice data and speaks on behalf of the user, for example, saying, "This assignment seems fun because it allows me to learn a new perspective."

[0390] In this way, the present invention allows users to communicate naturally in situations such as selective mutism.

[0391] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0392] Step 1:

[0393] Conversation data collection

[0394] The device records conversations that users have in their daily lives. The input is the user's voice data. Specifically, the device records scenes of the user conversing with a friend through a microphone and saves the recorded voice data in local storage. The output is a recorded voice file.

[0395] Step 2:

[0396] Speech-to-text conversion

[0397] The device sends the recorded audio data to a speech recognition API, which converts the audio into text. The input is the recorded audio file. Specifically, the device calls the speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the audio file into text data. The output is the converted text data.

[0398] Step 3:

[0399] Sending data

[0400] The terminal sends the converted text data to the server. The input is text data. In concrete terms, the terminal sends the text data to the server using a secure protocol (e.g., HTTPS). The output is the text data sent to the server.

[0401] Step 4:

[0402] Analysis of tastes, preferences and emotions

[0403] The server analyzes the received text data using a natural language processing engine to extract the user's hobbies, preferences, and characteristics. The input is text data. Specifically, the server extracts keywords from the text data using a natural language processing engine such as spaCy. The output is data on the extracted user's hobbies, preferences, and characteristics.

[0404] Step 5:

[0405] emotion recognition

[0406] The server uses an emotion recognition engine to analyze the user's emotions from the voice tone and text content. The input is text data and voice data. Specifically, the server uses an emotion recognition engine such as IBM Watson Tone Analyzer to detect emotions from the text and voice. The output is the analyzed emotion data.

[0407] Step 6:

[0408] Creating a User Profile

[0409] The server creates a user profile based on the analysis results. The inputs are hobby and preference data and emotion data. Specifically, the server integrates these data and saves the user profile in a database. The output is the created user profile.

[0410] Step 7:

[0411] Receiving and converting questions

[0412] During a class, a teacher asks a user, "Please explain why you are working on the next assignment." The device records the question and converts it into text data using a speech recognition API. The input is the teacher's voice data. Specifically, the device records the question through the microphone and converts it into text using the speech recognition API. The output is the text data of the question.

[0413] Step 8:

[0414] Generate candidate responses

[0415] The server uses a generative AI model to create multiple candidate responses based on the question text data and the user profile. The input is the question text data and the user profile. Specifically, the server uses a generative AI model such as GPT-4 to generate candidate responses such as "This task seems interesting because it allows me to learn a new perspective" or "The content of this task will be useful to me in the future." The output is the text data of the generated candidate responses.

[0416] Step 9:

[0417] Emotional adaptation and regulation

[0418] The server adapts the generated response candidates to the user's current emotions. The input is the text data of the response candidates and the emotion data. Specifically, the server adjusts the expression of the response candidates based on the emotion data. For example, it changes "That sounds interesting" to "That sounds fun." The output is the adjusted response candidates.

[0419] Step 10:

[0420] Presentation and selection of response candidates

[0421] The terminal presents the adjusted response candidates to the user. The input is the adjusted response candidates. In concrete terms, a GUI is used to display multiple response candidates to the user. The user selects the one they feel is most appropriate from the presented response candidates. The output is the response candidate selected by the user.

[0422] Step 11:

[0423] Speech synthesis and speech

[0424] The terminal sends the response selected by the user to the server. The server converts the selected response into audio data in the user's voice using speech synthesis technology. The input is the selected response candidate. Specifically, the server uses speech synthesis technology such as WaveNet to convert the response text into audio data in the user's voice. The output is the generated audio data.

[0425] The device plays back the received voice data and speaks on behalf of the user, for example, saying, "This assignment seems fun because it allows me to learn a new perspective."

[0426] (Application example 2)

[0427] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0428] Previously, it was difficult for users with selective mutism to ask questions related to ads or engage in interactive responses while viewing ads. Furthermore, there was a lack of mechanisms for automatically generating appropriate responses based on the user's preferences and emotions. This prevented users from responding naturally to ads and actively participating in them. Furthermore, there was no system that could speak the generated responses in the user's own voice, making it difficult to achieve more natural and effective communication.

[0429] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0430] In this invention, the server includes means for recording conversation data from the user's daily life, means for analyzing the recorded conversation data and interpreting the user's hobbies, preferences, and characteristics, means for analyzing what others say in situations where it is difficult for the user to speak and automatically generating reply candidates based on the user's hobbies, preferences, and characteristics, means for presenting the generated reply candidates to the user and allowing them to select one, means for outputting the selected reply candidate in the user's voice, means for generating reply candidates based on a specific prompt sentence using a generative AI model for the user viewing an advertisement and presenting them to the user, and means for uttering the selected reply candidate in the user's voice in relation to the advertisement, thereby enabling the user to engage in active and natural interactions while viewing the advertisement.

[0431] "User's daily conversation data" refers to data that records utterances and conversations that the user makes in his or her daily life.

[0432] "Hobbies, preferences and characteristics" refers to information about a user's personal interests, likes and dislikes, behavioral patterns and personality traits.

[0433] A "scene in which it is difficult for the user to speak" refers to a situation or scene in which the user has difficulty speaking due to selective mutism or other reasons.

[0434] "Other people's statements" refer to words or sentences spoken by people other than the user.

[0435] "Candidate responses" refer to multiple options for answers or replies that are presented to a user as a comment.

[0436] A "generative AI model" is an artificial intelligence model that learns from large datasets and generates new documents and answers.

[0437] A "prompt sentence" is an input sentence given to a generative AI model, which serves as a reference sentence for the model to generate a response based on.

[0438] "Means for speaking in relation to an advertisement" refers to a technique or mechanism for speaking a user-selected response in relation to an advertisement in the user's voice.

[0439] The system of the present invention provides natural and appropriate responses to users, even when they have difficulty speaking while viewing advertisements, and promotes active interaction with advertisements. This system has the following functions and configuration.

[0440] System configuration

[0441] 1. Conversational data collection:

[0442] Device: Record the conversations you have in your daily life. For example, if you are talking to a friend about the latest gadgets, record this conversation.

[0443] 2. Data analysis and emotion recognition:

[0444] Device: The recorded voice data is converted into text data using a speech recognition API (for example, Google Cloud Speech-to-Text). This text data will be in the format of, for example, "What gadgets are you interested in these days?"

[0445] Terminal: Sends text data to the server.

[0446] Server: Analyzes text data using a natural language processing engine (e.g., GPT-3) and extracts user preferences and characteristics.

[0447] Server: Analyzes the user's emotions from recorded voice and text data using an emotion recognition engine (e.g., DeepAffects). For example, detects "enjoyment" or "excitement" from voice tone or specific keywords.

[0448] 3. Create a user profile:

[0449] Server: Creates a user profile based on the analyzed hobbies, preferences, characteristics, and emotional information. The user profile may include such things as "likes gadgets," "is interested in the latest technology," and "is happy to talk."

[0450] 4. Real-time conversation processing during ad viewing:

[0451] User: While the user is watching the ad, they are asked a question related to the ad. For example, the ad might ask, "What do you think about this new smartphone?"

[0452] Device: The question is recorded and converted into text data using a speech recognition API. This text data is also sent to the server.

[0453] 5. Response candidate generation and emotion adaptation:

[0454] Server: Based on the received text data and user profile, a generative AI model (e.g., GPT-3) generates multiple response candidates, such as "This smartphone looks futuristic and interesting" or "This will make my life easier."

[0455] Server: Based on the emotion recognition results, the generated response candidates are adapted to the user's current emotion. For example, if the user is feeling happy, the server will adjust the response to something like, "This smartphone looks futuristic and fun."

[0456] Server: Sends multiple coordinated response candidates to the device.

[0457] 6. User Selection and Speech:

[0458] Terminal: Presents the received response candidates to the user through a graphical user interface (GUI), for example, displaying "This smartphone looks futuristic and interesting" or "This will make my life easier."

[0459] User: Choose the response that they feel is most appropriate from the suggested responses. For example, "This smartphone looks futuristic and interesting."

[0460] Terminal: Sends the user-selected response to the server.

[0461] Server: Convert the selected response into audio data in the user's voice using speech synthesis technology (e.g., Google Text-to-Speech).

[0462] Server: Sends the generated audio data to the device.

[0463] Device: Plays back the received voice data and speaks on behalf of the user. For example, it speaks in the user's voice, saying, "This smartphone looks futuristic and interesting."

[0464] Specific examples

[0465] Interactions during the ad:

[0466] The ad asks, "What do you think about the features of the new smart refrigerator?" If the user does not want to speak for a moment, the application generates and presents potential responses to the user based on the user profile and the context of the ad, such as "This refrigerator looks futuristic and interesting" or "Having this in my kitchen would make my life more enjoyable." If the user selects one response, it is spoken in a synthesized voice.

[0467] Example prompt sentence:

[0468] "Users are watching an ad and are interested in the latest gadgets and technology. The ad asks, 'What do you think about the features of the new smart refrigerator?' If the user has difficulty answering, generate appropriate response suggestions to get them through the moment."

[0469] This system allows users to provide natural and appropriate responses while watching an advertisement, encouraging active interaction with the advertisement.

[0470] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0471] Step 1:

[0472] Conversation data collection

[0473] The device records the user's daily conversations. Specifically, for example, when the user talks with a friend about a gadget, the conversation is collected. The input is the user's voice, and the output is the recorded voice data.

[0474] Step 2:

[0475] Data analysis and emotion recognition

[0476] The device converts the recorded voice data into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). The input is the recorded voice data, and the output is the text data converted from the voice.

[0477] The terminal sends text data to the server. The input is the text data after speech recognition, and the output is the text data sent to the server.

[0478] The server analyzes the text data using a natural language processing engine (e.g., GPT-3) and extracts the user's interests, preferences, and characteristics. The input is text data, and the output is the analyzed interests, preferences, and characteristics information.

[0479] The server uses an emotion recognition engine (e.g., DeepAffects) to analyze the user's emotions from voice tone or specific keywords. The input is voice data or text data, and the output is the user's emotional information.

[0480] Step 3:

[0481] Creating a User Profile

[0482] The server creates a user profile based on the analyzed hobbies, preferences, characteristics, and emotional information. The input is hobbies, preferences, characteristics, and emotional information, and the output is a user profile. Examples of this include information such as "likes gadgets," "is interested in the latest technology," and "is happy when talking."

[0483] Step 4:

[0484] Real-time conversation processing while watching advertisements

[0485] While a user is watching an ad, a question related to the ad is displayed, for example, "What do you think about this new smartphone?" The input is the ad question, and the output is the user viewing the question.

[0486] The device records the question and converts it into text data using a speech recognition API. The input is the speech of the advertising question, and the output is text data, which is then sent to the server.

[0487] Step 5:

[0488] Response candidate generation and emotional adaptation

[0489] Based on the text data and user profile received by the server, a generative AI model (e.g., GPT-3) is used to generate multiple response candidates. The input is the text data and user profile, and the output is response candidates. For example, response candidates such as "This smartphone looks futuristic and interesting" or "This will make my life easier" are generated.

[0490] The server adapts the response candidates to the user's current emotion based on the emotion recognition results. The inputs are response candidates and emotion recognition results, and the output is emotion-adapted response candidates. For example, if the user seems to be having fun, the server will adjust the response to "This smartphone looks futuristic and fun."

[0491] The server sends the adjusted candidate responses to the terminal. The input is the adjusted candidate responses, and the output is the candidate responses sent to the terminal.

[0492] Step 6:

[0493] User selection and speech

[0494] The device presents the received response candidates to the user. The input is the adjusted response candidate, and the output is a screen display for the user to see. For example, the GUI displays "This smartphone looks futuristic and interesting" or "This will make my life easier."

[0495] The user selects the most appropriate response from the presented candidate responses. The input is the displayed candidate responses, and the output is the user's selection. For example, they can select "This smartphone looks futuristic and interesting."

[0496] The terminal sends the user's selected response to the server. The input is the user's selected information, and the output is the selected information sent to the server.

[0497] The server converts the selected response into audio data in the user's voice using speech synthesis technology (e.g., Google Text-to-Speech). The input is the selected response and the user's voice profile, and the output is the audio data.

[0498] The server sends the generated voice data to the terminal. The input is the generated voice data, and the output is the voice data sent to the terminal.

[0499] The device plays the received voice data and speaks on behalf of the user. The input is the received voice data, and the output is voice output from the speaker. For example, it speaks in the user's voice, "This smartphone looks futuristic and interesting."

[0500] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0501] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0502] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0503] [Second embodiment]

[0504] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0505] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0506] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0507] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0508] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0509] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0510] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0511] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0512] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0513] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0514] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0515] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0516] This invention is a system to help resolve selective mutism, an anxiety disorder that makes it difficult or impossible to speak in certain social environments or situations. In situations where it is difficult for the user to speak for themselves, a generative AI model is used to generate response candidates based on the user's preferences, and a mechanism is built in which the content selected by the user is spoken in the user's voice instead.

[0517] System configuration

[0518] 1. Conversation data collection

[0519] Device: Record conversations in scenes where the user can interact. For example, if a user is talking to a friend about a game at home, record this conversation.

[0520] 2. Data Analysis

[0521] Device: The recorded voice data is converted into text data using a speech recognition API. This text data will be in a format such as "What games are you playing these days?"

[0522] Terminal: Sends text data to the server.

[0523] 3. Create a user profile

[0524] Server: Analyzes the received text data using a natural language processing engine. From the analyzed data, the user's hobbies, preferences, and characteristics are extracted to create a user profile.

[0525] Server: Store information such as "I like games" or "I'm interested in RPGs" in the user profile.

[0526] 4. Real-time conversation processing

[0527] Terminal: Records a teacher's response to a question in a classroom where the user has difficulty speaking. The question is, "Please explain why you are working on the following assignment."

[0528] Device: The recorded speech is converted into text data using a speech recognition API. This text data is also sent to the server.

[0529] 5. Response candidate generation

[0530] Server: Based on the received text data and user profile, the server uses a generative AI model to automatically generate appropriate response candidates, such as "This assignment seems interesting because it will allow me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[0531] Server: Sends multiple response candidates to the device.

[0532] 6. User Selection and Speech

[0533] Terminal: Presents the received response candidates to the user. The user selects an option such as "This assignment seems interesting because it allows me to learn a new perspective."

[0534] Terminal: Sends the user-selected response to the server.

[0535] Server: Converts the selected response into audio data in the user's voice using speech synthesis technology.

[0536] Server: Sends the generated audio data to the device.

[0537] Terminal: Plays back the received voice data and speaks on behalf of the user.

[0538] Specific examples

[0539] Example 1: Collecting everyday conversation data

[0540] Device: At home, the user speaks to a friend, "What games have you been playing recently?" The device records this conversation, converts it into text using voice recognition, and sends it to the server.

[0541] Server: Analyzes the received data and determines that "the user is interested in the latest RPG."

[0542] Example 2: Real-time conversation at school

[0543] Device: During a class, the teacher asks, "Please explain why you are going to tackle the next assignment." The device records this question, converts it into text using voice recognition, and sends it to the server.

[0544] Server: Based on the question, it generates response candidates such as "This assignment seems interesting because it will allow me to learn a new perspective" and "Because the content of this assignment will be useful to me in the future" and sends them to the device.

[0545] Terminal: Sends the user-selected response to the server.

[0546] Server: Synthesizes speech using the user's voice and sends the generated voice data to the device.

[0547] Device: Plays back the received audio data and speaks on behalf of the user, "This assignment seems interesting because it will allow me to learn a new perspective."

[0548] In this way, the present invention realizes smooth communication by uttering a response appropriate to the situation in the user's voice, even in situations where it is difficult for the user to speak.

[0549] The processing flow will be explained below.

[0550] Step 1:

[0551] The user initiates a conversation in a scene where they can interact, for example, a user is talking with a friend about a game at home.

[0552] Step 2:

[0553] The device records the conversation. The device's microphone captures what you and your friends say.

[0554] Step 3:

[0555] The voice data recorded by the device is converted into text data using a speech recognition API. This conversion results in the text data "What games have you been playing recently?"

[0556] Step 4:

[0557] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[0558] Step 5:

[0559] The server receives the text data and stores it in a database, which stores past conversation data.

[0560] Step 6:

[0561] The server analyzes the stored text data using a natural language processing engine, extracting keywords such as "game" or "RPG."

[0562] Step 7:

[0563] Based on the analysis results, the server reflects the user's hobbies, preferences, and characteristics in the profile. Information such as "I like games" and "I'm interested in RPGs" is added to the user profile.

[0564] Step 8:

[0565] In situations where it is difficult for the user to speak (e.g., a school class), the teacher may ask a question during the class, such as, "Please explain why you are working on the next assignment."

[0566] Step 9:

[0567] The device records questions asked during class, and the teacher's comments are captured through the device's microphone.

[0568] Step 10:

[0569] The device converts the recorded voice data into text data using a speech recognition API. After conversion, the resulting text data reads, "Please explain why you want to tackle the following assignment."

[0570] Step 11:

[0571] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[0572] Step 12:

[0573] The server receives the text data and compares it with the user's profile. It then creates response candidates based on the teacher's question and the user's hobbies and preferences.

[0574] Step 13:

[0575] The server uses a generative AI model to generate multiple response candidates, such as "This assignment seems interesting because it allows me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[0576] Step 14:

[0577] The server sends multiple response options to the device. The data is sent securely using the HTTPS protocol.

[0578] Step 15:

[0579] The terminal presents the received response candidates to the user through a graphical user interface (GUI), displaying options such as "This assignment seems interesting because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[0580] Step 16:

[0581] The user selects one of the suggested responses, for example, "This assignment seems interesting because it will allow me to learn a new perspective."

[0582] Step 17:

[0583] The device sends the user-selected response to the server, transmitting data securely using the HTTPS protocol.

[0584] Step 18:

[0585] The server converts the selected response into audio data in the user's voice using a speech synthesis technology, such as a speech synthesis technology (e.g., WaveNet).

[0586] Step 19:

[0587] The server sends the audio data to the device. The data is sent securely using the HTTPS protocol.

[0588] Step 20:

[0589] The device plays back the received audio data, saying on behalf of the user, "This assignment seems interesting because it allows me to learn a new perspective."

[0590] The above are the specific program processing steps for implementing this system, which enables users to communicate smoothly even in situations where speaking is difficult.

[0591] Example 1

[0592] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0593] In the past, when users with anxiety disorders such as selective mutism had difficulty communicating, there was a lack of means for them to utter appropriate responses themselves, which led to difficulties and stress in social interactions.

[0594] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0595] In this invention, the server includes means for acquiring voice data from the user's daily life, means for converting the acquired voice data into text data using a voice recognition system, means for analyzing the converted text data and extracting the user's interests and characteristics, means for converting speech by others into text data using a voice recognition system and transmitting the text data to the server in situations where it is difficult for the user to speak, and automatically generating candidate responses using a generative AI model based on the received text data, means for presenting the generated candidate responses to the user for selection, means for converting the candidate response selected by the user into voice data in the user's voice using voice synthesis technology, and means for playing back the converted voice data. This makes it possible for the user to produce appropriate responses in their own voice even in situations where it is difficult for the user to speak.

[0596] "Voice data" is digital audio information that is a collection of speech from the user and others.

[0597] A "voice recognition system" is a technology or device that analyzes voice data and converts the content into text data.

[0598] "Text data" refers to character information of voice data converted by a voice recognition system.

[0599] "User interests" refers to themes or fields in which a user shows interest or concern.

[0600] "User characteristics" refers to personal characteristics such as a user's preferences, behavioral patterns, and personality.

[0601] A "server" is a computing device that receives and analyzes voice and text data and generates candidate responses.

[0602] A "generative AI model" refers to an artificial intelligence algorithm that performs natural language processing based on received data and automatically generates response candidates.

[0603] "Response candidates" are utterance options presented to the user that are automatically generated by a generative AI model.

[0604] "Speech synthesis technology" is a technology that converts text data into voice data and enables it to be played in a specific voice.

[0605] "Means for reproducing audio data" refers to a device that outputs the generated audio data as sound through speakers, headphones, etc.

[0606] The present invention is a system for helping to resolve selective mutism, an anxiety disorder that makes it difficult or impossible to speak in certain social environments or situations. Below, we will explain in detail how to implement this system.

[0607] Conversation data collection

[0608] Device: A device that records a user's everyday conversations. For example, it uses hardware such as a smartphone or a dedicated voice recorder. This allows a user to record a conversation, such as when talking with a friend about a game at home.

[0609] Data analysis

[0610] Device: The recorded voice data is converted into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text API). Through this process, the recorded conversation is obtained as text data such as "What games have you been playing recently?"

[0611] Terminal: Sends the converted text data to the server.

[0612] Creating a User Profile

[0613] Server: The received text data is analyzed using a natural language processing engine (e.g., Google Cloud Natural Language API). This analysis extracts the user's interests and characteristics. For example, the analysis result may be information such as "The user is interested in the latest RPGs." This information is saved in a database as a user profile.

[0614] Real-time conversation processing

[0615] Device: Records situations where it is difficult for the user to speak, such as questions asked in class. If a teacher asks a question like, "Please explain why you are going to tackle the next assignment," the device will record the content.

[0616] Terminal: The recorded speech is converted into text data using a voice recognition system, and this text data is sent to the server.

[0617] Response candidate generation

[0618] Server: Based on the received text data and user profile, a generative AI model (e.g., OpenAI's GPT-3) is used to generate appropriate response candidates, such as "This assignment seems interesting because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[0619] User selection and speech

[0620] Terminal: Presents the generated response candidates to the user, showing options on a screen, etc. The user selects the appropriate response from the presented candidates.

[0621] Terminal: Sends the selected response to the server.

[0622] Server: The selected response is converted into voice data that resembles the user's voice using voice synthesis technology (e.g., Google Cloud Text-to-Speech API), and this voice data is sent to the device.

[0623] Terminal: Plays back the received voice data and speaks on behalf of the user.

[0624] Specific examples

[0625] Example 1: Collecting everyday conversation data

[0626] Device: At home, a user talks to a friend, "What games have you been playing recently?" This conversation is recorded, converted into text using voice recognition, and sent to the server.

[0627] Server: Analyzes the received data and determines that "the user is interested in the latest RPGs" and reflects this in the profile.

[0628] Example 2: Real-time conversation at school

[0629] Device: During a class, a teacher asks, "Please explain why you are going to tackle the following assignment." This question is recorded, converted into text using voice recognition, and sent to a server.

[0630] Server: Based on the question, it generates response candidates such as "This assignment seems interesting because it will allow me to learn a new perspective" and "Because the content of this assignment will be useful to me in the future." It then sends these to the device.

[0631] Terminal: The user selects "This assignment seems interesting because it allows me to learn a new perspective" from the presented response candidates. After the selection, the terminal sends this content to the server.

[0632] Server: The selected content is synthesized using the user's voice and the generated voice data is sent to the terminal.

[0633] Terminal: Plays back audio data and speaks on behalf of the user.

[0634] Example prompts to input to the generative AI model

[0635] "When a teacher asks a user why they are taking on the next assignment, generate appropriate response suggestions based on the user profile: The user likes games and is interested in RPGs."

[0636] As a result, the present invention can achieve smooth communication even in situations where it is difficult for the user to speak by uttering a response appropriate to the situation in the user's voice.

[0637] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0638] Step 1: Record the conversation

[0639] Device: Activates a function to record the user's everyday conversations and dialogue scenes. For example, by using a smartphone application, a user can start recording when talking to a friend at home, asking, "What games have you been playing recently?" The input is environmental sounds and human speech, and the output is digital audio data.

[0640] Step 2: Convert the audio data

[0641] Device: The recorded voice data is converted into text data using a speech recognition system such as the Google Cloud Speech-to-Text API. During the conversion process, the voice signal is analyzed and converted into text data. The input is digital voice data, and the output is text data such as "What games have you been playing lately?"

[0642] Step 3: Sending data

[0643] Terminal: Transmits the converted text data to the server. The input is text data, and the output is a digital message sent to the server via the Internet.

[0644] Step 4: Analyzing the text data

[0645] Server: The received text data is analyzed using a natural language processing engine such as the Google Cloud Natural Language API. During the analysis process, the user's interests and characteristics are extracted. The input is text data, and the output is user profile information such as "I like games" or "I'm interested in RPGs."

[0646] Step 5: Creating a User Profile

[0647] Server: Creates a user profile based on the analyzed interests and characteristics and stores it in a database. The input is the user's interests and characteristics data, and the output is a user profile stored in the database.

[0648] Step 6: Record real-time conversations

[0649] Terminal: Records speech in situations where it is difficult for the user to speak, such as when a teacher asks a question in a school class, such as "Please explain why you are going to work on the next assignment." The input is real-time speech, and the output is recorded audio data.

[0650] Step 7: Re-encode the audio data

[0651] Terminal: The recorded voice data is converted into text data using a voice recognition system and sent to the server. The input is the recorded voice data, and the output is the text data and its transmission.

[0652] Step 8: Generate candidate responses

[0653] Server: Based on the received text data and user profile, a generative AI model (e.g., OpenAI's GPT-3) is used to generate appropriate response candidates. The input is the text data and the user profile, and the output is response candidates such as "This assignment seems interesting because it allows me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[0654] Step 9: Suggesting possible responses

[0655] Terminal: A screen display that presents the generated response candidates to the user. The input is the response candidate data, and the output is the on-screen options that the user can see.

[0656] Step 10: Selecting the User

[0657] User: Select the appropriate response from the presented response candidates. The input is the response candidates on the screen, and the output is the user's selected data.

[0658] Step 11: Submit Selected Data

[0659] Terminal: Sends the user-selected response to the server. The input is the user-selected data, and the output is the selected data sent to the server.

[0660] Step 12: Speech synthesis

[0661] Server: The selected response is converted into audio data that resembles the user's voice using speech synthesis technology such as the Google Cloud Text-to-Speech API. The input is the selected text data, and the output is audio data.

[0662] Step 13: Sending audio data

[0663] Server: Sends the generated voice data to the terminal. The input is the voice data, and the output is the data sent to the terminal.

[0664] Step 14: Playing Audio

[0665] Terminal: The terminal plays the received voice data through a device such as a speaker or headphones and speaks on behalf of the user. The input is the voice data, and the output is the user's response, which is played as sound.

[0666] In this way, the system of the present invention makes it possible for the user to utter an appropriate response in his or her own voice even in situations where it is difficult for the user to speak.

[0667] (Application example 1)

[0668] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0669] Many people today use food delivery services, but people with anxiety disorders such as selective mutism have difficulty ordering by voice. This makes it difficult to use food delivery services smoothly. In particular, when selecting order contents based on the user's preferences or when confirming the contents by voice, problems arise in which the system is unable to respond appropriately. A system that can solve these problems is needed.

[0670] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0671] In this invention, the server includes means for recording conversation data from the user's daily life, means for analyzing the recorded conversation data and interpreting the user's hobbies, preferences, and characteristics, means for generating candidate responses based on the user's profile and input keywords, means for outputting the generated candidate responses as the user's voice using speech recognition and speech synthesis technology, and food delivery order support system means for automatically transmitting order details based on the user's selection and confirmation, thereby enabling even users who have difficulty speaking to order and confirm appropriate food delivery.

[0672] "Conversation data" refers to audio information of utterances and conversations that occur in the user's daily life.

[0673] "Hobbies and preferences" refers to the interests and concerns that a user has in their daily life, or the things that the user tends to avoid.

[0674] "Characteristics" refers to the user's personality, behavioral characteristics, habits, etc.

[0675] "Response candidates" refer to response options automatically generated by a generative AI model based on the user's preferences and characteristics.

[0676] "Speech recognition" refers to the technology of converting voice data into text data.

[0677] "Speech synthesis technology" refers to technology that converts text data into voice data and generates natural speaking voices.

[0678] "User profile" refers to individual information based on a user's hobbies, preferences, characteristics, and past behavioral history.

[0679] "Keywords" refer to specific words or phrases that users enter when placing an order.

[0680] A "food delivery order support system" refers to a system that automatically generates response candidates and outputs them in the user's voice using speech recognition and speech synthesis technology to assist users in ordering.

[0681] A "generative AI model" refers to an artificial intelligence model that generates appropriate responses to user input based on large amounts of data.

[0682] A "prompt sentence" refers to a sentence that serves as an instruction for generating response candidates based on a sentence input into a generative AI model.

[0683] MODE FOR CARRYING OUT THE INVENTION

[0684] The present invention provides a system that enables users to smoothly place orders when using a food delivery service even if they have difficulty speaking. An embodiment of this system will be described in detail below.

[0685] System configuration

[0686] 1. Conversation data recording

[0687] The device records and stores conversation data from the user's daily life, using devices such as smartphones and smart speakers. This voice data is stored in a database for later analysis.

[0688] 2. Data Analysis

[0689] The device converts the saved conversation data into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text), and the converted text data is sent to the server.

[0690] 3. Create a user profile

[0691] The server analyzes the received text data using a natural language processing engine (e.g., OpenAI GPT). Based on the analyzed data, the user's hobbies, preferences, and characteristics are extracted, and a user profile is created. The created profile includes information such as the user's favorite foods, ingredients they avoid, and their past ordering history.

[0692] 4. Real-time order support

[0693] If a user has difficulty speaking when using food delivery, they can enable the "order assistance" mode on the device, which allows the user to input the order details in keyword format without speaking.

[0694] 5. Response candidate generation

[0695] The server uses a generative AI model to generate appropriate response candidates based on the user profile and the entered keywords, and provides instructions to the AI ​​model using prompts. Examples of prompts include:

[0696] "User's favorite food: pizza. User's favorite ingredient: cheese. User's current restaurant: Pizza Hut. Generate menu response suggestions."

[0697] The generated response candidates are, for example, "Vegetable pizza (no cheese)" and "Hawaiian pizza (no cheese)."

[0698] 6. Presentation and Selection

[0699] The generated response candidates are displayed on the terminal and presented to the user, who then selects the appropriate answer from the presented options. The selected answer is then sent back to the server.

[0700] 7. Audio output and confirmation

[0701] The server uses voice synthesis technology (e.g., Google Cloud Text-to-Speech) to output the selected response as the user's voice. The generated voice data is sent back to the terminal and played as a final confirmation of the order details. The confirmation is something like, "Are you sure you want to order with this?"

[0702] 8. Completing your order

[0703] After the user confirms, the server automatically sends the order to the food delivery partner, allowing the user to place an accurate order without speaking.

[0704] Specific examples

[0705] For example, if a user wants to order a pizza, the system operates as follows:

[0706] The user selects the "order assistance" mode and inputs the text "I would like to order a pizza."

[0707] The server generates response candidates such as "vegetable pizza (no cheese)" or "Hawaiian pizza (no cheese)" based on the user's preferences and past history.

[0708] After the user selects from the displayed options, a final voice confirmation is given and the order is finally sent to the food delivery service.

[0709] In this way, the present invention can reduce the burden on users and provide a smoother ordering process through ordering support for food delivery services.

[0710] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0711] Step 1:

[0712] The device records and saves conversation data from the user's daily life. Specifically, it records the user's speech through devices such as smartphones and smart speakers. The conversations input as voice data are temporarily stored in the device's local storage.

[0713] Step 2:

[0714] The device converts the saved conversation data into text data using a speech recognition API. Specifically, it analyzes the voice data using the Google Cloud Speech-to-Text API and outputs it as text data. This text data is then sent to the server.

[0715] Step 3:

[0716] The server analyzes the received text data using a natural language processing engine. Specifically, it uses the OpenAI GPT model to extract the user's hobbies, preferences, and characteristics from the text data. A user profile is generated as a result of the analysis and stored in this profile database.

[0717] Step 4:

[0718] If a user has difficulty speaking when using food delivery, the device will activate an "order assistance" mode. The user enters the "keyword" they want to order in text format on the device. The keyword entered by the user is received as the initial input by the device.

[0719] Step 5:

[0720] The server generates candidate responses using a generative AI model based on the user profile and input keywords. The prompt is set as follows: "User's favorite food: pizza, ingredients the user wants to avoid: cheese, the restaurant the user is currently using: Pizza Hut. Please generate candidate responses for the menu." The server uses the OpenAI GPT model to generate appropriate candidate responses based on this prompt.

[0721] Step 6:

[0722] The generated response candidates are sent from the server to the terminal. The terminal displays the received response candidates on the screen and presents them to the user. The user then selects their order from the displayed options.

[0723] Step 7:

[0724] The order details selected by the user are sent from the terminal to the server again. The server then uses voice synthesis technology to convert the received order details into audio data in the user's voice. Specifically, it uses the Google Cloud Text-to-Speech API to generate natural audio data from text data.

[0725] Step 8:

[0726] The generated voice data is sent from the server to the terminal, and the terminal plays the voice data. The user makes a final confirmation by voice, asking, "Are you sure you want to order with this content?"

[0727] Step 9:

[0728] After the user confirms, the server automatically sends the order to the food delivery partner, which then sends the order in digital format to the food delivery service's API, where the order is finally fulfilled.

[0729] In this way, the system helps users smoothly order food delivery even in situations where they have difficulty speaking.

[0730] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0731] This invention is a system to help resolve selective mutism, an anxiety disorder that makes it difficult or impossible to speak in certain social environments or situations. In situations where it is difficult for the user to speak for themselves, the system combines a generative AI model and an emotion engine to generate response candidates based on the user's preferences and emotions, allowing the system to speak the content selected by the user in their own voice.

[0732] System configuration

[0733] 1. Conversation data collection

[0734] Device: Record conversations in scenes where the user can interact. For example, if a user is talking with a friend about a game at home, record this conversation.

[0735] 2. Data Analysis and Emotion Recognition

[0736] Device: The recorded voice data is converted into text data using a speech recognition API. This text data will be in the format of "What games have you been playing recently?"

[0737] Terminal: Sends text data to the server.

[0738] Server: Analyzes the received text data using a natural language processing engine to extract the user's hobbies, preferences, and characteristics.

[0739] Server: Analyzes the user's emotions from recorded voice and text data using an emotion recognition engine. For example, it detects "happiness" or "excitement" from voice tone and specific keywords.

[0740] 3. Create a user profile

[0741] Server: Creates a user profile based on the analyzed hobbies, preferences, characteristics, and emotional information. The user profile may include such things as "I like games," "I'm interested in RPGs," and "I talk with a happy attitude."

[0742] 4. Real-time conversation processing

[0743] User: In situations where it is difficult to speak (e.g., a school class), a question is asked during class. For example, the teacher might ask, "Please explain why you are working on the next assignment."

[0744] Device: The question is recorded and converted into text data using a speech recognition API. This text data is also sent to the server.

[0745] 5. Response candidate generation and emotion adaptation

[0746] Server: Based on the received text data and user profile, the generative AI model creates multiple response candidates, such as "This assignment seems interesting because it will allow me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[0747] Server: Based on the emotion recognition results, the generated response candidates are adapted to the user's current emotions. For example, if the user is feeling happy, the server can adjust the response to something like, "This task seems fun because it will allow me to learn a new perspective."

[0748] Server: Sends multiple coordinated response candidates to the device.

[0749] 6. User Selection and Speech

[0750] Terminal: Presents the received response candidates to the user through a graphical user interface (GUI), displaying responses such as "This assignment seems fun because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[0751] User: Choose the response that they feel is most appropriate from the suggested responses. For example, "This assignment seems fun because it will help me learn new perspectives."

[0752] Terminal: Sends the user-selected response to the server.

[0753] Server: Converts the selected response into audio data in the user's voice using speech synthesis technology, for example, using speech synthesis technology (e.g., WaveNet).

[0754] Server: Sends the generated audio data to the device.

[0755] Device: Plays back the received audio data and speaks on behalf of the user. For example, it might say in the user's voice, "This assignment seems fun because it allows me to learn a new perspective."

[0756] Specific examples

[0757] Example 1: Daily conversation data collection and emotion recognition

[0758] Device: At home, the user speaks to a friend, "What games have you been playing recently?" The device records this conversation, converts it into text using voice recognition, and sends it to the server.

[0759] Server: Analyzes the received data and determines that "the user is interested in the latest RPG." At the same time, the emotion engine analyzes it and recognizes that the user is "enjoying the conversation."

[0760] Example 2: Real-time conversations and emotional adaptation in schools

[0761] Device: During a class, the teacher asks, "Please explain why you are going to tackle the next assignment." The device records this question, converts it into text using voice recognition, and sends it to the server.

[0762] Server: Based on the question, it generates response candidates such as "This assignment seems interesting because it will allow me to learn a new perspective" and "Because the content of this assignment will be useful to me in the future." At the same time, the emotion engine analyzes the user's current emotions and recognizes that they are "interested."

[0763] Server: Adapts the generated response candidates to the user's emotions and generates an adjusted response such as, "This task seems fun because it will allow me to learn a new perspective."

[0764] Terminal: Sends the user-selected response to the server.

[0765] Server: Synthesizes speech using the user's voice and sends the generated voice data to the device.

[0766] Device: Plays back the received audio data and speaks on behalf of the user, "This assignment seems fun because it allows me to learn a new perspective."

[0767] In this way, by combining emotion engines, more natural responses that reflect the user's emotional state become possible, resulting in even smoother communication.

[0768] The processing flow will be explained below.

[0769] Step 1:

[0770] The user begins a conversation in an interactive scene, for example, a user talking about gaming with a friend at home.

[0771] Step 2:

[0772] The device records the conversation. The device's microphone captures what you and your friends say.

[0773] Step 3:

[0774] The voice data recorded by the device is converted into text data using a speech recognition API. This conversion results in the text data "What games have you been playing recently?"

[0775] Step 4:

[0776] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[0777] Step 5:

[0778] The server receives the text data and stores it in a database, which stores past conversation data.

[0779] Step 6:

[0780] The server analyzes the stored text data using a natural language processing engine, extracting keywords such as "game" or "RPG."

[0781] Step 7:

[0782] Based on the analysis results, the server extracts the user's hobbies, preferences, and characteristics and reflects them in the user's profile. Information such as "I like games" and "I'm interested in RPGs" is added to the user profile.

[0783] Step 8:

[0784] The server sends the voice and text data to an emotion recognition engine, which analyzes emotions from voice tone and specific keywords.

[0785] Step 9:

[0786] The server analyzes the emotion recognition results and adds the user's emotional state to the profile, for example, storing emotional information such as "the person is talking happily."

[0787] Step 10:

[0788] In situations where it is difficult for the user to speak (e.g., a school class), the teacher may ask a question during the class, such as, "Please explain why you are working on the next assignment."

[0789] Step 11:

[0790] The device records questions asked during class, and the teacher's comments are captured through the device's microphone.

[0791] Step 12:

[0792] The device converts the recorded voice data into text data using a speech recognition API. After conversion, the resulting text data reads, "Please explain why you want to tackle the following assignment."

[0793] Step 13:

[0794] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[0795] Step 14:

[0796] The server receives the text data and compares it with the user's profile. It then creates response candidates based on the teacher's question, the user's hobbies, preferences, and emotional state.

[0797] Step 15:

[0798] The server uses a generative AI model to generate multiple response candidates, such as "This assignment seems interesting because it allows me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[0799] Step 16:

[0800] The server then uses the emotion recognition results to tailor the generated response candidates to the user's current emotions. For example, if the user is interested, the server might tailor the response to something like, "This task seems fun because it will allow me to learn a new perspective."

[0801] Step 17:

[0802] The server sends multiple coordinated response candidates to the device, and transmits the data securely using the HTTPS protocol.

[0803] Step 18:

[0804] The terminal presents the received response candidates to the user through a graphical user interface (GUI), displaying options such as "This assignment seems fun because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[0805] Step 19:

[0806] The user selects one of the suggested responses, for example, "This assignment seems fun because it allows me to learn a new perspective."

[0807] Step 20:

[0808] The device sends the user-selected response to the server, transmitting data securely using the HTTPS protocol.

[0809] Step 21:

[0810] The server converts the selected response into audio data in the user's voice using a speech synthesis technology, such as a speech synthesis technology (e.g., WaveNet).

[0811] Step 22:

[0812] The server sends the generated audio data to the device, securely using the HTTPS protocol.

[0813] Step 23:

[0814] The device plays back the received audio data, saying on behalf of the user, "This assignment seems fun because it allows me to learn a new perspective."

[0815] These are the specific program processing steps of this system. This system allows users to communicate smoothly even in situations where speaking is difficult, and by combining it with an emotion engine, it becomes possible to respond more naturally and appropriately.

[0816] Example 2

[0817] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0818] The challenge is to provide a means to support smooth communication for users with anxiety disorders, such as selective mutism, who find it difficult or impossible to speak in certain social environments or situations. In particular, it is necessary to build a system that can generate appropriate responses based on the user's own preferences and emotions and speak them in the user's own voice, even in situations where it is difficult for the user to speak.

[0819] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0820] In this invention, the server includes a means for creating a user profile based on the user's hobbies, preferences, and emotional information, a means for automatically generating candidate responses based on the utterances of others using a generative AI model, and a means for vocalizing the selected candidate responses in the user's voice using speech synthesis technology, thereby enabling the user to smoothly utter natural responses that reflect their emotions, hobbies, preferences, and emotional information in appropriate situations.

[0821] "User's daily conversation data" refers to all voice data uttered by the user in daily life.

[0822] "Recorded conversation data" refers to voice data recorded during the user's daily life.

[0823] "Hobbies and interests" refers to activities and areas of interest that a user is particularly interested in and likes to do.

[0824] "Characteristics" refers to a user's individual characteristics, personality, habits, etc.

[0825] "Speech recognition technology" refers to the technology that converts voice data into text data. Specifically, it is a technology that utilizes a speech recognition API.

[0826] "Text data" refers to text information converted using voice recognition technology.

[0827] A "server" refers to a computer system that provides functions such as data processing and storage over a network.

[0828] A "natural language processing engine" refers to technology for analyzing text data and understanding its meaning and context.

[0829] An "emotion recognition engine" refers to technology for analyzing emotions from voice or text data.

[0830] "User profile" refers to a user's individual data set created based on analyzed hobbies, preferences, characteristics, and emotional information.

[0831] A "generative AI model" refers to a model that uses artificial intelligence (AI) to generate candidate answers to questions.

[0832] "Candidate answers" refer to multiple options that indicate how a user should answer a question from another person.

[0833] "Speech synthesis technology" refers to the technology that converts text data into voice data. Specifically, it is a technology that reproduces the user's voice.

[0834] "Audio data" refers to audio files or audio signals generated using speech synthesis technology.

[0835] "Speech" refers to the act of outputting voice on behalf of the user by playing back voice data.

[0836] The present invention is a system for supporting users with anxiety disorders such as selective mutism, which makes it difficult or impossible for them to speak in certain social environments or situations. This system generates response candidates based on the user's hobbies, preferences, and emotions in situations where it is difficult for the user to speak, and speaks on the user's behalf.

[0837] The system is configured using the following hardware and software:

[0838] 1. Hardware:

[0839] Device: A device such as a smartphone or tablet used by a user.

[0840] Server: A remote server that analyzes data and synthesizes speech.

[0841] 2. Software:

[0842] Speech Recognition API: Speech recognition technologies such as Google Cloud Speech-to-Text.

[0843] Natural language processing engine: An engine for analyzing text data, such as spaCy.

[0844] Emotion recognition engine: An engine for analyzing emotions, such as IBM Watson Tone Analyzer.

[0845] Generative AI models: Generative AI models such as GPT-4.

[0846] Speech synthesis technology: Speech synthesis technology such as WaveNet.

[0847] The detailed process of the system is as follows:

[0848] Conversation data collection

[0849] The device records conversations that users have in their daily lives. For example, if a user is talking with a friend about a game at home, the device records this conversation.

[0850] Speech-to-text conversion

[0851] The device sends the recorded data to a speech recognition API and converts the speech into text. For example, the device converts the utterance, "What games are you playing these days?" into text.

[0852] Data transmission and analysis

[0853] The terminal transmits the converted text data to the server.

[0854] The server analyzes the received text data using a natural language processing engine to extract the user's hobbies, preferences, and characteristics, and then uses an emotion recognition engine to analyze the user's emotions from the voice tone and text content.

[0855] Creating a User Profile

[0856] The server creates a user profile based on the analysis results, which includes information such as "likes games," "interested in RPGs," and "is happy to talk."

[0857] Receiving and converting questions

[0858] During a class, the user is asked by a teacher, "Please explain why you are taking on the following assignment."

[0859] The device records the questions it receives and converts them into text data using a voice recognition API.

[0860] Generate and refine response candidates

[0861] The server uses a generative AI model to generate multiple response candidates based on the question text data and the user profile, such as "This assignment seems interesting because it will allow me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[0862] Based on the emotion recognition results, the generated response candidates are adapted to the user's current emotion, for example, to expressions such as "It looks like you're having fun."

[0863] Presentation and selection of response candidates

[0864] The terminal presents the adjusted response candidates to the user, displaying multiple response candidates through a GUI for the user to select from.

[0865] The user selects the most appropriate response from the presented options, for example, "This assignment seems fun because it allows me to learn a new perspective."

[0866] Speech synthesis and speech

[0867] The terminal sends the response selected by the user to the server.

[0868] The server converts the selected response into audio data in the user's voice using voice synthesis technology.

[0869] The device plays back the received voice data and speaks on behalf of the user, for example, saying, "This assignment seems fun because it allows me to learn a new perspective."

[0870] In this way, the present invention allows users to communicate naturally in situations such as selective mutism.

[0871] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0872] Step 1:

[0873] Conversation data collection

[0874] The device records conversations that users have in their daily lives. The input is the user's voice data. Specifically, the device records scenes of the user conversing with a friend through a microphone and saves the recorded voice data in local storage. The output is a recorded voice file.

[0875] Step 2:

[0876] Speech-to-text conversion

[0877] The device sends the recorded audio data to a speech recognition API, which converts the audio into text. The input is the recorded audio file. Specifically, the device calls the speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the audio file into text data. The output is the converted text data.

[0878] Step 3:

[0879] Sending data

[0880] The terminal sends the converted text data to the server. The input is text data. In concrete terms, the terminal sends the text data to the server using a secure protocol (e.g., HTTPS). The output is the text data sent to the server.

[0881] Step 4:

[0882] Analysis of tastes, preferences and emotions

[0883] The server analyzes the received text data using a natural language processing engine to extract the user's hobbies, preferences, and characteristics. The input is text data. Specifically, the server extracts keywords from the text data using a natural language processing engine such as spaCy. The output is data on the extracted user's hobbies, preferences, and characteristics.

[0884] Step 5:

[0885] emotion recognition

[0886] The server uses an emotion recognition engine to analyze the user's emotions from the voice tone and text content. The input is text data and voice data. Specifically, the server uses an emotion recognition engine such as IBM Watson Tone Analyzer to detect emotions from the text and voice. The output is the analyzed emotion data.

[0887] Step 6:

[0888] Creating a User Profile

[0889] The server creates a user profile based on the analysis results. The inputs are hobby and preference data and emotion data. Specifically, the server integrates these data and saves the user profile in a database. The output is the created user profile.

[0890] Step 7:

[0891] Receiving and converting questions

[0892] During a class, a teacher asks a user, "Please explain why you are working on the next assignment." The device records the question and converts it into text data using a speech recognition API. The input is the teacher's voice data. Specifically, the device records the question through the microphone and converts it into text using the speech recognition API. The output is the text data of the question.

[0893] Step 8:

[0894] Generate candidate responses

[0895] The server uses a generative AI model to create multiple candidate responses based on the question text data and the user profile. The input is the question text data and the user profile. Specifically, the server uses a generative AI model such as GPT-4 to generate candidate responses such as "This task seems interesting because it allows me to learn a new perspective" or "The content of this task will be useful to me in the future." The output is the text data of the generated candidate responses.

[0896] Step 9:

[0897] Emotional adaptation and regulation

[0898] The server adapts the generated response candidates to the user's current emotions. The input is the text data of the response candidates and the emotion data. Specifically, the server adjusts the expression of the response candidates based on the emotion data. For example, it changes "That sounds interesting" to "That sounds fun." The output is the adjusted response candidates.

[0899] Step 10:

[0900] Presentation and selection of response candidates

[0901] The terminal presents the adjusted response candidates to the user. The input is the adjusted response candidates. In concrete terms, a GUI is used to display multiple response candidates to the user. The user selects the one they feel is most appropriate from the presented response candidates. The output is the response candidate selected by the user.

[0902] Step 11:

[0903] Speech synthesis and speech

[0904] The terminal sends the response selected by the user to the server. The server converts the selected response into audio data in the user's voice using speech synthesis technology. The input is the selected response candidate. Specifically, the server uses speech synthesis technology such as WaveNet to convert the response text into audio data in the user's voice. The output is the generated audio data.

[0905] The device plays back the received voice data and speaks on behalf of the user, for example, saying, "This assignment seems fun because it allows me to learn a new perspective."

[0906] (Application example 2)

[0907] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0908] Previously, it was difficult for users with selective mutism to ask questions related to ads or engage in interactive responses while viewing ads. Furthermore, there was a lack of mechanisms for automatically generating appropriate responses based on the user's preferences and emotions. This prevented users from responding naturally to ads and actively participating in them. Furthermore, there was no system that could speak the generated responses in the user's own voice, making it difficult to achieve more natural and effective communication.

[0909] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0910] In this invention, the server includes means for recording conversation data from the user's daily life, means for analyzing the recorded conversation data and interpreting the user's hobbies, preferences, and characteristics, means for analyzing what others say in situations where it is difficult for the user to speak and automatically generating reply candidates based on the user's hobbies, preferences, and characteristics, means for presenting the generated reply candidates to the user and allowing them to select one, means for outputting the selected reply candidate in the user's voice, means for generating reply candidates based on a specific prompt sentence using a generative AI model for the user viewing an advertisement and presenting them to the user, and means for uttering the selected reply candidate in the user's voice in relation to the advertisement, thereby enabling the user to engage in active and natural interactions while viewing the advertisement.

[0911] "User's daily conversation data" refers to data that records utterances and conversations that the user makes in his or her daily life.

[0912] "Hobbies, preferences and characteristics" refers to information about a user's personal interests, likes and dislikes, behavioral patterns and personality traits.

[0913] A "scene in which it is difficult for the user to speak" refers to a situation or scene in which the user has difficulty speaking due to selective mutism or other reasons.

[0914] "Other people's statements" refer to words or sentences spoken by people other than the user.

[0915] "Candidate responses" refer to multiple options for answers or replies that are presented to a user as a comment.

[0916] A "generative AI model" is an artificial intelligence model that learns from large datasets and generates new documents and answers.

[0917] A "prompt sentence" is an input sentence given to a generative AI model, which serves as a reference sentence for the model to generate a response based on.

[0918] "Means for speaking in relation to an advertisement" refers to a technique or mechanism for speaking a user-selected response in relation to an advertisement in the user's voice.

[0919] The system of the present invention provides natural and appropriate responses to users, even when they have difficulty speaking while viewing advertisements, and promotes active interaction with advertisements. This system has the following functions and configuration.

[0920] System configuration

[0921] 1. Conversational data collection:

[0922] Device: Record the conversations you have in your daily life. For example, if you are talking to a friend about the latest gadgets, record this conversation.

[0923] 2. Data analysis and emotion recognition:

[0924] Device: The recorded voice data is converted into text data using a speech recognition API (for example, Google Cloud Speech-to-Text). This text data will be in the format of, for example, "What gadgets are you interested in these days?"

[0925] Terminal: Sends text data to the server.

[0926] Server: Analyzes text data using a natural language processing engine (e.g., GPT-3) and extracts user preferences and characteristics.

[0927] Server: Analyzes the user's emotions from recorded voice and text data using an emotion recognition engine (e.g., DeepAffects). For example, detects "enjoyment" or "excitement" from voice tone or specific keywords.

[0928] 3. Create a user profile:

[0929] Server: Creates a user profile based on the analyzed hobbies, preferences, characteristics, and emotional information. The user profile may include such things as "likes gadgets," "is interested in the latest technology," and "is happy to talk."

[0930] 4. Real-time conversation processing during ad viewing:

[0931] User: While the user is watching the ad, they are asked a question related to the ad. For example, the ad might ask, "What do you think about this new smartphone?"

[0932] Device: The question is recorded and converted into text data using a speech recognition API. This text data is also sent to the server.

[0933] 5. Response candidate generation and emotion adaptation:

[0934] Server: Based on the received text data and user profile, a generative AI model (e.g., GPT-3) generates multiple response candidates, such as "This smartphone looks futuristic and interesting" or "This will make my life easier."

[0935] Server: Based on the emotion recognition results, the generated response candidates are adapted to the user's current emotion. For example, if the user is feeling happy, the server will adjust the response to something like, "This smartphone looks futuristic and fun."

[0936] Server: Sends multiple coordinated response candidates to the device.

[0937] 6. User Selection and Speech:

[0938] Terminal: Presents the received response candidates to the user through a graphical user interface (GUI), for example, displaying "This smartphone looks futuristic and interesting" or "This will make my life easier."

[0939] User: Choose the response that they feel is most appropriate from the suggested responses. For example, "This smartphone looks futuristic and interesting."

[0940] Terminal: Sends the user-selected response to the server.

[0941] Server: Convert the selected response into audio data in the user's voice using speech synthesis technology (e.g., Google Text-to-Speech).

[0942] Server: Sends the generated audio data to the device.

[0943] Device: Plays back the received voice data and speaks on behalf of the user. For example, it speaks in the user's voice, saying, "This smartphone looks futuristic and interesting."

[0944] Specific examples

[0945] Interactions during the ad:

[0946] The ad asks, "What do you think about the features of the new smart refrigerator?" If the user does not want to speak for a moment, the application generates and presents potential responses to the user based on the user profile and the context of the ad, such as "This refrigerator looks futuristic and interesting" or "Having this in my kitchen would make my life more enjoyable." If the user selects one response, it is spoken in a synthesized voice.

[0947] Example prompt sentence:

[0948] "Users are watching an ad and are interested in the latest gadgets and technology. The ad asks, 'What do you think about the features of the new smart refrigerator?' If the user has difficulty answering, generate appropriate response suggestions to get them through the moment."

[0949] This system allows users to provide natural and appropriate responses while watching an advertisement, encouraging active interaction with the advertisement.

[0950] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0951] Step 1:

[0952] Conversation data collection

[0953] The device records the user's daily conversations. Specifically, for example, when the user talks with a friend about a gadget, the conversation is collected. The input is the user's voice, and the output is the recorded voice data.

[0954] Step 2:

[0955] Data analysis and emotion recognition

[0956] The device converts the recorded voice data into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). The input is the recorded voice data, and the output is the text data converted from the voice.

[0957] The terminal sends text data to the server. The input is the text data after speech recognition, and the output is the text data sent to the server.

[0958] The server analyzes the text data using a natural language processing engine (e.g., GPT-3) and extracts the user's interests, preferences, and characteristics. The input is text data, and the output is the analyzed interests, preferences, and characteristics information.

[0959] The server uses an emotion recognition engine (e.g., DeepAffects) to analyze the user's emotions from voice tone or specific keywords. The input is voice data or text data, and the output is the user's emotional information.

[0960] Step 3:

[0961] Creating a User Profile

[0962] The server creates a user profile based on the analyzed hobbies, preferences, characteristics, and emotional information. The input is hobbies, preferences, characteristics, and emotional information, and the output is a user profile. Examples of this include information such as "likes gadgets," "is interested in the latest technology," and "is happy when talking."

[0963] Step 4:

[0964] Real-time conversation processing while watching advertisements

[0965] While a user is watching an ad, a question related to the ad is displayed, for example, "What do you think about this new smartphone?" The input is the ad question, and the output is the user viewing the question.

[0966] The device records the question and converts it into text data using a speech recognition API. The input is the speech of the advertising question, and the output is text data, which is then sent to the server.

[0967] Step 5:

[0968] Response candidate generation and emotional adaptation

[0969] Based on the text data and user profile received by the server, a generative AI model (e.g., GPT-3) is used to generate multiple response candidates. The input is the text data and user profile, and the output is response candidates. For example, response candidates such as "This smartphone looks futuristic and interesting" or "This will make my life easier" are generated.

[0970] The server adapts the response candidates to the user's current emotion based on the emotion recognition results. The inputs are response candidates and emotion recognition results, and the output is emotion-adapted response candidates. For example, if the user seems to be having fun, the server will adjust the response to "This smartphone looks futuristic and fun."

[0971] The server sends the adjusted candidate responses to the terminal. The input is the adjusted candidate responses, and the output is the candidate responses sent to the terminal.

[0972] Step 6:

[0973] User selection and speech

[0974] The device presents the received response candidates to the user. The input is the adjusted response candidate, and the output is a screen display for the user to see. For example, the GUI displays "This smartphone looks futuristic and interesting" or "This will make my life easier."

[0975] The user selects the most appropriate response from the presented candidate responses. The input is the displayed candidate responses, and the output is the user's selection. For example, they can select "This smartphone looks futuristic and interesting."

[0976] The terminal sends the user's selected response to the server. The input is the user's selected information, and the output is the selected information sent to the server.

[0977] The server converts the selected response into audio data in the user's voice using speech synthesis technology (e.g., Google Text-to-Speech). The input is the selected response and the user's voice profile, and the output is the audio data.

[0978] The server sends the generated voice data to the terminal. The input is the generated voice data, and the output is the voice data sent to the terminal.

[0979] The device plays the received voice data and speaks on behalf of the user. The input is the received voice data, and the output is voice output from the speaker. For example, it speaks in the user's voice, "This smartphone looks futuristic and interesting."

[0980] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0981] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0982] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0983] [Third embodiment]

[0984] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0985] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0986] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0987] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0988] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0989] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0990] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0991] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0992] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0993] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0994] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0995] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0996] This invention is a system to help resolve selective mutism, an anxiety disorder that makes it difficult or impossible to speak in certain social environments or situations. In situations where it is difficult for the user to speak for themselves, a generative AI model is used to generate response candidates based on the user's preferences, and a mechanism is built in which the content selected by the user is spoken in the user's voice instead.

[0997] System configuration

[0998] 1. Conversation data collection

[0999] Device: Record conversations in scenes where the user can interact. For example, if a user is talking to a friend about a game at home, record this conversation.

[1000] 2. Data Analysis

[1001] Device: The recorded voice data is converted into text data using a speech recognition API. This text data will be in a format such as "What games are you playing these days?"

[1002] Terminal: Sends text data to the server.

[1003] 3. Create a user profile

[1004] Server: Analyzes the received text data using a natural language processing engine. From the analyzed data, the user's hobbies, preferences, and characteristics are extracted to create a user profile.

[1005] Server: Store information such as "I like games" or "I'm interested in RPGs" in the user profile.

[1006] 4. Real-time conversation processing

[1007] Terminal: Records a teacher's response to a question in a classroom where the user has difficulty speaking. The question is, "Please explain why you are working on the following assignment."

[1008] Device: The recorded speech is converted into text data using a speech recognition API. This text data is also sent to the server.

[1009] 5. Response candidate generation

[1010] Server: Based on the received text data and user profile, the server uses a generative AI model to automatically generate appropriate response candidates, such as "This assignment seems interesting because it will allow me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[1011] Server: Sends multiple response candidates to the device.

[1012] 6. User Selection and Speech

[1013] Terminal: Presents the received response candidates to the user. The user selects an option such as "This assignment seems interesting because it allows me to learn a new perspective."

[1014] Terminal: Sends the user-selected response to the server.

[1015] Server: Converts the selected response into audio data in the user's voice using speech synthesis technology.

[1016] Server: Sends the generated audio data to the device.

[1017] Terminal: Plays back the received voice data and speaks on behalf of the user.

[1018] Specific examples

[1019] Example 1: Collecting everyday conversation data

[1020] Device: At home, the user speaks to a friend, "What games have you been playing recently?" The device records this conversation, converts it into text using voice recognition, and sends it to the server.

[1021] Server: Analyzes the received data and determines that "the user is interested in the latest RPG."

[1022] Example 2: Real-time conversation at school

[1023] Device: During a class, the teacher asks, "Please explain why you are going to tackle the next assignment." The device records this question, converts it into text using voice recognition, and sends it to the server.

[1024] Server: Based on the question, it generates response candidates such as "This assignment seems interesting because it will allow me to learn a new perspective" and "Because the content of this assignment will be useful to me in the future" and sends them to the device.

[1025] Terminal: Sends the user-selected response to the server.

[1026] Server: Synthesizes speech using the user's voice and sends the generated voice data to the device.

[1027] Device: Plays back the received audio data and speaks on behalf of the user, "This assignment seems interesting because it will allow me to learn a new perspective."

[1028] In this way, the present invention realizes smooth communication by uttering a response appropriate to the situation in the user's voice, even in situations where it is difficult for the user to speak.

[1029] The processing flow will be explained below.

[1030] Step 1:

[1031] The user initiates a conversation in a scene where they can interact, for example, a user is talking with a friend about a game at home.

[1032] Step 2:

[1033] The device records the conversation. The device's microphone captures what you and your friends say.

[1034] Step 3:

[1035] The voice data recorded by the device is converted into text data using a speech recognition API. This conversion results in the text data "What games have you been playing recently?"

[1036] Step 4:

[1037] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[1038] Step 5:

[1039] The server receives the text data and stores it in a database, which stores past conversation data.

[1040] Step 6:

[1041] The server analyzes the stored text data using a natural language processing engine, extracting keywords such as "game" or "RPG."

[1042] Step 7:

[1043] Based on the analysis results, the server reflects the user's hobbies, preferences, and characteristics in the profile. Information such as "I like games" and "I'm interested in RPGs" is added to the user profile.

[1044] Step 8:

[1045] In situations where it is difficult for the user to speak (e.g., a school class), the teacher may ask a question during the class, such as, "Please explain why you are working on the next assignment."

[1046] Step 9:

[1047] The device records questions asked during class, and the teacher's comments are captured through the device's microphone.

[1048] Step 10:

[1049] The device converts the recorded voice data into text data using a speech recognition API. After conversion, the resulting text data reads, "Please explain why you want to tackle the following assignment."

[1050] Step 11:

[1051] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[1052] Step 12:

[1053] The server receives the text data and compares it with the user's profile. It then creates response candidates based on the teacher's question and the user's hobbies and preferences.

[1054] Step 13:

[1055] The server uses a generative AI model to generate multiple response candidates, such as "This assignment seems interesting because it allows me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[1056] Step 14:

[1057] The server sends multiple response options to the device. The data is sent securely using the HTTPS protocol.

[1058] Step 15:

[1059] The terminal presents the received response candidates to the user through a graphical user interface (GUI), displaying options such as "This assignment seems interesting because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[1060] Step 16:

[1061] The user selects one of the suggested responses, for example, "This assignment seems interesting because it will allow me to learn a new perspective."

[1062] Step 17:

[1063] The device sends the user-selected response to the server, transmitting data securely using the HTTPS protocol.

[1064] Step 18:

[1065] The server converts the selected response into audio data in the user's voice using a speech synthesis technology, such as a speech synthesis technology (e.g., WaveNet).

[1066] Step 19:

[1067] The server sends the audio data to the device. The data is sent securely using the HTTPS protocol.

[1068] Step 20:

[1069] The device plays back the received audio data, saying on behalf of the user, "This assignment seems interesting because it allows me to learn a new perspective."

[1070] The above are the specific program processing steps for implementing this system, which enables users to communicate smoothly even in situations where speaking is difficult.

[1071] Example 1

[1072] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1073] In the past, when users with anxiety disorders such as selective mutism had difficulty communicating, there was a lack of means for them to utter appropriate responses themselves, which led to difficulties and stress in social interactions.

[1074] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1075] In this invention, the server includes means for acquiring voice data from the user's daily life, means for converting the acquired voice data into text data using a voice recognition system, means for analyzing the converted text data and extracting the user's interests and characteristics, means for converting speech by others into text data using a voice recognition system and transmitting the text data to the server in situations where it is difficult for the user to speak, and automatically generating candidate responses using a generative AI model based on the received text data, means for presenting the generated candidate responses to the user for selection, means for converting the candidate response selected by the user into voice data in the user's voice using voice synthesis technology, and means for playing back the converted voice data. This makes it possible for the user to produce appropriate responses in their own voice even in situations where it is difficult for the user to speak.

[1076] "Voice data" is digital audio information that is a collection of speech from the user and others.

[1077] A "voice recognition system" is a technology or device that analyzes voice data and converts the content into text data.

[1078] "Text data" refers to character information of voice data converted by a voice recognition system.

[1079] "User interests" refers to themes or fields in which a user shows interest or concern.

[1080] "User characteristics" refers to personal characteristics such as a user's preferences, behavioral patterns, and personality.

[1081] A "server" is a computing device that receives and analyzes voice and text data and generates candidate responses.

[1082] A "generative AI model" refers to an artificial intelligence algorithm that performs natural language processing based on received data and automatically generates response candidates.

[1083] "Response candidates" are utterance options presented to the user that are automatically generated by a generative AI model.

[1084] "Speech synthesis technology" is a technology that converts text data into voice data and enables it to be played in a specific voice.

[1085] "Means for reproducing audio data" refers to a device that outputs the generated audio data as sound through speakers, headphones, etc.

[1086] The present invention is a system for helping to resolve selective mutism, an anxiety disorder that makes it difficult or impossible to speak in certain social environments or situations. Below, we will explain in detail how to implement this system.

[1087] Conversation data collection

[1088] Device: A device that records a user's everyday conversations. For example, it uses hardware such as a smartphone or a dedicated voice recorder. This allows a user to record a conversation, such as when talking with a friend about a game at home.

[1089] Data analysis

[1090] Device: The recorded voice data is converted into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text API). Through this process, the recorded conversation is obtained as text data such as "What games have you been playing recently?"

[1091] Terminal: Sends the converted text data to the server.

[1092] Creating a User Profile

[1093] Server: The received text data is analyzed using a natural language processing engine (e.g., Google Cloud Natural Language API). This analysis extracts the user's interests and characteristics. For example, the analysis result may be information such as "The user is interested in the latest RPGs." This information is saved in a database as a user profile.

[1094] Real-time conversation processing

[1095] Device: Records situations where it is difficult for the user to speak, such as questions asked in class. If a teacher asks a question like, "Please explain why you are going to tackle the next assignment," the device will record the content.

[1096] Terminal: The recorded speech is converted into text data using a voice recognition system, and this text data is sent to the server.

[1097] Response candidate generation

[1098] Server: Based on the received text data and user profile, a generative AI model (e.g., OpenAI's GPT-3) is used to generate appropriate response candidates, such as "This assignment seems interesting because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[1099] User selection and speech

[1100] Terminal: Presents the generated response candidates to the user, showing options on a screen, etc. The user selects the appropriate response from the presented candidates.

[1101] Terminal: Sends the selected response to the server.

[1102] Server: The selected response is converted into voice data that resembles the user's voice using voice synthesis technology (e.g., Google Cloud Text-to-Speech API), and this voice data is sent to the device.

[1103] Terminal: Plays back the received voice data and speaks on behalf of the user.

[1104] Specific examples

[1105] Example 1: Collecting everyday conversation data

[1106] Device: At home, a user talks to a friend, "What games have you been playing recently?" This conversation is recorded, converted into text using voice recognition, and sent to the server.

[1107] Server: Analyzes the received data and determines that "the user is interested in the latest RPGs" and reflects this in the profile.

[1108] Example 2: Real-time conversation at school

[1109] Device: During a class, a teacher asks, "Please explain why you are going to tackle the following assignment." This question is recorded, converted into text using voice recognition, and sent to a server.

[1110] Server: Based on the question, it generates response candidates such as "This assignment seems interesting because it will allow me to learn a new perspective" and "Because the content of this assignment will be useful to me in the future." It then sends these to the device.

[1111] Terminal: The user selects "This assignment seems interesting because it allows me to learn a new perspective" from the presented response candidates. After the selection, the terminal sends this content to the server.

[1112] Server: The selected content is synthesized using the user's voice and the generated voice data is sent to the terminal.

[1113] Terminal: Plays back audio data and speaks on behalf of the user.

[1114] Example prompts to input to the generative AI model

[1115] "When a teacher asks a user why they are taking on the next assignment, generate appropriate response suggestions based on the user profile: The user likes games and is interested in RPGs."

[1116] As a result, the present invention can achieve smooth communication even in situations where it is difficult for the user to speak by uttering a response appropriate to the situation in the user's voice.

[1117] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1118] Step 1: Record the conversation

[1119] Device: Activates a function to record the user's everyday conversations and dialogue scenes. For example, by using a smartphone application, a user can start recording when talking to a friend at home, asking, "What games have you been playing recently?" The input is environmental sounds and human speech, and the output is digital audio data.

[1120] Step 2: Convert the audio data

[1121] Device: The recorded voice data is converted into text data using a speech recognition system such as the Google Cloud Speech-to-Text API. During the conversion process, the voice signal is analyzed and converted into text data. The input is digital voice data, and the output is text data such as "What games have you been playing lately?"

[1122] Step 3: Sending data

[1123] Terminal: Transmits the converted text data to the server. The input is text data, and the output is a digital message sent to the server via the Internet.

[1124] Step 4: Analyzing the text data

[1125] Server: The received text data is analyzed using a natural language processing engine such as the Google Cloud Natural Language API. During the analysis process, the user's interests and characteristics are extracted. The input is text data, and the output is user profile information such as "I like games" or "I'm interested in RPGs."

[1126] Step 5: Creating a User Profile

[1127] Server: Creates a user profile based on the analyzed interests and characteristics and stores it in a database. The input is the user's interests and characteristics data, and the output is a user profile stored in the database.

[1128] Step 6: Record real-time conversations

[1129] Terminal: Records speech in situations where it is difficult for the user to speak, such as when a teacher asks a question in a school class, such as "Please explain why you are going to work on the next assignment." The input is real-time speech, and the output is recorded audio data.

[1130] Step 7: Re-encode the audio data

[1131] Terminal: The recorded voice data is converted into text data using a voice recognition system and sent to the server. The input is the recorded voice data, and the output is the text data and its transmission.

[1132] Step 8: Generate candidate responses

[1133] Server: Based on the received text data and user profile, a generative AI model (e.g., OpenAI's GPT-3) is used to generate appropriate response candidates. The input is the text data and the user profile, and the output is response candidates such as "This assignment seems interesting because it allows me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[1134] Step 9: Suggesting possible responses

[1135] Terminal: A screen display that presents the generated response candidates to the user. The input is the response candidate data, and the output is the on-screen options that the user can see.

[1136] Step 10: Selecting the User

[1137] User: Select the appropriate response from the presented response candidates. The input is the response candidates on the screen, and the output is the user's selected data.

[1138] Step 11: Submit Selected Data

[1139] Terminal: Sends the user-selected response to the server. The input is the user-selected data, and the output is the selected data sent to the server.

[1140] Step 12: Speech synthesis

[1141] Server: The selected response is converted into audio data that resembles the user's voice using speech synthesis technology such as the Google Cloud Text-to-Speech API. The input is the selected text data, and the output is audio data.

[1142] Step 13: Sending audio data

[1143] Server: Sends the generated voice data to the terminal. The input is the voice data, and the output is the data sent to the terminal.

[1144] Step 14: Playing Audio

[1145] Terminal: The terminal plays the received voice data through a device such as a speaker or headphones and speaks on behalf of the user. The input is the voice data, and the output is the user's response, which is played as sound.

[1146] In this way, the system of the present invention makes it possible for the user to utter an appropriate response in his or her own voice even in situations where it is difficult for the user to speak.

[1147] (Application example 1)

[1148] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1149] Many people today use food delivery services, but people with anxiety disorders such as selective mutism have difficulty ordering by voice. This makes it difficult to use food delivery services smoothly. In particular, when selecting order contents based on the user's preferences or when confirming the contents by voice, problems arise in which the system is unable to respond appropriately. A system that can solve these problems is needed.

[1150] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1151] In this invention, the server includes means for recording conversation data from the user's daily life, means for analyzing the recorded conversation data and interpreting the user's hobbies, preferences, and characteristics, means for generating candidate responses based on the user's profile and input keywords, means for outputting the generated candidate responses as the user's voice using speech recognition and speech synthesis technology, and food delivery order support system means for automatically transmitting order details based on the user's selection and confirmation, thereby enabling even users who have difficulty speaking to order and confirm appropriate food delivery.

[1152] "Conversation data" refers to audio information of utterances and conversations that occur in the user's daily life.

[1153] "Hobbies and preferences" refers to the interests and concerns that a user has in their daily life, or the things that the user tends to avoid.

[1154] "Characteristics" refers to the user's personality, behavioral characteristics, habits, etc.

[1155] "Response candidates" refer to response options automatically generated by a generative AI model based on the user's preferences and characteristics.

[1156] "Speech recognition" refers to the technology of converting voice data into text data.

[1157] "Speech synthesis technology" refers to technology that converts text data into voice data and generates natural speaking voices.

[1158] "User profile" refers to individual information based on a user's hobbies, preferences, characteristics, and past behavioral history.

[1159] "Keywords" refer to specific words or phrases that users enter when placing an order.

[1160] A "food delivery order support system" refers to a system that automatically generates response candidates and outputs them in the user's voice using speech recognition and speech synthesis technology to assist users in ordering.

[1161] A "generative AI model" refers to an artificial intelligence model that generates appropriate responses to user input based on large amounts of data.

[1162] A "prompt sentence" refers to a sentence that serves as an instruction for generating response candidates based on a sentence input into a generative AI model.

[1163] MODE FOR CARRYING OUT THE INVENTION

[1164] The present invention provides a system that enables users to smoothly place orders when using a food delivery service even if they have difficulty speaking. An embodiment of this system will be described in detail below.

[1165] System configuration

[1166] 1. Conversation data recording

[1167] The device records and stores conversation data from the user's daily life, using devices such as smartphones and smart speakers. This voice data is stored in a database for later analysis.

[1168] 2. Data Analysis

[1169] The device converts the saved conversation data into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text), and the converted text data is sent to the server.

[1170] 3. Create a user profile

[1171] The server analyzes the received text data using a natural language processing engine (e.g., OpenAI GPT). Based on the analyzed data, the user's hobbies, preferences, and characteristics are extracted, and a user profile is created. The created profile includes information such as the user's favorite foods, ingredients they avoid, and their past ordering history.

[1172] 4. Real-time order support

[1173] If a user has difficulty speaking when using food delivery, they can enable the "order assistance" mode on the device, which allows the user to input the order details in keyword format without speaking.

[1174] 5. Response candidate generation

[1175] The server uses a generative AI model to generate appropriate response candidates based on the user profile and the entered keywords, and provides instructions to the AI ​​model using prompts. Examples of prompts include:

[1176] "User's favorite food: pizza. User's favorite ingredient: cheese. User's current restaurant: Pizza Hut. Generate menu response suggestions."

[1177] The generated response candidates are, for example, "Vegetable pizza (no cheese)" and "Hawaiian pizza (no cheese)."

[1178] 6. Presentation and Selection

[1179] The generated response candidates are displayed on the terminal and presented to the user, who then selects the appropriate answer from the presented options. The selected answer is then sent back to the server.

[1180] 7. Audio output and confirmation

[1181] The server uses voice synthesis technology (e.g., Google Cloud Text-to-Speech) to output the selected response as the user's voice. The generated voice data is sent back to the terminal and played as a final confirmation of the order details. The confirmation is something like, "Are you sure you want to order with this?"

[1182] 8. Completing your order

[1183] After the user confirms, the server automatically sends the order to the food delivery partner, allowing the user to place an accurate order without speaking.

[1184] Specific examples

[1185] For example, if a user wants to order a pizza, the system operates as follows:

[1186] The user selects the "order assistance" mode and inputs the text "I would like to order a pizza."

[1187] The server generates response candidates such as "vegetable pizza (no cheese)" or "Hawaiian pizza (no cheese)" based on the user's preferences and past history.

[1188] After the user selects from the displayed options, a final voice confirmation is given and the order is finally sent to the food delivery service.

[1189] In this way, the present invention can reduce the burden on users and provide a smoother ordering process through ordering support for food delivery services.

[1190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1191] Step 1:

[1192] The device records and saves conversation data from the user's daily life. Specifically, it records the user's speech through devices such as smartphones and smart speakers. The conversations input as voice data are temporarily stored in the device's local storage.

[1193] Step 2:

[1194] The device converts the saved conversation data into text data using a speech recognition API. Specifically, it analyzes the voice data using the Google Cloud Speech-to-Text API and outputs it as text data. This text data is then sent to the server.

[1195] Step 3:

[1196] The server analyzes the received text data using a natural language processing engine. Specifically, it uses the OpenAI GPT model to extract the user's hobbies, preferences, and characteristics from the text data. A user profile is generated as a result of the analysis and stored in this profile database.

[1197] Step 4:

[1198] If a user has difficulty speaking when using food delivery, the device will activate an "order assistance" mode. The user enters the "keyword" they want to order in text format on the device. The keyword entered by the user is received as the initial input by the device.

[1199] Step 5:

[1200] The server generates candidate responses using a generative AI model based on the user profile and input keywords. The prompt is set as follows: "User's favorite food: pizza, ingredients the user wants to avoid: cheese, the restaurant the user is currently using: Pizza Hut. Please generate candidate responses for the menu." The server uses the OpenAI GPT model to generate appropriate candidate responses based on this prompt.

[1201] Step 6:

[1202] The generated response candidates are sent from the server to the terminal. The terminal displays the received response candidates on the screen and presents them to the user. The user then selects their order from the displayed options.

[1203] Step 7:

[1204] The order details selected by the user are sent from the terminal to the server again. The server then uses voice synthesis technology to convert the received order details into audio data in the user's voice. Specifically, it uses the Google Cloud Text-to-Speech API to generate natural audio data from text data.

[1205] Step 8:

[1206] The generated voice data is sent from the server to the terminal, and the terminal plays the voice data. The user makes a final confirmation by voice, asking, "Are you sure you want to order with this content?"

[1207] Step 9:

[1208] After the user confirms, the server automatically sends the order to the food delivery partner, which then sends the order in digital format to the food delivery service's API, where the order is finally fulfilled.

[1209] In this way, the system helps users smoothly order food delivery even in situations where they have difficulty speaking.

[1210] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1211] This invention is a system to help resolve selective mutism, an anxiety disorder that makes it difficult or impossible to speak in certain social environments or situations. In situations where it is difficult for the user to speak for themselves, the system combines a generative AI model and an emotion engine to generate response candidates based on the user's preferences and emotions, allowing the system to speak the content selected by the user in their own voice.

[1212] System configuration

[1213] 1. Conversation data collection

[1214] Device: Record conversations in scenes where the user can interact. For example, if a user is talking with a friend about a game at home, record this conversation.

[1215] 2. Data Analysis and Emotion Recognition

[1216] Device: The recorded voice data is converted into text data using a speech recognition API. This text data will be in the format of "What games have you been playing recently?"

[1217] Terminal: Sends text data to the server.

[1218] Server: Analyzes the received text data using a natural language processing engine to extract the user's hobbies, preferences, and characteristics.

[1219] Server: Analyzes the user's emotions from recorded voice and text data using an emotion recognition engine. For example, it detects "happiness" or "excitement" from voice tone and specific keywords.

[1220] 3. Create a user profile

[1221] Server: Creates a user profile based on the analyzed hobbies, preferences, characteristics, and emotional information. The user profile may include such things as "I like games," "I'm interested in RPGs," and "I talk with a happy attitude."

[1222] 4. Real-time conversation processing

[1223] User: In situations where it is difficult to speak (e.g., a school class), a question is asked during class. For example, the teacher might ask, "Please explain why you are working on the next assignment."

[1224] Device: The question is recorded and converted into text data using a speech recognition API. This text data is also sent to the server.

[1225] 5. Response candidate generation and emotion adaptation

[1226] Server: Based on the received text data and user profile, the generative AI model creates multiple response candidates, such as "This assignment seems interesting because it will allow me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[1227] Server: Based on the emotion recognition results, the generated response candidates are adapted to the user's current emotions. For example, if the user is feeling happy, the server can adjust the response to something like, "This task seems fun because it will allow me to learn a new perspective."

[1228] Server: Sends multiple coordinated response candidates to the device.

[1229] 6. User Selection and Speech

[1230] Terminal: Presents the received response candidates to the user through a graphical user interface (GUI), displaying responses such as "This assignment seems fun because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[1231] User: Choose the response that they feel is most appropriate from the suggested responses. For example, "This assignment seems fun because it will help me learn new perspectives."

[1232] Terminal: Sends the user-selected response to the server.

[1233] Server: Converts the selected response into audio data in the user's voice using speech synthesis technology, for example, using speech synthesis technology (e.g., WaveNet).

[1234] Server: Sends the generated audio data to the device.

[1235] Device: Plays back the received audio data and speaks on behalf of the user. For example, it might say in the user's voice, "This assignment seems fun because it allows me to learn a new perspective."

[1236] Specific examples

[1237] Example 1: Daily conversation data collection and emotion recognition

[1238] Device: At home, the user speaks to a friend, "What games have you been playing recently?" The device records this conversation, converts it into text using voice recognition, and sends it to the server.

[1239] Server: Analyzes the received data and determines that "the user is interested in the latest RPG." At the same time, the emotion engine analyzes it and recognizes that the user is "enjoying the conversation."

[1240] Example 2: Real-time conversations and emotional adaptation in schools

[1241] Device: During a class, the teacher asks, "Please explain why you are going to tackle the next assignment." The device records this question, converts it into text using voice recognition, and sends it to the server.

[1242] Server: Based on the question, it generates response candidates such as "This assignment seems interesting because it will allow me to learn a new perspective" and "Because the content of this assignment will be useful to me in the future." At the same time, the emotion engine analyzes the user's current emotions and recognizes that they are "interested."

[1243] Server: Adapts the generated response candidates to the user's emotions and generates an adjusted response such as, "This task seems fun because it will allow me to learn a new perspective."

[1244] Terminal: Sends the user-selected response to the server.

[1245] Server: Synthesizes speech using the user's voice and sends the generated voice data to the device.

[1246] Device: Plays back the received audio data and speaks on behalf of the user, "This assignment seems fun because it allows me to learn a new perspective."

[1247] In this way, by combining emotion engines, more natural responses that reflect the user's emotional state become possible, resulting in even smoother communication.

[1248] The processing flow will be explained below.

[1249] Step 1:

[1250] The user begins a conversation in an interactive scene, for example, a user talking about gaming with a friend at home.

[1251] Step 2:

[1252] The device records the conversation. The device's microphone captures what you and your friends say.

[1253] Step 3:

[1254] The voice data recorded by the device is converted into text data using a speech recognition API. This conversion results in the text data "What games have you been playing recently?"

[1255] Step 4:

[1256] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[1257] Step 5:

[1258] The server receives the text data and stores it in a database, which stores past conversation data.

[1259] Step 6:

[1260] The server analyzes the stored text data using a natural language processing engine, extracting keywords such as "game" or "RPG."

[1261] Step 7:

[1262] Based on the analysis results, the server extracts the user's hobbies, preferences, and characteristics and reflects them in the user's profile. Information such as "I like games" and "I'm interested in RPGs" is added to the user profile.

[1263] Step 8:

[1264] The server sends the voice and text data to an emotion recognition engine, which analyzes emotions from voice tone and specific keywords.

[1265] Step 9:

[1266] The server analyzes the emotion recognition results and adds the user's emotional state to the profile, for example, storing emotional information such as "the person is talking happily."

[1267] Step 10:

[1268] In situations where it is difficult for the user to speak (e.g., a school class), the teacher may ask a question during the class, such as, "Please explain why you are working on the next assignment."

[1269] Step 11:

[1270] The device records questions asked during class, and the teacher's comments are captured through the device's microphone.

[1271] Step 12:

[1272] The device converts the recorded voice data into text data using a speech recognition API. After conversion, the resulting text data reads, "Please explain why you want to tackle the following assignment."

[1273] Step 13:

[1274] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[1275] Step 14:

[1276] The server receives the text data and compares it with the user's profile. It then creates response candidates based on the teacher's question, the user's hobbies, preferences, and emotional state.

[1277] Step 15:

[1278] The server uses a generative AI model to generate multiple response candidates, such as "This assignment seems interesting because it allows me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[1279] Step 16:

[1280] The server then uses the emotion recognition results to tailor the generated response candidates to the user's current emotions. For example, if the user is interested, the server might tailor the response to something like, "This task seems fun because it will allow me to learn a new perspective."

[1281] Step 17:

[1282] The server sends multiple coordinated response candidates to the device, and transmits the data securely using the HTTPS protocol.

[1283] Step 18:

[1284] The terminal presents the received response candidates to the user through a graphical user interface (GUI), displaying options such as "This assignment seems fun because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[1285] Step 19:

[1286] The user selects one of the suggested responses, for example, "This assignment seems fun because it allows me to learn a new perspective."

[1287] Step 20:

[1288] The device sends the user-selected response to the server, transmitting data securely using the HTTPS protocol.

[1289] Step 21:

[1290] The server converts the selected response into audio data in the user's voice using a speech synthesis technology, such as a speech synthesis technology (e.g., WaveNet).

[1291] Step 22:

[1292] The server sends the generated audio data to the device, securely using the HTTPS protocol.

[1293] Step 23:

[1294] The device plays back the received audio data, saying on behalf of the user, "This assignment seems fun because it allows me to learn a new perspective."

[1295] These are the specific program processing steps of this system. This system allows users to communicate smoothly even in situations where speaking is difficult, and by combining it with an emotion engine, it becomes possible to respond more naturally and appropriately.

[1296] Example 2

[1297] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1298] The challenge is to provide a means to support smooth communication for users with anxiety disorders, such as selective mutism, who find it difficult or impossible to speak in certain social environments or situations. In particular, it is necessary to build a system that can generate appropriate responses based on the user's own preferences and emotions and speak them in the user's own voice, even in situations where it is difficult for the user to speak.

[1299] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1300] In this invention, the server includes a means for creating a user profile based on the user's hobbies, preferences, and emotional information, a means for automatically generating candidate responses based on the utterances of others using a generative AI model, and a means for vocalizing the selected candidate responses in the user's voice using speech synthesis technology, thereby enabling the user to smoothly utter natural responses that reflect their emotions, hobbies, preferences, and emotional information in appropriate situations.

[1301] "User's daily conversation data" refers to all voice data uttered by the user in daily life.

[1302] "Recorded conversation data" refers to voice data recorded during the user's daily life.

[1303] "Hobbies and interests" refers to activities and areas of interest that a user is particularly interested in and likes to do.

[1304] "Characteristics" refers to a user's individual characteristics, personality, habits, etc.

[1305] "Speech recognition technology" refers to the technology that converts voice data into text data. Specifically, it is a technology that utilizes a speech recognition API.

[1306] "Text data" refers to text information converted using voice recognition technology.

[1307] A "server" refers to a computer system that provides functions such as data processing and storage over a network.

[1308] A "natural language processing engine" refers to technology for analyzing text data and understanding its meaning and context.

[1309] An "emotion recognition engine" refers to technology for analyzing emotions from voice or text data.

[1310] "User profile" refers to a user's individual data set created based on analyzed hobbies, preferences, characteristics, and emotional information.

[1311] A "generative AI model" refers to a model that uses artificial intelligence (AI) to generate candidate answers to questions.

[1312] "Candidate answers" refer to multiple options that indicate how a user should answer a question from another person.

[1313] "Speech synthesis technology" refers to the technology that converts text data into voice data. Specifically, it is a technology that reproduces the user's voice.

[1314] "Audio data" refers to audio files or audio signals generated using speech synthesis technology.

[1315] "Speech" refers to the act of outputting voice on behalf of the user by playing back voice data.

[1316] The present invention is a system for supporting users with anxiety disorders such as selective mutism, which makes it difficult or impossible for them to speak in certain social environments or situations. This system generates response candidates based on the user's hobbies, preferences, and emotions in situations where it is difficult for the user to speak, and speaks on the user's behalf.

[1317] The system is configured using the following hardware and software:

[1318] 1. Hardware:

[1319] Device: A device such as a smartphone or tablet used by a user.

[1320] Server: A remote server that analyzes data and synthesizes speech.

[1321] 2. Software:

[1322] Speech Recognition API: Speech recognition technologies such as Google Cloud Speech-to-Text.

[1323] Natural language processing engine: An engine for analyzing text data, such as spaCy.

[1324] Emotion recognition engine: An engine for analyzing emotions, such as IBM Watson Tone Analyzer.

[1325] Generative AI models: Generative AI models such as GPT-4.

[1326] Speech synthesis technology: Speech synthesis technology such as WaveNet.

[1327] The detailed process of the system is as follows:

[1328] Conversation data collection

[1329] The device records conversations that users have in their daily lives. For example, if a user is talking with a friend about a game at home, the device records this conversation.

[1330] Speech-to-text conversion

[1331] The device sends the recorded data to a speech recognition API and converts the speech into text. For example, the device converts the utterance, "What games are you playing these days?" into text.

[1332] Data transmission and analysis

[1333] The terminal transmits the converted text data to the server.

[1334] The server analyzes the received text data using a natural language processing engine to extract the user's hobbies, preferences, and characteristics, and then uses an emotion recognition engine to analyze the user's emotions from the voice tone and text content.

[1335] Creating a User Profile

[1336] The server creates a user profile based on the analysis results, which includes information such as "likes games," "interested in RPGs," and "is happy to talk."

[1337] Receiving and converting questions

[1338] During a class, the user is asked by a teacher, "Please explain why you are taking on the following assignment."

[1339] The device records the questions it receives and converts them into text data using a voice recognition API.

[1340] Generate and refine response candidates

[1341] The server uses a generative AI model to generate multiple response candidates based on the question text data and the user profile, such as "This assignment seems interesting because it will allow me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[1342] Based on the emotion recognition results, the generated response candidates are adapted to the user's current emotion, for example, to expressions such as "It looks like you're having fun."

[1343] Presentation and selection of response candidates

[1344] The terminal presents the adjusted response candidates to the user, displaying multiple response candidates through a GUI for the user to select from.

[1345] The user selects the most appropriate response from the presented options, for example, "This assignment seems fun because it allows me to learn a new perspective."

[1346] Speech synthesis and speech

[1347] The terminal sends the response selected by the user to the server.

[1348] The server converts the selected response into audio data in the user's voice using voice synthesis technology.

[1349] The device plays back the received voice data and speaks on behalf of the user, for example, saying, "This assignment seems fun because it allows me to learn a new perspective."

[1350] In this way, the present invention allows users to communicate naturally in situations such as selective mutism.

[1351] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1352] Step 1:

[1353] Conversation data collection

[1354] The device records conversations that users have in their daily lives. The input is the user's voice data. Specifically, the device records scenes of the user conversing with a friend through a microphone and saves the recorded voice data in local storage. The output is a recorded voice file.

[1355] Step 2:

[1356] Speech-to-text conversion

[1357] The device sends the recorded audio data to a speech recognition API, which converts the audio into text. The input is the recorded audio file. Specifically, the device calls the speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the audio file into text data. The output is the converted text data.

[1358] Step 3:

[1359] Sending data

[1360] The terminal sends the converted text data to the server. The input is text data. In concrete terms, the terminal sends the text data to the server using a secure protocol (e.g., HTTPS). The output is the text data sent to the server.

[1361] Step 4:

[1362] Analysis of tastes, preferences and emotions

[1363] The server analyzes the received text data using a natural language processing engine to extract the user's hobbies, preferences, and characteristics. The input is text data. Specifically, the server extracts keywords from the text data using a natural language processing engine such as spaCy. The output is data on the extracted user's hobbies, preferences, and characteristics.

[1364] Step 5:

[1365] emotion recognition

[1366] The server uses an emotion recognition engine to analyze the user's emotions from the voice tone and text content. The input is text data and voice data. Specifically, the server uses an emotion recognition engine such as IBM Watson Tone Analyzer to detect emotions from the text and voice. The output is the analyzed emotion data.

[1367] Step 6:

[1368] Creating a User Profile

[1369] The server creates a user profile based on the analysis results. The inputs are hobby and preference data and emotion data. Specifically, the server integrates these data and saves the user profile in a database. The output is the created user profile.

[1370] Step 7:

[1371] Receiving and converting questions

[1372] During a class, a teacher asks a user, "Please explain why you are working on the next assignment." The device records the question and converts it into text data using a speech recognition API. The input is the teacher's voice data. Specifically, the device records the question through the microphone and converts it into text using the speech recognition API. The output is the text data of the question.

[1373] Step 8:

[1374] Generate candidate responses

[1375] The server uses a generative AI model to create multiple candidate responses based on the question text data and the user profile. The input is the question text data and the user profile. Specifically, the server uses a generative AI model such as GPT-4 to generate candidate responses such as "This task seems interesting because it allows me to learn a new perspective" or "The content of this task will be useful to me in the future." The output is the text data of the generated candidate responses.

[1376] Step 9:

[1377] Emotional adaptation and regulation

[1378] The server adapts the generated response candidates to the user's current emotions. The input is the text data of the response candidates and the emotion data. Specifically, the server adjusts the expression of the response candidates based on the emotion data. For example, it changes "That sounds interesting" to "That sounds fun." The output is the adjusted response candidates.

[1379] Step 10:

[1380] Presentation and selection of response candidates

[1381] The terminal presents the adjusted response candidates to the user. The input is the adjusted response candidates. In concrete terms, a GUI is used to display multiple response candidates to the user. The user selects the one they feel is most appropriate from the presented response candidates. The output is the response candidate selected by the user.

[1382] Step 11:

[1383] Speech synthesis and speech

[1384] The terminal sends the response selected by the user to the server. The server converts the selected response into audio data in the user's voice using speech synthesis technology. The input is the selected response candidate. Specifically, the server uses speech synthesis technology such as WaveNet to convert the response text into audio data in the user's voice. The output is the generated audio data.

[1385] The device plays back the received voice data and speaks on behalf of the user, for example, saying, "This assignment seems fun because it allows me to learn a new perspective."

[1386] (Application example 2)

[1387] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1388] Previously, it was difficult for users with selective mutism to ask questions related to ads or engage in interactive responses while viewing ads. Furthermore, there was a lack of mechanisms for automatically generating appropriate responses based on the user's preferences and emotions. This prevented users from responding naturally to ads and actively participating in them. Furthermore, there was no system that could speak the generated responses in the user's own voice, making it difficult to achieve more natural and effective communication.

[1389] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1390] In this invention, the server includes means for recording conversation data from the user's daily life, means for analyzing the recorded conversation data and interpreting the user's hobbies, preferences, and characteristics, means for analyzing what others say in situations where it is difficult for the user to speak and automatically generating reply candidates based on the user's hobbies, preferences, and characteristics, means for presenting the generated reply candidates to the user and allowing them to select one, means for outputting the selected reply candidate in the user's voice, means for generating reply candidates based on a specific prompt sentence using a generative AI model for the user viewing an advertisement and presenting them to the user, and means for uttering the selected reply candidate in the user's voice in relation to the advertisement, thereby enabling the user to engage in active and natural interactions while viewing the advertisement.

[1391] "User's daily conversation data" refers to data that records utterances and conversations that the user makes in his or her daily life.

[1392] "Hobbies, preferences and characteristics" refers to information about a user's personal interests, likes and dislikes, behavioral patterns and personality traits.

[1393] A "scene in which it is difficult for the user to speak" refers to a situation or scene in which the user has difficulty speaking due to selective mutism or other reasons.

[1394] "Other people's statements" refer to words or sentences spoken by people other than the user.

[1395] "Candidate responses" refer to multiple options for answers or replies that are presented to a user as a comment.

[1396] A "generative AI model" is an artificial intelligence model that learns from large datasets and generates new documents and answers.

[1397] A "prompt sentence" is an input sentence given to a generative AI model, which serves as a reference sentence for the model to generate a response based on.

[1398] "Means for speaking in relation to an advertisement" refers to a technique or mechanism for speaking a user-selected response in relation to an advertisement in the user's voice.

[1399] The system of the present invention provides natural and appropriate responses to users, even when they have difficulty speaking while viewing advertisements, and promotes active interaction with advertisements. This system has the following functions and configuration.

[1400] System configuration

[1401] 1. Conversational data collection:

[1402] Device: Record the conversations you have in your daily life. For example, if you are talking to a friend about the latest gadgets, record this conversation.

[1403] 2. Data analysis and emotion recognition:

[1404] Device: The recorded voice data is converted into text data using a speech recognition API (for example, Google Cloud Speech-to-Text). This text data will be in the format of, for example, "What gadgets are you interested in these days?"

[1405] Terminal: Sends text data to the server.

[1406] Server: Analyzes text data using a natural language processing engine (e.g., GPT-3) and extracts user preferences and characteristics.

[1407] Server: Analyzes the user's emotions from recorded voice and text data using an emotion recognition engine (e.g., DeepAffects). For example, detects "enjoyment" or "excitement" from voice tone or specific keywords.

[1408] 3. Create a user profile:

[1409] Server: Creates a user profile based on the analyzed hobbies, preferences, characteristics, and emotional information. The user profile may include such things as "likes gadgets," "is interested in the latest technology," and "is happy to talk."

[1410] 4. Real-time conversation processing during ad viewing:

[1411] User: While the user is watching the ad, they are asked a question related to the ad. For example, the ad might ask, "What do you think about this new smartphone?"

[1412] Device: The question is recorded and converted into text data using a speech recognition API. This text data is also sent to the server.

[1413] 5. Response candidate generation and emotion adaptation:

[1414] Server: Based on the received text data and user profile, a generative AI model (e.g., GPT-3) generates multiple response candidates, such as "This smartphone looks futuristic and interesting" or "This will make my life easier."

[1415] Server: Based on the emotion recognition results, the generated response candidates are adapted to the user's current emotion. For example, if the user is feeling happy, the server will adjust the response to something like, "This smartphone looks futuristic and fun."

[1416] Server: Sends multiple coordinated response candidates to the device.

[1417] 6. User Selection and Speech:

[1418] Terminal: Presents the received response candidates to the user through a graphical user interface (GUI), for example, displaying "This smartphone looks futuristic and interesting" or "This will make my life easier."

[1419] User: Choose the response that they feel is most appropriate from the suggested responses. For example, "This smartphone looks futuristic and interesting."

[1420] Terminal: Sends the user-selected response to the server.

[1421] Server: Convert the selected response into audio data in the user's voice using speech synthesis technology (e.g., Google Text-to-Speech).

[1422] Server: Sends the generated audio data to the device.

[1423] Device: Plays back the received voice data and speaks on behalf of the user. For example, it speaks in the user's voice, saying, "This smartphone looks futuristic and interesting."

[1424] Specific examples

[1425] Interactions during the ad:

[1426] The ad asks, "What do you think about the features of the new smart refrigerator?" If the user does not want to speak for a moment, the application generates and presents potential responses to the user based on the user profile and the context of the ad, such as "This refrigerator looks futuristic and interesting" or "Having this in my kitchen would make my life more enjoyable." If the user selects one response, it is spoken in a synthesized voice.

[1427] Example prompt sentence:

[1428] "Users are watching an ad and are interested in the latest gadgets and technology. The ad asks, 'What do you think about the features of the new smart refrigerator?' If the user has difficulty answering, generate appropriate response suggestions to get them through the moment."

[1429] This system allows users to provide natural and appropriate responses while watching an advertisement, encouraging active interaction with the advertisement.

[1430] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1431] Step 1:

[1432] Conversation data collection

[1433] The device records the user's daily conversations. Specifically, for example, when the user talks with a friend about a gadget, the conversation is collected. The input is the user's voice, and the output is the recorded voice data.

[1434] Step 2:

[1435] Data analysis and emotion recognition

[1436] The device converts the recorded voice data into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). The input is the recorded voice data, and the output is the text data converted from the voice.

[1437] The terminal sends text data to the server. The input is the text data after speech recognition, and the output is the text data sent to the server.

[1438] The server analyzes the text data using a natural language processing engine (e.g., GPT-3) and extracts the user's interests, preferences, and characteristics. The input is text data, and the output is the analyzed interests, preferences, and characteristics information.

[1439] The server uses an emotion recognition engine (e.g., DeepAffects) to analyze the user's emotions from voice tone or specific keywords. The input is voice data or text data, and the output is the user's emotional information.

[1440] Step 3:

[1441] Creating a User Profile

[1442] The server creates a user profile based on the analyzed hobbies, preferences, characteristics, and emotional information. The input is hobbies, preferences, characteristics, and emotional information, and the output is a user profile. Examples of this include information such as "likes gadgets," "is interested in the latest technology," and "is happy when talking."

[1443] Step 4:

[1444] Real-time conversation processing while watching advertisements

[1445] While a user is watching an ad, a question related to the ad is displayed, for example, "What do you think about this new smartphone?" The input is the ad question, and the output is the user viewing the question.

[1446] The device records the question and converts it into text data using a speech recognition API. The input is the speech of the advertising question, and the output is text data, which is then sent to the server.

[1447] Step 5:

[1448] Response candidate generation and emotional adaptation

[1449] Based on the text data and user profile received by the server, a generative AI model (e.g., GPT-3) is used to generate multiple response candidates. The input is the text data and user profile, and the output is response candidates. For example, response candidates such as "This smartphone looks futuristic and interesting" or "This will make my life easier" are generated.

[1450] The server adapts the response candidates to the user's current emotion based on the emotion recognition results. The inputs are response candidates and emotion recognition results, and the output is emotion-adapted response candidates. For example, if the user seems to be having fun, the server will adjust the response to "This smartphone looks futuristic and fun."

[1451] The server sends the adjusted candidate responses to the terminal. The input is the adjusted candidate responses, and the output is the candidate responses sent to the terminal.

[1452] Step 6:

[1453] User selection and speech

[1454] The device presents the received response candidates to the user. The input is the adjusted response candidate, and the output is a screen display for the user to see. For example, the GUI displays "This smartphone looks futuristic and interesting" or "This will make my life easier."

[1455] The user selects the most appropriate response from the presented candidate responses. The input is the displayed candidate responses, and the output is the user's selection. For example, they can select "This smartphone looks futuristic and interesting."

[1456] The terminal sends the user's selected response to the server. The input is the user's selected information, and the output is the selected information sent to the server.

[1457] The server converts the selected response into audio data in the user's voice using speech synthesis technology (e.g., Google Text-to-Speech). The input is the selected response and the user's voice profile, and the output is the audio data.

[1458] The server sends the generated voice data to the terminal. The input is the generated voice data, and the output is the voice data sent to the terminal.

[1459] The device plays the received voice data and speaks on behalf of the user. The input is the received voice data, and the output is voice output from the speaker. For example, it speaks in the user's voice, "This smartphone looks futuristic and interesting."

[1460] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1461] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1462] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1463] [Fourth embodiment]

[1464] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1465] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1466] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1467] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1468] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1469] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1470] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1471] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1472] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1473] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1474] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1475] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1476] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1477] This invention is a system to help resolve selective mutism, an anxiety disorder that makes it difficult or impossible to speak in certain social environments or situations. In situations where it is difficult for the user to speak for themselves, a generative AI model is used to generate response candidates based on the user's preferences, and a mechanism is built in which the content selected by the user is spoken in the user's voice instead.

[1478] System configuration

[1479] 1. Conversation data collection

[1480] Device: Record conversations in scenes where the user can interact. For example, if a user is talking to a friend about a game at home, record this conversation.

[1481] 2. Data Analysis

[1482] Device: The recorded voice data is converted into text data using a speech recognition API. This text data will be in a format such as "What games are you playing these days?"

[1483] Terminal: Sends text data to the server.

[1484] 3. Create a user profile

[1485] Server: Analyzes the received text data using a natural language processing engine. From the analyzed data, the user's hobbies, preferences, and characteristics are extracted to create a user profile.

[1486] Server: Store information such as "I like games" or "I'm interested in RPGs" in the user profile.

[1487] 4. Real-time conversation processing

[1488] Terminal: Records a teacher's response to a question in a classroom where the user has difficulty speaking. The question is, "Please explain why you are working on the following assignment."

[1489] Device: The recorded speech is converted into text data using a speech recognition API. This text data is also sent to the server.

[1490] 5. Response candidate generation

[1491] Server: Based on the received text data and user profile, the server uses a generative AI model to automatically generate appropriate response candidates, such as "This assignment seems interesting because it will allow me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[1492] Server: Sends multiple response candidates to the device.

[1493] 6. User Selection and Speech

[1494] Terminal: Presents the received response candidates to the user. The user selects an option such as "This assignment seems interesting because it allows me to learn a new perspective."

[1495] Terminal: Sends the user-selected response to the server.

[1496] Server: Converts the selected response into audio data in the user's voice using speech synthesis technology.

[1497] Server: Sends the generated audio data to the device.

[1498] Terminal: Plays back the received voice data and speaks on behalf of the user.

[1499] Specific examples

[1500] Example 1: Collecting everyday conversation data

[1501] Device: At home, the user speaks to a friend, "What games have you been playing recently?" The device records this conversation, converts it into text using voice recognition, and sends it to the server.

[1502] Server: Analyzes the received data and determines that "the user is interested in the latest RPG."

[1503] Example 2: Real-time conversation at school

[1504] Device: During a class, the teacher asks, "Please explain why you are going to tackle the next assignment." The device records this question, converts it into text using voice recognition, and sends it to the server.

[1505] Server: Based on the question, it generates response candidates such as "This assignment seems interesting because it will allow me to learn a new perspective" and "Because the content of this assignment will be useful to me in the future" and sends them to the device.

[1506] Terminal: Sends the user-selected response to the server.

[1507] Server: Synthesizes speech using the user's voice and sends the generated voice data to the device.

[1508] Device: Plays back the received audio data and speaks on behalf of the user, "This assignment seems interesting because it will allow me to learn a new perspective."

[1509] In this way, the present invention realizes smooth communication by uttering a response appropriate to the situation in the user's voice, even in situations where it is difficult for the user to speak.

[1510] The processing flow will be explained below.

[1511] Step 1:

[1512] The user initiates a conversation in a scene where they can interact, for example, a user is talking with a friend about a game at home.

[1513] Step 2:

[1514] The device records the conversation. The device's microphone captures what you and your friends say.

[1515] Step 3:

[1516] The voice data recorded by the device is converted into text data using a speech recognition API. This conversion results in the text data "What games have you been playing recently?"

[1517] Step 4:

[1518] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[1519] Step 5:

[1520] The server receives the text data and stores it in a database, which stores past conversation data.

[1521] Step 6:

[1522] The server analyzes the stored text data using a natural language processing engine, extracting keywords such as "game" or "RPG."

[1523] Step 7:

[1524] Based on the analysis results, the server reflects the user's hobbies, preferences, and characteristics in the profile. Information such as "I like games" and "I'm interested in RPGs" is added to the user profile.

[1525] Step 8:

[1526] In situations where it is difficult for the user to speak (e.g., a school class), the teacher may ask a question during the class, such as, "Please explain why you are working on the next assignment."

[1527] Step 9:

[1528] The device records questions asked during class, and the teacher's comments are captured through the device's microphone.

[1529] Step 10:

[1530] The device converts the recorded voice data into text data using a speech recognition API. After conversion, the resulting text data reads, "Please explain why you want to tackle the following assignment."

[1531] Step 11:

[1532] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[1533] Step 12:

[1534] The server receives the text data and compares it with the user's profile. It then creates response candidates based on the teacher's question and the user's hobbies and preferences.

[1535] Step 13:

[1536] The server uses a generative AI model to generate multiple response candidates, such as "This assignment seems interesting because it allows me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[1537] Step 14:

[1538] The server sends multiple response options to the device. The data is sent securely using the HTTPS protocol.

[1539] Step 15:

[1540] The terminal presents the received response candidates to the user through a graphical user interface (GUI), displaying options such as "This assignment seems interesting because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[1541] Step 16:

[1542] The user selects one of the suggested responses, for example, "This assignment seems interesting because it will allow me to learn a new perspective."

[1543] Step 17:

[1544] The device sends the user-selected response to the server, transmitting data securely using the HTTPS protocol.

[1545] Step 18:

[1546] The server converts the selected response into audio data in the user's voice using a speech synthesis technology, such as a speech synthesis technology (e.g., WaveNet).

[1547] Step 19:

[1548] The server sends the audio data to the device. The data is sent securely using the HTTPS protocol.

[1549] Step 20:

[1550] The device plays back the received audio data, saying on behalf of the user, "This assignment seems interesting because it allows me to learn a new perspective."

[1551] The above are the specific program processing steps for implementing this system, which enables users to communicate smoothly even in situations where speaking is difficult.

[1552] Example 1

[1553] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1554] In the past, when users with anxiety disorders such as selective mutism had difficulty communicating, there was a lack of means for them to utter appropriate responses themselves, which led to difficulties and stress in social interactions.

[1555] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1556] In this invention, the server includes means for acquiring voice data from the user's daily life, means for converting the acquired voice data into text data using a voice recognition system, means for analyzing the converted text data and extracting the user's interests and characteristics, means for converting speech by others into text data using a voice recognition system and transmitting the text data to the server in situations where it is difficult for the user to speak, and automatically generating candidate responses using a generative AI model based on the received text data, means for presenting the generated candidate responses to the user for selection, means for converting the candidate response selected by the user into voice data in the user's voice using voice synthesis technology, and means for playing back the converted voice data. This makes it possible for the user to produce appropriate responses in their own voice even in situations where it is difficult for the user to speak.

[1557] "Voice data" is digital audio information that is a collection of speech from the user and others.

[1558] A "voice recognition system" is a technology or device that analyzes voice data and converts the content into text data.

[1559] "Text data" refers to character information of voice data converted by a voice recognition system.

[1560] "User interests" refers to themes or fields in which a user shows interest or concern.

[1561] "User characteristics" refers to personal characteristics such as a user's preferences, behavioral patterns, and personality.

[1562] A "server" is a computing device that receives and analyzes voice and text data and generates candidate responses.

[1563] A "generative AI model" refers to an artificial intelligence algorithm that performs natural language processing based on received data and automatically generates response candidates.

[1564] "Response candidates" are utterance options presented to the user that are automatically generated by a generative AI model.

[1565] "Speech synthesis technology" is a technology that converts text data into voice data and enables it to be played in a specific voice.

[1566] "Means for reproducing audio data" refers to a device that outputs the generated audio data as sound through speakers, headphones, etc.

[1567] The present invention is a system for helping to resolve selective mutism, an anxiety disorder that makes it difficult or impossible to speak in certain social environments or situations. Below, we will explain in detail how to implement this system.

[1568] Conversation data collection

[1569] Device: A device that records a user's everyday conversations. For example, it uses hardware such as a smartphone or a dedicated voice recorder. This allows a user to record a conversation, such as when talking with a friend about a game at home.

[1570] Data analysis

[1571] Device: The recorded voice data is converted into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text API). Through this process, the recorded conversation is obtained as text data such as "What games have you been playing recently?"

[1572] Terminal: Sends the converted text data to the server.

[1573] Creating a User Profile

[1574] Server: The received text data is analyzed using a natural language processing engine (e.g., Google Cloud Natural Language API). This analysis extracts the user's interests and characteristics. For example, the analysis result may be information such as "The user is interested in the latest RPGs." This information is saved in a database as a user profile.

[1575] Real-time conversation processing

[1576] Device: Records situations where it is difficult for the user to speak, such as questions asked in class. If a teacher asks a question like, "Please explain why you are going to tackle the next assignment," the device will record the content.

[1577] Terminal: The recorded speech is converted into text data using a voice recognition system, and this text data is sent to the server.

[1578] Response candidate generation

[1579] Server: Based on the received text data and user profile, a generative AI model (e.g., OpenAI's GPT-3) is used to generate appropriate response candidates, such as "This assignment seems interesting because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[1580] User selection and speech

[1581] Terminal: Presents the generated response candidates to the user, showing options on a screen, etc. The user selects the appropriate response from the presented candidates.

[1582] Terminal: Sends the selected response to the server.

[1583] Server: The selected response is converted into voice data that resembles the user's voice using voice synthesis technology (e.g., Google Cloud Text-to-Speech API), and this voice data is sent to the device.

[1584] Terminal: Plays back the received voice data and speaks on behalf of the user.

[1585] Specific examples

[1586] Example 1: Collecting everyday conversation data

[1587] Device: At home, a user talks to a friend, "What games have you been playing recently?" This conversation is recorded, converted into text using voice recognition, and sent to the server.

[1588] Server: Analyzes the received data and determines that "the user is interested in the latest RPGs" and reflects this in the profile.

[1589] Example 2: Real-time conversation at school

[1590] Device: During a class, a teacher asks, "Please explain why you are going to tackle the following assignment." This question is recorded, converted into text using voice recognition, and sent to a server.

[1591] Server: Based on the question, it generates response candidates such as "This assignment seems interesting because it will allow me to learn a new perspective" and "Because the content of this assignment will be useful to me in the future." It then sends these to the device.

[1592] Terminal: The user selects "This assignment seems interesting because it allows me to learn a new perspective" from the presented response candidates. After the selection, the terminal sends this content to the server.

[1593] Server: The selected content is synthesized using the user's voice and the generated voice data is sent to the terminal.

[1594] Terminal: Plays back audio data and speaks on behalf of the user.

[1595] Example prompts to input to the generative AI model

[1596] "When a teacher asks a user why they are taking on the next assignment, generate appropriate response suggestions based on the user profile: The user likes games and is interested in RPGs."

[1597] As a result, the present invention can achieve smooth communication even in situations where it is difficult for the user to speak by uttering a response appropriate to the situation in the user's voice.

[1598] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1599] Step 1: Record the conversation

[1600] Device: Activates a function to record the user's everyday conversations and dialogue scenes. For example, by using a smartphone application, a user can start recording when talking to a friend at home, asking, "What games have you been playing recently?" The input is environmental sounds and human speech, and the output is digital audio data.

[1601] Step 2: Convert the audio data

[1602] Device: The recorded voice data is converted into text data using a speech recognition system such as the Google Cloud Speech-to-Text API. During the conversion process, the voice signal is analyzed and converted into text data. The input is digital voice data, and the output is text data such as "What games have you been playing lately?"

[1603] Step 3: Sending data

[1604] Terminal: Transmits the converted text data to the server. The input is text data, and the output is a digital message sent to the server via the Internet.

[1605] Step 4: Analyzing the text data

[1606] Server: The received text data is analyzed using a natural language processing engine such as the Google Cloud Natural Language API. During the analysis process, the user's interests and characteristics are extracted. The input is text data, and the output is user profile information such as "I like games" or "I'm interested in RPGs."

[1607] Step 5: Creating a User Profile

[1608] Server: Creates a user profile based on the analyzed interests and characteristics and stores it in a database. The input is the user's interests and characteristics data, and the output is a user profile stored in the database.

[1609] Step 6: Record real-time conversations

[1610] Terminal: Records speech in situations where it is difficult for the user to speak, such as when a teacher asks a question in a school class, such as "Please explain why you are going to work on the next assignment." The input is real-time speech, and the output is recorded audio data.

[1611] Step 7: Re-encode the audio data

[1612] Terminal: The recorded voice data is converted into text data using a voice recognition system and sent to the server. The input is the recorded voice data, and the output is the text data and its transmission.

[1613] Step 8: Generate candidate responses

[1614] Server: Based on the received text data and user profile, a generative AI model (e.g., OpenAI's GPT-3) is used to generate appropriate response candidates. The input is the text data and the user profile, and the output is response candidates such as "This assignment seems interesting because it allows me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[1615] Step 9: Suggesting possible responses

[1616] Terminal: A screen display that presents the generated response candidates to the user. The input is the response candidate data, and the output is the on-screen options that the user can see.

[1617] Step 10: Selecting the User

[1618] User: Select the appropriate response from the presented response candidates. The input is the response candidates on the screen, and the output is the user's selected data.

[1619] Step 11: Submit Selected Data

[1620] Terminal: Sends the user-selected response to the server. The input is the user-selected data, and the output is the selected data sent to the server.

[1621] Step 12: Speech synthesis

[1622] Server: The selected response is converted into audio data that resembles the user's voice using speech synthesis technology such as the Google Cloud Text-to-Speech API. The input is the selected text data, and the output is audio data.

[1623] Step 13: Sending audio data

[1624] Server: Sends the generated voice data to the terminal. The input is the voice data, and the output is the data sent to the terminal.

[1625] Step 14: Playing Audio

[1626] Terminal: The terminal plays the received voice data through a device such as a speaker or headphones and speaks on behalf of the user. The input is the voice data, and the output is the user's response, which is played as sound.

[1627] In this way, the system of the present invention makes it possible for the user to utter an appropriate response in his or her own voice even in situations where it is difficult for the user to speak.

[1628] (Application example 1)

[1629] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1630] Many people today use food delivery services, but people with anxiety disorders such as selective mutism have difficulty ordering by voice. This makes it difficult to use food delivery services smoothly. In particular, when selecting order contents based on the user's preferences or when confirming the contents by voice, problems arise in which the system is unable to respond appropriately. A system that can solve these problems is needed.

[1631] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1632] In this invention, the server includes means for recording conversation data from the user's daily life, means for analyzing the recorded conversation data and interpreting the user's hobbies, preferences, and characteristics, means for generating candidate responses based on the user's profile and input keywords, means for outputting the generated candidate responses as the user's voice using speech recognition and speech synthesis technology, and food delivery order support system means for automatically transmitting order details based on the user's selection and confirmation, thereby enabling even users who have difficulty speaking to order and confirm appropriate food delivery.

[1633] "Conversation data" refers to audio information of utterances and conversations that occur in the user's daily life.

[1634] "Hobbies and preferences" refers to the interests and concerns that a user has in their daily life, or the things that the user tends to avoid.

[1635] "Characteristics" refers to the user's personality, behavioral characteristics, habits, etc.

[1636] "Response candidates" refer to response options automatically generated by a generative AI model based on the user's preferences and characteristics.

[1637] "Speech recognition" refers to the technology of converting voice data into text data.

[1638] "Speech synthesis technology" refers to technology that converts text data into voice data and generates natural speaking voices.

[1639] "User profile" refers to individual information based on a user's hobbies, preferences, characteristics, and past behavioral history.

[1640] "Keywords" refer to specific words or phrases that users enter when placing an order.

[1641] A "food delivery order support system" refers to a system that automatically generates response candidates and outputs them in the user's voice using speech recognition and speech synthesis technology to assist users in ordering.

[1642] A "generative AI model" refers to an artificial intelligence model that generates appropriate responses to user input based on large amounts of data.

[1643] A "prompt sentence" refers to a sentence that serves as an instruction for generating response candidates based on a sentence input into a generative AI model.

[1644] MODE FOR CARRYING OUT THE INVENTION

[1645] The present invention provides a system that enables users to smoothly place orders when using a food delivery service even if they have difficulty speaking. An embodiment of this system will be described in detail below.

[1646] System configuration

[1647] 1. Conversation data recording

[1648] The device records and stores conversation data from the user's daily life, using devices such as smartphones and smart speakers. This voice data is stored in a database for later analysis.

[1649] 2. Data Analysis

[1650] The device converts the saved conversation data into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text), and the converted text data is sent to the server.

[1651] 3. Create a user profile

[1652] The server analyzes the received text data using a natural language processing engine (e.g., OpenAI GPT). Based on the analyzed data, the user's hobbies, preferences, and characteristics are extracted, and a user profile is created. The created profile includes information such as the user's favorite foods, ingredients they avoid, and their past ordering history.

[1653] 4. Real-time order support

[1654] If a user has difficulty speaking when using food delivery, they can enable the "order assistance" mode on the device, which allows the user to input the order details in keyword format without speaking.

[1655] 5. Response candidate generation

[1656] The server uses a generative AI model to generate appropriate response candidates based on the user profile and the entered keywords, and provides instructions to the AI ​​model using prompts. Examples of prompts include:

[1657] "User's favorite food: pizza. User's favorite ingredient: cheese. User's current restaurant: Pizza Hut. Generate menu response suggestions."

[1658] The generated response candidates are, for example, "Vegetable pizza (no cheese)" and "Hawaiian pizza (no cheese)."

[1659] 6. Presentation and Selection

[1660] The generated response candidates are displayed on the terminal and presented to the user, who then selects the appropriate answer from the presented options. The selected answer is then sent back to the server.

[1661] 7. Audio output and confirmation

[1662] The server uses voice synthesis technology (e.g., Google Cloud Text-to-Speech) to output the selected response as the user's voice. The generated voice data is sent back to the terminal and played as a final confirmation of the order details. The confirmation is something like, "Are you sure you want to order with this?"

[1663] 8. Completing your order

[1664] After the user confirms, the server automatically sends the order to the food delivery partner, allowing the user to place an accurate order without speaking.

[1665] Specific examples

[1666] For example, if a user wants to order a pizza, the system operates as follows:

[1667] The user selects the "order assistance" mode and inputs the text "I would like to order a pizza."

[1668] The server generates response candidates such as "vegetable pizza (no cheese)" or "Hawaiian pizza (no cheese)" based on the user's preferences and past history.

[1669] After the user selects from the displayed options, a final voice confirmation is given and the order is finally sent to the food delivery service.

[1670] In this way, the present invention can reduce the burden on users and provide a smoother ordering process through ordering support for food delivery services.

[1671] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1672] Step 1:

[1673] The device records and saves conversation data from the user's daily life. Specifically, it records the user's speech through devices such as smartphones and smart speakers. The conversations input as voice data are temporarily stored in the device's local storage.

[1674] Step 2:

[1675] The device converts the saved conversation data into text data using a speech recognition API. Specifically, it analyzes the voice data using the Google Cloud Speech-to-Text API and outputs it as text data. This text data is then sent to the server.

[1676] Step 3:

[1677] The server analyzes the received text data using a natural language processing engine. Specifically, it uses the OpenAI GPT model to extract the user's hobbies, preferences, and characteristics from the text data. A user profile is generated as a result of the analysis and stored in this profile database.

[1678] Step 4:

[1679] If a user has difficulty speaking when using food delivery, the device will activate an "order assistance" mode. The user enters the "keyword" they want to order in text format on the device. The keyword entered by the user is received as the initial input by the device.

[1680] Step 5:

[1681] The server generates candidate responses using a generative AI model based on the user profile and input keywords. The prompt is set as follows: "User's favorite food: pizza, ingredients the user wants to avoid: cheese, the restaurant the user is currently using: Pizza Hut. Please generate candidate responses for the menu." The server uses the OpenAI GPT model to generate appropriate candidate responses based on this prompt.

[1682] Step 6:

[1683] The generated response candidates are sent from the server to the terminal. The terminal displays the received response candidates on the screen and presents them to the user. The user then selects their order from the displayed options.

[1684] Step 7:

[1685] The order details selected by the user are sent from the terminal to the server again. The server then uses voice synthesis technology to convert the received order details into audio data in the user's voice. Specifically, it uses the Google Cloud Text-to-Speech API to generate natural audio data from text data.

[1686] Step 8:

[1687] The generated voice data is sent from the server to the terminal, and the terminal plays the voice data. The user makes a final confirmation by voice, asking, "Are you sure you want to order with this content?"

[1688] Step 9:

[1689] After the user confirms, the server automatically sends the order to the food delivery partner, which then sends the order in digital format to the food delivery service's API, where the order is finally fulfilled.

[1690] In this way, the system helps users smoothly order food delivery even in situations where they have difficulty speaking.

[1691] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1692] This invention is a system to help resolve selective mutism, an anxiety disorder that makes it difficult or impossible to speak in certain social environments or situations. In situations where it is difficult for the user to speak for themselves, the system combines a generative AI model and an emotion engine to generate response candidates based on the user's preferences and emotions, allowing the system to speak the content selected by the user in their own voice.

[1693] System configuration

[1694] 1. Conversation data collection

[1695] Device: Record conversations in scenes where the user can interact. For example, if a user is talking with a friend about a game at home, record this conversation.

[1696] 2. Data Analysis and Emotion Recognition

[1697] Device: The recorded voice data is converted into text data using a speech recognition API. This text data will be in the format of "What games have you been playing recently?"

[1698] Terminal: Sends text data to the server.

[1699] Server: Analyzes the received text data using a natural language processing engine to extract the user's hobbies, preferences, and characteristics.

[1700] Server: Analyzes the user's emotions from recorded voice and text data using an emotion recognition engine. For example, it detects "happiness" or "excitement" from voice tone and specific keywords.

[1701] 3. Create a user profile

[1702] Server: Creates a user profile based on the analyzed hobbies, preferences, characteristics, and emotional information. The user profile may include such things as "I like games," "I'm interested in RPGs," and "I talk with a happy attitude."

[1703] 4. Real-time conversation processing

[1704] User: In situations where it is difficult to speak (e.g., a school class), a question is asked during class. For example, the teacher might ask, "Please explain why you are working on the next assignment."

[1705] Device: The question is recorded and converted into text data using a speech recognition API. This text data is also sent to the server.

[1706] 5. Response candidate generation and emotion adaptation

[1707] Server: Based on the received text data and user profile, the generative AI model creates multiple response candidates, such as "This assignment seems interesting because it will allow me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[1708] Server: Based on the emotion recognition results, the generated response candidates are adapted to the user's current emotions. For example, if the user is feeling happy, the server can adjust the response to something like, "This task seems fun because it will allow me to learn a new perspective."

[1709] Server: Sends multiple coordinated response candidates to the device.

[1710] 6. User Selection and Speech

[1711] Terminal: Presents the received response candidates to the user through a graphical user interface (GUI), displaying responses such as "This assignment seems fun because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[1712] User: Choose the response that they feel is most appropriate from the suggested responses. For example, "This assignment seems fun because it will help me learn new perspectives."

[1713] Terminal: Sends the user-selected response to the server.

[1714] Server: Converts the selected response into audio data in the user's voice using speech synthesis technology, for example, using speech synthesis technology (e.g., WaveNet).

[1715] Server: Sends the generated audio data to the device.

[1716] Device: Plays back the received audio data and speaks on behalf of the user. For example, it might say in the user's voice, "This assignment seems fun because it allows me to learn a new perspective."

[1717] Specific examples

[1718] Example 1: Daily conversation data collection and emotion recognition

[1719] Device: At home, the user speaks to a friend, "What games have you been playing recently?" The device records this conversation, converts it into text using voice recognition, and sends it to the server.

[1720] Server: Analyzes the received data and determines that "the user is interested in the latest RPG." At the same time, the emotion engine analyzes it and recognizes that the user is "enjoying the conversation."

[1721] Example 2: Real-time conversations and emotional adaptation in schools

[1722] Device: During a class, the teacher asks, "Please explain why you are going to tackle the next assignment." The device records this question, converts it into text using voice recognition, and sends it to the server.

[1723] Server: Based on the question, it generates response candidates such as "This assignment seems interesting because it will allow me to learn a new perspective" and "Because the content of this assignment will be useful to me in the future." At the same time, the emotion engine analyzes the user's current emotions and recognizes that they are "interested."

[1724] Server: Adapts the generated response candidates to the user's emotions and generates an adjusted response such as, "This task seems fun because it will allow me to learn a new perspective."

[1725] Terminal: Sends the user-selected response to the server.

[1726] Server: Synthesizes speech using the user's voice and sends the generated voice data to the device.

[1727] Device: Plays back the received audio data and speaks on behalf of the user, "This assignment seems fun because it allows me to learn a new perspective."

[1728] In this way, by combining emotion engines, more natural responses that reflect the user's emotional state become possible, resulting in even smoother communication.

[1729] The processing flow will be explained below.

[1730] Step 1:

[1731] The user begins a conversation in an interactive scene, for example, a user talking about gaming with a friend at home.

[1732] Step 2:

[1733] The device records the conversation. The device's microphone captures what you and your friends say.

[1734] Step 3:

[1735] The voice data recorded by the device is converted into text data using a speech recognition API. This conversion results in the text data "What games have you been playing recently?"

[1736] Step 4:

[1737] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[1738] Step 5:

[1739] The server receives the text data and stores it in a database, which stores past conversation data.

[1740] Step 6:

[1741] The server analyzes the stored text data using a natural language processing engine, extracting keywords such as "game" or "RPG."

[1742] Step 7:

[1743] Based on the analysis results, the server extracts the user's hobbies, preferences, and characteristics and reflects them in the user's profile. Information such as "I like games" and "I'm interested in RPGs" is added to the user profile.

[1744] Step 8:

[1745] The server sends the voice and text data to an emotion recognition engine, which analyzes emotions from voice tone and specific keywords.

[1746] Step 9:

[1747] The server analyzes the emotion recognition results and adds the user's emotional state to the profile, for example, storing emotional information such as "the person is talking happily."

[1748] Step 10:

[1749] In situations where it is difficult for the user to speak (e.g., a school class), the teacher may ask a question during the class, such as, "Please explain why you are working on the next assignment."

[1750] Step 11:

[1751] The device records questions asked during class, and the teacher's comments are captured through the device's microphone.

[1752] Step 12:

[1753] The device converts the recorded voice data into text data using a speech recognition API. After conversion, the resulting text data reads, "Please explain why you want to tackle the following assignment."

[1754] Step 13:

[1755] The device sends the converted text data to the server, using the HTTPS protocol to ensure data security.

[1756] Step 14:

[1757] The server receives the text data and compares it with the user's profile. It then creates response candidates based on the teacher's question, the user's hobbies, preferences, and emotional state.

[1758] Step 15:

[1759] The server uses a generative AI model to generate multiple response candidates, such as "This assignment seems interesting because it allows me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[1760] Step 16:

[1761] The server then uses the emotion recognition results to tailor the generated response candidates to the user's current emotions. For example, if the user is interested, the server might tailor the response to something like, "This task seems fun because it will allow me to learn a new perspective."

[1762] Step 17:

[1763] The server sends multiple coordinated response candidates to the device, and transmits the data securely using the HTTPS protocol.

[1764] Step 18:

[1765] The terminal presents the received response candidates to the user through a graphical user interface (GUI), displaying options such as "This assignment seems fun because it allows me to learn a new perspective" and "The content of this assignment will be useful to me in the future."

[1766] Step 19:

[1767] The user selects one of the suggested responses, for example, "This assignment seems fun because it allows me to learn a new perspective."

[1768] Step 20:

[1769] The device sends the user-selected response to the server, transmitting data securely using the HTTPS protocol.

[1770] Step 21:

[1771] The server converts the selected response into audio data in the user's voice using a speech synthesis technology, such as a speech synthesis technology (e.g., WaveNet).

[1772] Step 22:

[1773] The server sends the generated audio data to the device, securely using the HTTPS protocol.

[1774] Step 23:

[1775] The device plays back the received audio data, saying on behalf of the user, "This assignment seems fun because it allows me to learn a new perspective."

[1776] These are the specific program processing steps of this system. This system allows users to communicate smoothly even in situations where speaking is difficult, and by combining it with an emotion engine, it becomes possible to respond more naturally and appropriately.

[1777] Example 2

[1778] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1779] The challenge is to provide a means to support smooth communication for users with anxiety disorders, such as selective mutism, who find it difficult or impossible to speak in certain social environments or situations. In particular, it is necessary to build a system that can generate appropriate responses based on the user's own preferences and emotions and speak them in the user's own voice, even in situations where it is difficult for the user to speak.

[1780] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1781] In this invention, the server includes a means for creating a user profile based on the user's hobbies, preferences, and emotional information, a means for automatically generating candidate responses based on the utterances of others using a generative AI model, and a means for vocalizing the selected candidate responses in the user's voice using speech synthesis technology, thereby enabling the user to smoothly utter natural responses that reflect their emotions, hobbies, preferences, and emotional information in appropriate situations.

[1782] "User's daily conversation data" refers to all voice data uttered by the user in daily life.

[1783] "Recorded conversation data" refers to voice data recorded during the user's daily life.

[1784] "Hobbies and interests" refers to activities and areas of interest that a user is particularly interested in and likes to do.

[1785] "Characteristics" refers to a user's individual characteristics, personality, habits, etc.

[1786] "Speech recognition technology" refers to the technology that converts voice data into text data. Specifically, it is a technology that utilizes a speech recognition API.

[1787] "Text data" refers to text information converted using voice recognition technology.

[1788] A "server" refers to a computer system that provides functions such as data processing and storage over a network.

[1789] A "natural language processing engine" refers to technology for analyzing text data and understanding its meaning and context.

[1790] An "emotion recognition engine" refers to technology for analyzing emotions from voice or text data.

[1791] "User profile" refers to a user's individual data set created based on analyzed hobbies, preferences, characteristics, and emotional information.

[1792] A "generative AI model" refers to a model that uses artificial intelligence (AI) to generate candidate answers to questions.

[1793] "Candidate answers" refer to multiple options that indicate how a user should answer a question from another person.

[1794] "Speech synthesis technology" refers to the technology that converts text data into voice data. Specifically, it is a technology that reproduces the user's voice.

[1795] "Audio data" refers to audio files or audio signals generated using speech synthesis technology.

[1796] "Speech" refers to the act of outputting voice on behalf of the user by playing back voice data.

[1797] The present invention is a system for supporting users with anxiety disorders such as selective mutism, which makes it difficult or impossible for them to speak in certain social environments or situations. This system generates response candidates based on the user's hobbies, preferences, and emotions in situations where it is difficult for the user to speak, and speaks on the user's behalf.

[1798] The system is configured using the following hardware and software:

[1799] 1. Hardware:

[1800] Device: A device such as a smartphone or tablet used by a user.

[1801] Server: A remote server that analyzes data and synthesizes speech.

[1802] 2. Software:

[1803] Speech Recognition API: Speech recognition technologies such as Google Cloud Speech-to-Text.

[1804] Natural language processing engine: An engine for analyzing text data, such as spaCy.

[1805] Emotion recognition engine: An engine for analyzing emotions, such as IBM Watson Tone Analyzer.

[1806] Generative AI models: Generative AI models such as GPT-4.

[1807] Speech synthesis technology: Speech synthesis technology such as WaveNet.

[1808] The detailed process of the system is as follows:

[1809] Conversation data collection

[1810] The device records conversations that users have in their daily lives. For example, if a user is talking with a friend about a game at home, the device records this conversation.

[1811] Speech-to-text conversion

[1812] The device sends the recorded data to a speech recognition API and converts the speech into text. For example, the device converts the utterance, "What games are you playing these days?" into text.

[1813] Data transmission and analysis

[1814] The terminal transmits the converted text data to the server.

[1815] The server analyzes the received text data using a natural language processing engine to extract the user's hobbies, preferences, and characteristics, and then uses an emotion recognition engine to analyze the user's emotions from the voice tone and text content.

[1816] Creating a User Profile

[1817] The server creates a user profile based on the analysis results, which includes information such as "likes games," "interested in RPGs," and "is happy to talk."

[1818] Receiving and converting questions

[1819] During a class, the user is asked by a teacher, "Please explain why you are taking on the following assignment."

[1820] The device records the questions it receives and converts them into text data using a voice recognition API.

[1821] Generate and refine response candidates

[1822] The server uses a generative AI model to generate multiple response candidates based on the question text data and the user profile, such as "This assignment seems interesting because it will allow me to learn a new perspective" or "The content of this assignment will be useful to me in the future."

[1823] Based on the emotion recognition results, the generated response candidates are adapted to the user's current emotion, for example, to expressions such as "It looks like you're having fun."

[1824] Presentation and selection of response candidates

[1825] The terminal presents the adjusted response candidates to the user, displaying multiple response candidates through a GUI for the user to select from.

[1826] The user selects the most appropriate response from the presented options, for example, "This assignment seems fun because it allows me to learn a new perspective."

[1827] Speech synthesis and speech

[1828] The terminal sends the response selected by the user to the server.

[1829] The server converts the selected response into audio data in the user's voice using voice synthesis technology.

[1830] The device plays back the received voice data and speaks on behalf of the user, for example, saying, "This assignment seems fun because it allows me to learn a new perspective."

[1831] In this way, the present invention allows users to communicate naturally in situations such as selective mutism.

[1832] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1833] Step 1:

[1834] Conversation data collection

[1835] The device records conversations that users have in their daily lives. The input is the user's voice data. Specifically, the device records scenes of the user conversing with a friend through a microphone and saves the recorded voice data in local storage. The output is a recorded voice file.

[1836] Step 2:

[1837] Speech-to-text conversion

[1838] The device sends the recorded audio data to a speech recognition API, which converts the audio into text. The input is the recorded audio file. Specifically, the device calls the speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the audio file into text data. The output is the converted text data.

[1839] Step 3:

[1840] Sending data

[1841] The terminal sends the converted text data to the server. The input is text data. In concrete terms, the terminal sends the text data to the server using a secure protocol (e.g., HTTPS). The output is the text data sent to the server.

[1842] Step 4:

[1843] Analysis of tastes, preferences and emotions

[1844] The server analyzes the received text data using a natural language processing engine to extract the user's hobbies, preferences, and characteristics. The input is text data. Specifically, the server extracts keywords from the text data using a natural language processing engine such as spaCy. The output is data on the extracted user's hobbies, preferences, and characteristics.

[1845] Step 5:

[1846] emotion recognition

[1847] The server uses an emotion recognition engine to analyze the user's emotions from the voice tone and text content. The input is text data and voice data. Specifically, the server uses an emotion recognition engine such as IBM Watson Tone Analyzer to detect emotions from the text and voice. The output is the analyzed emotion data.

[1848] Step 6:

[1849] Creating a User Profile

[1850] The server creates a user profile based on the analysis results. The inputs are hobby and preference data and emotion data. Specifically, the server integrates these data and saves the user profile in a database. The output is the created user profile.

[1851] Step 7:

[1852] Receiving and converting questions

[1853] During a class, a teacher asks a user, "Please explain why you are working on the next assignment." The device records the question and converts it into text data using a speech recognition API. The input is the teacher's voice data. Specifically, the device records the question through the microphone and converts it into text using the speech recognition API. The output is the text data of the question.

[1854] Step 8:

[1855] Generate candidate responses

[1856] The server uses a generative AI model to create multiple candidate responses based on the question text data and the user profile. The input is the question text data and the user profile. Specifically, the server uses a generative AI model such as GPT-4 to generate candidate responses such as "This task seems interesting because it allows me to learn a new perspective" or "The content of this task will be useful to me in the future." The output is the text data of the generated candidate responses.

[1857] Step 9:

[1858] Emotional adaptation and regulation

[1859] The server adapts the generated response candidates to the user's current emotions. The input is the text data of the response candidates and the emotion data. Specifically, the server adjusts the expression of the response candidates based on the emotion data. For example, it changes "That sounds interesting" to "That sounds fun." The output is the adjusted response candidates.

[1860] Step 10:

[1861] Presentation and selection of response candidates

[1862] The terminal presents the adjusted response candidates to the user. The input is the adjusted response candidates. In concrete terms, a GUI is used to display multiple response candidates to the user. The user selects the one they feel is most appropriate from the presented response candidates. The output is the response candidate selected by the user.

[1863] Step 11:

[1864] Speech synthesis and speech

[1865] The terminal sends the response selected by the user to the server. The server converts the selected response into audio data in the user's voice using speech synthesis technology. The input is the selected response candidate. Specifically, the server uses speech synthesis technology such as WaveNet to convert the response text into audio data in the user's voice. The output is the generated audio data.

[1866] The device plays back the received voice data and speaks on behalf of the user, for example, saying, "This assignment seems fun because it allows me to learn a new perspective."

[1867] (Application example 2)

[1868] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1869] Previously, it was difficult for users with selective mutism to ask questions related to ads or engage in interactive responses while viewing ads. Furthermore, there was a lack of mechanisms for automatically generating appropriate responses based on the user's preferences and emotions. This prevented users from responding naturally to ads and actively participating in them. Furthermore, there was no system that could speak the generated responses in the user's own voice, making it difficult to achieve more natural and effective communication.

[1870] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1871] In this invention, the server includes means for recording conversation data from the user's daily life, means for analyzing the recorded conversation data and interpreting the user's hobbies, preferences, and characteristics, means for analyzing what others say in situations where it is difficult for the user to speak and automatically generating reply candidates based on the user's hobbies, preferences, and characteristics, means for presenting the generated reply candidates to the user and allowing them to select one, means for outputting the selected reply candidate in the user's voice, means for generating reply candidates based on a specific prompt sentence using a generative AI model for the user viewing an advertisement and presenting them to the user, and means for uttering the selected reply candidate in the user's voice in relation to the advertisement, thereby enabling the user to engage in active and natural interactions while viewing the advertisement.

[1872] "User's daily conversation data" refers to data that records utterances and conversations that the user makes in his or her daily life.

[1873] "Hobbies, preferences and characteristics" refers to information about a user's personal interests, likes and dislikes, behavioral patterns and personality traits.

[1874] A "scene in which it is difficult for the user to speak" refers to a situation or scene in which the user has difficulty speaking due to selective mutism or other reasons.

[1875] "Other people's statements" refer to words or sentences spoken by people other than the user.

[1876] "Candidate responses" refer to multiple options for answers or replies that are presented to a user as a comment.

[1877] A "generative AI model" is an artificial intelligence model that learns from large datasets and generates new documents and answers.

[1878] A "prompt sentence" is an input sentence given to a generative AI model, which serves as a reference sentence for the model to generate a response based on.

[1879] "Means for speaking in relation to an advertisement" refers to a technique or mechanism for speaking a user-selected response in relation to an advertisement in the user's voice.

[1880] The system of the present invention provides natural and appropriate responses to users, even when they have difficulty speaking while viewing advertisements, and promotes active interaction with advertisements. This system has the following functions and configuration.

[1881] System configuration

[1882] 1. Conversational data collection:

[1883] Device: Record the conversations you have in your daily life. For example, if you are talking to a friend about the latest gadgets, record this conversation.

[1884] 2. Data analysis and emotion recognition:

[1885] Device: The recorded voice data is converted into text data using a speech recognition API (for example, Google Cloud Speech-to-Text). This text data will be in the format of, for example, "What gadgets are you interested in these days?"

[1886] Terminal: Sends text data to the server.

[1887] Server: Analyzes text data using a natural language processing engine (e.g., GPT-3) and extracts user preferences and characteristics.

[1888] Server: Analyzes the user's emotions from recorded voice and text data using an emotion recognition engine (e.g., DeepAffects). For example, detects "enjoyment" or "excitement" from voice tone or specific keywords.

[1889] 3. Create a user profile:

[1890] Server: Creates a user profile based on the analyzed hobbies, preferences, characteristics, and emotional information. The user profile may include such things as "likes gadgets," "is interested in the latest technology," and "is happy to talk."

[1891] 4. Real-time conversation processing during ad viewing:

[1892] User: While the user is watching the ad, they are asked a question related to the ad. For example, the ad might ask, "What do you think about this new smartphone?"

[1893] Device: The question is recorded and converted into text data using a speech recognition API. This text data is also sent to the server.

[1894] 5. Response candidate generation and emotion adaptation:

[1895] Server: Based on the received text data and user profile, a generative AI model (e.g., GPT-3) generates multiple response candidates, such as "This smartphone looks futuristic and interesting" or "This will make my life easier."

[1896] Server: Based on the emotion recognition results, the generated response candidates are adapted to the user's current emotion. For example, if the user is feeling happy, the server will adjust the response to something like, "This smartphone looks futuristic and fun."

[1897] Server: Sends multiple coordinated response candidates to the device.

[1898] 6. User Selection and Speech:

[1899] Terminal: Presents the received response candidates to the user through a graphical user interface (GUI), for example, displaying "This smartphone looks futuristic and interesting" or "This will make my life easier."

[1900] User: Choose the response that they feel is most appropriate from the suggested responses. For example, "This smartphone looks futuristic and interesting."

[1901] Terminal: Sends the user-selected response to the server.

[1902] Server: Convert the selected response into audio data in the user's voice using speech synthesis technology (e.g., Google Text-to-Speech).

[1903] Server: Sends the generated audio data to the device.

[1904] Device: Plays back the received voice data and speaks on behalf of the user. For example, it speaks in the user's voice, saying, "This smartphone looks futuristic and interesting."

[1905] Specific examples

[1906] Interactions during the ad:

[1907] The ad asks, "What do you think about the features of the new smart refrigerator?" If the user does not want to speak for a moment, the application generates and presents potential responses to the user based on the user profile and the context of the ad, such as "This refrigerator looks futuristic and interesting" or "Having this in my kitchen would make my life more enjoyable." If the user selects one response, it is spoken in a synthesized voice.

[1908] Example prompt sentence:

[1909] "Users are watching an ad and are interested in the latest gadgets and technology. The ad asks, 'What do you think about the features of the new smart refrigerator?' If the user has difficulty answering, generate appropriate response suggestions to get them through the moment."

[1910] This system allows users to provide natural and appropriate responses while watching an advertisement, encouraging active interaction with the advertisement.

[1911] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1912] Step 1:

[1913] Conversation data collection

[1914] The device records the user's daily conversations. Specifically, for example, when the user talks with a friend about a gadget, the conversation is collected. The input is the user's voice, and the output is the recorded voice data.

[1915] Step 2:

[1916] Data analysis and emotion recognition

[1917] The device converts the recorded voice data into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). The input is the recorded voice data, and the output is the text data converted from the voice.

[1918] The terminal sends text data to the server. The input is the text data after speech recognition, and the output is the text data sent to the server.

[1919] The server analyzes the text data using a natural language processing engine (e.g., GPT-3) and extracts the user's interests, preferences, and characteristics. The input is text data, and the output is the analyzed interests, preferences, and characteristics information.

[1920] The server uses an emotion recognition engine (e.g., DeepAffects) to analyze the user's emotions from voice tone or specific keywords. The input is voice data or text data, and the output is the user's emotional information.

[1921] Step 3:

[1922] Creating a User Profile

[1923] The server creates a user profile based on the analyzed hobbies, preferences, characteristics, and emotional information. The input is hobbies, preferences, characteristics, and emotional information, and the output is a user profile. Examples of this include information such as "likes gadgets," "is interested in the latest technology," and "is happy when talking."

[1924] Step 4:

[1925] Real-time conversation processing while watching advertisements

[1926] While a user is watching an ad, a question related to the ad is displayed, for example, "What do you think about this new smartphone?" The input is the ad question, and the output is the user viewing the question.

[1927] The device records the question and converts it into text data using a speech recognition API. The input is the speech of the advertising question, and the output is text data, which is then sent to the server.

[1928] Step 5:

[1929] Response candidate generation and emotional adaptation

[1930] Based on the text data and user profile received by the server, a generative AI model (e.g., GPT-3) is used to generate multiple response candidates. The input is the text data and user profile, and the output is response candidates. For example, response candidates such as "This smartphone looks futuristic and interesting" or "This will make my life easier" are generated.

[1931] The server adapts the response candidates to the user's current em...

Claims

1. A means for recording conversation data of a user in his / her daily life; means for analyzing the recorded conversation data and interpreting the user's preferences and characteristics; A means for automatically generating response candidates based on the user's interests, preferences, and characteristics by analyzing the utterances of others in a situation where the user has difficulty speaking; means for presenting the generated answer candidates to a user and allowing the user to select one; means for outputting the selected response candidate by voice of the user; A system including:

2. 2. The system according to claim 1, wherein the recorded conversation data is converted into text data by speech recognition.

3. 10. The system of claim 1, wherein the system uses voice synthesis technology to output the user's voice.

Images