system

Abstract

Provide a system. A means for allowing a user to select a predetermined scene; means for generating a character image that matches a predetermined scene using an image generation means; a means for generating a sound based on the character image using a sound generating means; a means for generating conversation content according to a scene using a text generation means; A means for transcribing user utterances in real time; The system includes means for generating a response based on the generated transcription.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional training and education methods for improving communication skills often make it difficult to practice in real situations and provide insufficient feedback. This makes it difficult for university students and working adults, in particular, to efficiently acquire the verbalization skills necessary for real meetings and everyday communication situations. This makes it difficult for users to effectively express their opinions and thoughts, which can result in a fear of communication. [Means for solving the problem]

[0005] The present invention provides a system that recreates communication situations in a realistic manner and allows users to receive practical training. Specifically, the system includes a means for allowing a user to select a predetermined scene, a means for generating a character image that matches the predetermined scene using an image generation means, a means for generating audio based on the character image using an audio generation means, a means for generating conversation content appropriate to the scene using a text generation means, a means for transcribing user utterances in real time, and a means for generating a response based on the generated transcription. The system also includes a means for analyzing the generated conversation content and the user utterances and providing appropriate feedback, and a means for displaying the generated character image and audio in an AR environment.

[0006] "User" refers to a person who intends to use this system to improve their communication skills.

[0007] The "predetermined scene" refers to one of a plurality of virtual communication scenes selected by the user.

[0008] "Image generation means" refers to a series of processes and techniques for generating character images that match a given scene.

[0009] "Sound Generation Means" refers to a set of processes and techniques for generating sounds corresponding to the characters generated by the Image Generation Means.

[0010] "Text generation means" refers to a series of processes and techniques for generating conversational content for a given scene.

[0011] "Transcription" refers to the process or result of converting a user's speech into text data in real time.

[0012] "Response generator" refers to a set of techniques or processes for generating appropriate responses based on user utterances or transcriptions.

[0013] "Feedback" refers to information such as evaluation, advice, and areas for improvement provided based on the content of the user's speech and the progress of the conversation.

[0014] An "AR environment" refers to a virtual space that uses augmented reality technology to display generated character images and sounds superimposed on the real environment. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The verbalization ability development system of the present invention is equipped with multiple generation means and a real-time feedback function to efficiently improve the user's communication ability. This system is composed of an image generation means, a voice generation means, a text generation means, and a real-time transcription means, which work together to provide the user with an environment that is similar to an actual communication situation.

[0037] Overview of the program

[0038] 1. Launch the application and select a scene

[0039] User: Launches the dedicated application on a smartphone and selects a specific scene (e.g., a university club meeting, a company meeting, etc.).

[0040] Terminal: The initial application screen is displayed and the user's selection is accepted.

[0041] 2. Character image and voice generation

[0042] Terminal: Sends information about the scene selected by the user to the server and makes a request to generate a character image.

[0043] Server: Uses the image generation means to generate a character image suitable for the specified scene and sends the image data to the terminal.

[0044] Terminal: Next, the terminal uses the voice generating means to send a request to the server to generate a voice corresponding to the character image.

[0045] Server: Using the voice generation means, generates a voice corresponding to the specified character and sends the voice data to the terminal.

[0046] Device: Combines the received character image and audio and displays them in the user's AR environment.

[0047] 3. Conversation scenario generation

[0048] Server: Generates conversation content using a text generation means appropriate for the scene, and sends the text data to the terminal.

[0049] Terminal: The received text data is converted into voice data using a voice generation means and played back as the character's speech.

[0050] 4. User utterances and response generation

[0051] User: Listen to the character's speech through earphones and express their own thoughts and opinions.

[0052] Terminal: Receives user speech and transcribes it in real time.

[0053] Terminal: Sends the text of the utterance to the server and requests that a response be generated.

[0054] Server: Uses the text generation means to generate an appropriate response text in response to the user's utterance and sends that data to the terminal.

[0055] Terminal: The received response text is converted into audio data using a voice generation means, and played back as if spoken by a character.

[0056] 5. Providing Feedback

[0057] Server: Analyzes the data from the entire conversation and evaluates the user's comments and responses.

[0058] Server: Uses a text generation means to generate feedback including evaluation results and advice, and sends the data to the terminal.

[0059] Terminal: Display received feedback to the user.

[0060] Specific examples

[0061] Example 1: University club meeting

[0062] 1. The user launches the application and selects the "University Club Meeting" scene.

[0063] 2. The device sends the scene information to the server and makes a request to generate an appropriate character image.

[0064] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[0065] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[0066] 5. The device displays the received character image and audio in the AR environment.

[0067] 6. The user begins a conversation with the character, and the device transcribes what is said and sends it to the server.

[0068] 7. The server generates an appropriate response and sends it to the terminal.

[0069] 8. The device plays the character's response as if it were speaking it.

[0070] 9. After the conversation is over, the server generates feedback and sends it to the device.

[0071] 10. The device displays feedback to the user.

[0072] In this way, the system allows users to practice in an environment that is similar to a real communication situation, and can efficiently improve their verbalization ability.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] The user launches the dedicated application on their smartphone. The initial screen of the application is displayed, and they select a specific scene (e.g., a university club meeting, a company meeting, etc.).

[0076] Step 2:

[0077] The terminal receives the user's selection and sends scene information based on the selection to the server as a request, the request including detailed information about the selected scene.

[0078] Step 3:

[0079] The server analyzes the received request and sends instructions to the image generation means to generate a character image suitable for the scene. The image generation means generates a character image based on the specified features and returns the generated image data to the server.

[0080] Step 4:

[0081] The server receives the generated character image data and then transmits an instruction to generate a voice for the character to the voice generation means. The voice generation means generates a voice based on the specified characteristics and returns the generated voice data to the server.

[0082] Step 5:

[0083] The server then sends the received character image and audio data to the device, which then displays the character image in the AR environment and plays the audio based on the received data.

[0084] Step 6:

[0085] At the same time, the server uses a text generation means to generate conversation content according to the scene. The conversation text data generated based on the scene and character settings is transmitted from the server to the terminal.

[0086] Step 7:

[0087] The terminal transmits the received conversation text to the voice generation means, which generates voice data for the character to speak, and plays back the generated voice data so that the character can speak.

[0088] Step 8:

[0089] The user listens to the character's speech through earphones and expresses their own opinions and thoughts. The device receives the user's speech in real time and converts it into text (transcription).

[0090] Step 9:

[0091] The device transmits the transcribed user speech data in real time to the server, which then uses a text generation means to generate an appropriate response based on the received data.

[0092] Step 10:

[0093] The server transmits the generated response text data to the terminal. The terminal transmits the response text data to the voice generation means, which generates voice data for the character to speak. The voice data is then played back so that it sounds like the character is speaking.

[0094] Step 11:

[0095] After the conversation ends, the server analyzes the entire conversation data, evaluates the user's comments and responses, and generates feedback including the evaluation results and advice.

[0096] Step 12:

[0097] The server transmits the generated feedback data to the terminal, which displays the feedback to the user.

[0098] Step 13:

[0099] The user can choose to end the session and decide whether to save the conversation data and feedback. If the user chooses to save, the device sends the session data to the server, which then saves the data.

[0100] Example 1

[0101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0102] Conventional communication skills development systems have had the problem of making it difficult for users to practice in an environment that closely resembles a real-life communication situation. In particular, the lack of real-time feedback makes it difficult for users to immediately evaluate the accuracy and appropriateness of their own speech, hindering efficient skill improvement.

[0103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0104] In this invention, the server includes means for allowing a user to select a predetermined scene, means for generating a character image matching the predetermined scene using an image generation means, means for generating a voice based on the character image using a voice generation means, means for generating a conversation content corresponding to the scene using a text generation means, means for transcribing the user's utterance in real time, means for generating a response based on the generated transcription, means for displaying the generated character image and voice in an AR environment, means for generating and playing a response in real time in response to the user's utterance, and means for analyzing the generated conversation content and the user's utterance and providing appropriate feedback. This allows the user to efficiently train in an environment similar to an actual communication situation and instantly evaluate and improve the accuracy and appropriateness of their utterance.

[0105] "User" means an individual or organization that uses the system to improve their communication skills.

[0106] A "scene" refers to a specific situation or occasion selected by the user (e.g., a university club meeting, a company meeting, etc.).

[0107] "Image generation means" refers to a technology or device that enables a computer to generate corresponding images based on specified scenes or conditions, including, for example, deep learning models.

[0108] "Character image" refers to an image of a virtual character that is appropriate for the scene.

[0109] "Speech generation means" refers to technology or devices that synthesize speech based on text or digital data, including, for example, a speech synthesis engine.

[0110] "Text generation means" refers to a technology or device that generates natural language sentences based on input data, including, for example, a natural language processing model.

[0111] "Real-time transcription means" refers to technology or devices that convert user speech into text in real time, including, for example, speech recognition APIs.

[0112] The "means for generating a response" refers to a technology or device for generating an appropriate response text in response to a user's utterance.

[0113] An "AR environment" refers to a technological environment that displays virtual objects and information superimposed on the real environment.

[0114] "Means for providing feedback" refers to a technology or device that analyzes the content of a conversation that a user has had and provides advice or evaluation based on the results.

[0115] The verbalization ability development system of the present invention is designed to efficiently improve a user's communication ability and is equipped with multiple generation means and a real-time feedback function. This system is composed of an image generation means, a voice generation means, a text generation means, and a real-time transcription means, which work together to provide the user with an environment that is similar to an actual communication situation.

[0116] 1. Launching the application and selecting a scene

[0117] A user starts a dedicated application using a device such as a smartphone. On the initial screen, the user selects a specific scene (e.g., a university club meeting, a company meeting, etc.). The device then sends this scene information to the server.

[0118] 2. Character image and voice generation

[0119] The device sends a request to the server to generate a character image. The server uses an image generation means (e.g., a deep learning-based model) to generate a character image suitable for the specified scene and sends the data to the device. Next, the device sends a request to the server to generate audio based on the character image. The server uses an audio generation means (e.g., a speech synthesis engine) to generate audio corresponding to the specified character and sends this audio data to the device. The device combines these pieces of data and displays them in the user's AR environment.

[0120] 3. Conversation scenario generation

[0121] The terminal uses a text generation means to send a request to the server to generate conversation content appropriate for the scene. The server uses a text generation model (e.g., GPT-4 (registered trademark)) to generate conversation text appropriate for the scene and sends this text data to the terminal. The terminal converts the received text data into audio data and plays it as the character's speech.

[0122] 4. User utterances and response generation

[0123] The user listens to the character's speech through earphones and expresses their own thoughts and opinions. The device transcribes the user's speech in real time using the Google (registered trademark) Speech-to-Text API and sends this text data to the server. The server uses a text generation means to generate an appropriate response text in response to the user's speech and sends this data to the device. The device converts the received response text into audio data and plays it back as if the character were speaking the response.

[0124] 5. Providing Feedback

[0125] The server analyzes the data from the entire conversation and evaluates the user's comments and responses. Based on the analysis results, the server uses a text generation means to generate feedback including the evaluation results and advice, and sends this data to the terminal. The terminal displays the received feedback to the user. The user can then identify areas for improvement based on the feedback and use them in the next conversation.

[0126] Specific examples

[0127] For example, if the user selects the scene "University Club Meeting," the following steps are taken:

[0128] 1. The user launches the application and selects the "University Club Meeting" scene.

[0129] 2. The device sends the scene information to the server and makes a request to generate an appropriate character image.

[0130] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[0131] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[0132] 5. The device displays the received character image and audio in the AR environment.

[0133] 6. The user begins a conversation with the character, and the device transcribes what is said and sends it to the server.

[0134] 7. The server generates an appropriate response and sends it to the terminal.

[0135] 8. The device plays the character's response as if it were speaking it.

[0136] 9. After the conversation is over, the server generates feedback and sends it to the device.

[0137] 10. The device displays feedback to the user.

[0138] In this way, the system allows users to train in an environment that is close to a real communication situation, and can efficiently improve their verbalization ability. In this system, the following are examples of prompt sentences when using a generative AI model:

[0139] ---

[0140] Example prompt sentence:

[0141] "Imagine a scene at a university club meeting where an idea for a new project is proposed, and generate how the characters would react."

[0142] ---

[0143] This allows users to hone their practical communication skills through conversations tailored to specific situations.

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1: Launch the application and select a scene

[0146] 1. The user taps the dedicated application on their smartphone to launch the app.

[0147] Input: Tap the app icon.

[0148] Output: The initial screen of the application is displayed.

[0149] 2. The terminal displays the initial screen and waits for user operation.

[0150] What it does: Draws the UI components on the initial screen.

[0151] 3. On the initial screen that appears, the user selects a scene such as a "university club meeting" or a "company meeting."

[0152] Input: Tap to select a scene.

[0153] Output: The information of the selected scene is saved to the device.

[0154] Step 2: Generate character images and sounds

[0155] 1. The device sends information about the scene selected by the user to the server and makes a request to generate a character image.

[0156] Input: Scene information.

[0157] Output: Request sent.

[0158] What it does: Creates and sends an HTTP request.

[0159] 2. The server receives the request and uses the image generation means to generate a character image suitable for the specified scene.

[0160] Input: A request for scene information.

[0161] Output: Character image data.

[0162] Action: Running an image generation model.

[0163] 3. The server sends the generated character image data to the terminal.

[0164] Input: Character image data.

[0165] Output: Sending image data.

[0166] What it does: Generates and sends an HTTP response.

[0167] 4. The device receives the character image and stores it in memory.

[0168] Input: Character image data.

[0169] Output: Saving image data.

[0170] Action: Processing data storage.

[0171] 5. The device sends a request to the server to generate audio based on the character image.

[0172] Input: Character image information.

[0173] Output: Request sent.

[0174] What it does: Creates and sends an HTTP request.

[0175] 6. The server uses the voice generation means to generate a voice corresponding to the specified character.

[0176] Input: Request for character information.

[0177] Output: Character voice data.

[0178] What it does: Runs the speech synthesis engine.

[0179] 7. The server sends the generated voice data to the terminal.

[0180] Input: Character voice data.

[0181] Output: Sending audio data.

[0182] What it does: Generates and sends an HTTP response.

[0183] 8. The device combines the character image and audio data and prepares to display them in the user's AR environment.

[0184] Input: Character image and audio data.

[0185] Output: AR display preparation.

[0186] What it does: Synchronize and prepare data.

[0187] Step 3: Generate conversation scenarios

[0188] 1. The device sends a request to the server to generate conversation content appropriate for the scene.

[0189] Input: Scene information.

[0190] Output: Request sent.

[0191] What it does: Creates and sends an HTTP request.

[0192] 2. The server uses a text generation means to generate conversation content appropriate for the scene.

[0193] Input: A request for scene information.

[0194] Output: Text data of the conversation scenario.

[0195] What it does: Run a text generation model.

[0196] 3. The server sends the text data of the generated conversation scenario to the terminal.

[0197] Input: Text data of a conversation scenario.

[0198] Output: Sending text data.

[0199] What it does: Generates and sends an HTTP response.

[0200] 4. The device converts the received text data into audio data and plays it back as the character's speech.

[0201] Input: Text data of a conversation scenario.

[0202] Output: Generate audio data.

[0203] What it does: Performs text-to-speech (TTS).

[0204] Step 4: User utterances and response generation

[0205] 1. The device plays the character's voice.

[0206] Input: Audio data.

[0207] Output: Audio playback.

[0208] Action: Performs audio playback function.

[0209] 2. The user listens to the character's speech and expresses their own thoughts and opinions.

[0210] Input: Character speech.

[0211] Output: User utterance.

[0212] Action: User's auditory comprehension and speech.

[0213] 3. The device collects the user's speech and transcribes it in real time.

[0214] Input: User speech.

[0215] Output: Text data.

[0216] What it does: Transcription using the Google Speech-to-Text API.

[0217] 4. The device sends the transcribed text data to the server and makes a request to generate a response.

[0218] Input: Speech text data.

[0219] Output: Request sent.

[0220] What it does: Creates and sends an HTTP request.

[0221] 5. The server uses a text generation means to generate an appropriate response based on the user's utterance.

[0222] Input: User utterance text data.

[0223] Output: Response text data.

[0224] What it does: Run a text generation model.

[0225] 6. The server sends the response text data to the terminal.

[0226] Input: Response text data.

[0227] Output: Sending text data.

[0228] What it does: Generates and sends an HTTP response.

[0229] 7. The device converts the received response text into audio data and plays it back as if the character were speaking the response.

[0230] Input: Response text data.

[0231] Output: Generation and playback of audio data.

[0232] What it does: Performs text-to-speech (TTS) and plays audio.

[0233] Step 5: Provide feedback

[0234] 1. The server analyzes the data from the entire conversation and evaluates the user's comments and responses.

[0235] Input: Utterance and response data.

[0236] Output: Evaluation results.

[0237] Action: Run the data analysis model.

[0238] 2. The server generates feedback based on the evaluation results using a text generation means.

[0239] Input: Analysis results.

[0240] Output: Feedback text data.

[0241] What it does: Run a text generation model.

[0242] 3. The server sends the feedback data to the terminal.

[0243] Input: Feedback text data.

[0244] Output: Sending text data.

[0245] What it does: Generates and sends an HTTP response.

[0246] 4. The device displays the received feedback to the user.

[0247] Input: Feedback text data.

[0248] Output: Display feedback.

[0249] Action: Drawing a UI component.

[0250] (Application example 1)

[0251] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0252] There is a need to provide a training environment in brick-and-mortar stores that allows store employees to efficiently improve their customer service skills. In particular, it is important to improve conversation skills in specific situations, such as explaining new products or handling complaints. However, because training that replicates real-world situations is time-consuming and costly, there is a need to develop an easily accessible simulation tool.

[0253] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0254] In this invention, the server includes means for allowing a user to select a predetermined scene, means for generating a character image that matches the predetermined scene using an image generation means, means for generating a voice based on the character image using a voice generation means, means for generating a conversation content corresponding to the scene using a text generation means, means for transcribing utterances from the user in real time, means for generating a response based on the generated transcription, and means for generating a conversation scenario corresponding to a specific situation in customer service work. This provides a simulation environment that is close to a real customer service situation, enabling store clerks to efficiently improve their customer service skills.

[0255] "Means for allowing the user to select a specific scene" refers to a function that provides an interface for the user to use the application to select a specific scene or situation (e.g., explaining a new product, handling a complaint, etc.).

[0256] "Image generation means" refers to a technology for generating character images that match a specific scene, and is a means for generating high-quality images using machine learning and deep learning.

[0257] The "voice generation means" is a technology for generating appropriate voice based on the generated character image, and is a means for converting text data into voice data.

[0258] The "text generation means" is a technology for generating conversation content appropriate to the scene, and is a means for generating appropriate text data using natural language processing technology.

[0259] "Means for transcribing user speech in real time" refers to a technology for instantly converting what a user says into text data, and is a means that uses voice recognition technology.

[0260] The "means for generating a response based on the generated transcription" is a technology for receiving a user's speech as text data and generating an appropriate response in response to it.

[0261] "Means for generating conversation scenarios according to specific situations in customer service work" refers to technology for generating scenarios in specific situations related to customer service work, and is a means for automatically generating conversation content according to user selections.

[0262] An "augmented reality environment" is a technology that displays virtual information overlaid on the real world, and is an environment that integrates the real environment with virtual information using multimedia content such as 3D models, videos, and audio.

[0263] This invention provides a system for efficiently training in customer service work. The system has a function that allows a user to select a predetermined scene and experience a virtual customer service situation based on that scene. The following describes in detail how this system can be realized.

[0264] Program Structure

[0265] The system consists of the following main components:

[0266] 1. A means for allowing the user to select a specific scene: A user interface is provided that allows the user to select a scene, such as explaining a new product or handling a complaint.

[0267] 2. Image generation means: A deep learning model is used to generate character images suitable for a given scene. For example, we use TENSORFLOW (registered trademark).

[0268] 3. Voice generation method: Google Cloud Text-to-Speech is used to generate voice based on the generated character image.

[0269] 4. Text generation means: Uses natural language processing models such as GPT-3 (registered trademark) to generate conversational content appropriate to the situation.

[0270] 5. A way to transcribe user speech in real time: Use Google Cloud Speech-to-Text.

[0271] 6. Generating responses based on the generated transcription: The GPT-3 model is used to generate appropriate responses to the user's utterances.

[0272] 7. A means of generating conversation scenarios for specific situations in customer service: Automatically create conversation scenarios for specific situations based on user selections.

[0273] Hardware and Software Configuration

[0274] Hardware: Smartphone (iOS or ANDROID (registered trademark)), earphones

[0275] software:

[0276] TensorFlow: Image Generation

[0277] Google Cloud Text-to-Speech: Audio Generation

[0278] Google Cloud Speech-to-Text: Real-time transcription

[0279] GPT-3: Text and response generation

[0280] Specific examples

[0281] Example: New product description

[0282] The user launches the application and selects the "Explanation of a new product" scene. The application sends the selection information to the server and requests the generation of an appropriate character image and sound. The server generates a character image and sound using the image generation means and sound generation means, and sends them to the smartphone. The smartphone displays the received character image and sound in the augmented reality environment, and the user begins a virtual customer service interaction with the character.

[0283] When a user says, "Today I'll talk about our new product," the smartphone transcribes the speech in real time and sends it to the server. The server then generates an appropriate response, such as repeating what the user said or asking follow-up questions, and sends it to the smartphone. The character then replies, "Can you tell me about the features of our new product?" and the conversation continues.

[0284] Prompt Sentence Examples

[0285] Below are some example inputs for a generative AI model:

[0286] Please generate a conversation scenario with questions and answers from customers when explaining a new product. The specific product name is Smartwatch Z. Please create the scenario by including points that customers are likely to have questions about.

[0287] In this way, the present invention provides a system for efficiently conducting customer service training in a real store, allowing users to train in a realistic simulation environment.

[0288] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0289] Step 1:

[0290] The user selects a given scene

[0291] The user launches the customer service trainer app on their smartphone and selects a specific scene, such as "explaining a new product" or "handling a complaint," from the scene selection screen. The smartphone then sends the information about the selected scene to the server.

[0292] Input: User scene selection

[0293] Output: Send scene selection information to the server

[0294] Step 2:

[0295] Character image generation

[0296] Based on the received scene selection information, the server uses a deep learning model (e.g., TensorFlow) to generate a character image suitable for the specified scene. The generated character image is then sent from the server to the smartphone.

[0297] Input: Scene selection information

[0298] Output: Character image

[0299] Step 3:

[0300] Audio generation

[0301] The server uses Google Cloud Text-to-Speech to generate voice for the character based on the generated character image, and the generated voice is sent from the server to the smartphone.

[0302] Input: Character image

[0303] Output: Character voice

[0304] Step 4:

[0305] Conversation scenario generation using text generation

[0306] The server uses GPT-3 to generate conversation content based on the scene selected by the user. The conversation scenario is sent as text data from the server to the smartphone, which then converts it into audio data and plays it back.

[0307] Input: Scene selection information

[0308] Output: Conversation scenario (text data)

[0309] Step 5:

[0310] Transcription of user utterances

[0311] Users listen to the character's speech through headphones and express their thoughts and opinions. The smartphone then transcribes the speech in real time using Google Cloud Speech-to-Text and sends the data to a server.

[0312] Input: User utterance

[0313] Output: Transcription data

[0314] Step 6:

[0315] Generating a response

[0316] The server analyzes the text data using the GPT-3 model and generates an appropriate response, which is then sent to the smartphone, where it is converted into audio and played back.

[0317] Input: Transcription data

[0318] Output: Response text data

[0319] Step 7:

[0320] Providing feedback

[0321] The server analyzes the entire conversation data, evaluates the user's comments and responses, and generates appropriate feedback. The feedback is sent as text data from the server to the smartphone and displayed to the user.

[0322] Input: Data for the entire conversation

[0323] Output: Feedback data

[0324] In this way, the system allows users to efficiently improve their customer service skills in a realistic simulation environment.

[0325] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0326] The verbalization ability development system of the present invention is equipped with multiple generation means and a real-time feedback function to efficiently improve the user's communication ability. This system is composed of an image generation means, a voice generation means, a text generation means, an emotion engine, and a real-time transcription means, which work together to provide the user with an environment that is similar to an actual communication situation.

[0327] Overview of the program

[0328] 1. Launch the application and select a scene

[0329] User: Launches the dedicated application on a smartphone and selects a specific scene (e.g., a university club meeting, a company meeting, etc.).

[0330] Terminal: The initial application screen is displayed and the user's selection is accepted.

[0331] 2. Character image and voice generation

[0332] Terminal: Sends information about the scene selected by the user to the server and makes a request to generate a character image.

[0333] Server: Uses the image generation means to generate a character image suitable for the specified scene and sends the image data to the terminal.

[0334] Terminal: Next, the terminal uses the voice generating means to send a request to the server to generate a voice corresponding to the character image.

[0335] Server: Using the voice generation means, generates a voice corresponding to the specified character and sends the voice data to the terminal.

[0336] Device: Combines the received character image and audio and displays them in the user's AR environment.

[0337] 3. Conversation scenario generation

[0338] Server: Generates conversation content using a text generation means appropriate for the scene, and sends the text data to the terminal.

[0339] Terminal: The received text data is converted into voice data using a voice generation means and played back as the character's speech.

[0340] 4. User utterances and response generation

[0341] User: Listens to the character's speech through earphones and expresses his or her own opinions and thoughts. The device receives the user's speech in real time and converts it into text (transcription).

[0342] Device: The transcribed user speech data is sent to the server in real time. The server then analyzes the user's emotions using the received data and the emotion engine.

[0343] Server: Uses a text generation means to generate an appropriate response text based on the user's utterance and emotions, and sends the data to the terminal.

[0344] Terminal: The received response text is converted into audio data using a voice generation means, and played back as if spoken by a character.

[0345] 5. Providing feedback and using emotions

[0346] Server: Analyzes the overall conversation data and the emotional data obtained from the emotion engine, and evaluates the user's comments and responses.

[0347] Server: Generates feedback including evaluation results and advice that take emotions into account, and sends this data to the device.

[0348] Terminal: Display received feedback to the user.

[0349] Specific examples

[0350] Example 1: University club meeting

[0351] 1. The user launches the application and selects the "University Club Meeting" scene.

[0352] 2. The device sends the scene information to the server and makes a request to generate an appropriate character image.

[0353] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[0354] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[0355] 5. The device displays the received character image and audio in the AR environment.

[0356] 6. The user begins a conversation with the character, and the device transcribes what is said and sends it to the server.

[0357] 7. The server generates a response based on the user's speech and the emotion engine and sends it to the device.

[0358] 8. The device plays the character's response as if it were speaking it.

[0359] 9. After the conversation ends, the server generates a rating based on the emotional data along with feedback and sends it to the device.

[0360] 10. The device displays feedback to the user.

[0361] In this way, the system allows users to train in an environment that is similar to a real communication situation, and can efficiently improve their verbalization and emotion recognition abilities.

[0362] The processing flow will be explained below.

[0363] Step 1:

[0364] The user launches the dedicated application on their smartphone. The initial screen of the application is displayed, and they select a specific scene (e.g., a university club meeting, a company meeting, etc.).

[0365] Step 2:

[0366] The terminal receives the user's selection and sends scene information based on the selection to the server as a request, the request including detailed information about the selected scene.

[0367] Step 3:

[0368] The server analyzes the received request and sends instructions to the image generation means to generate a character image suitable for the scene. The image generation means generates a character image based on the specified features and returns the generated image data to the server.

[0369] Step 4:

[0370] The server receives the generated character image data and then transmits an instruction to generate a voice for the character to the voice generation means. The voice generation means generates a voice based on the specified characteristics and returns the generated voice data to the server.

[0371] Step 5:

[0372] The server then sends the received character image and audio data to the device, which then displays the character image in the AR environment and plays the audio based on the received data.

[0373] Step 6:

[0374] At the same time, the server uses a text generation means to generate conversation content according to the scene. The conversation text data generated based on the scene and character settings is transmitted from the server to the terminal.

[0375] Step 7:

[0376] The terminal transmits the received conversation text to the voice generation means, which generates voice data for the character to speak, and plays back the generated voice data so that the character can speak.

[0377] Step 8:

[0378] The user listens to the character's speech through earphones and expresses their own opinions and thoughts. The device receives the user's speech in real time and converts the speech into text (transcription).

[0379] Step 9:

[0380] The device transmits the transcribed user speech data to the server in real time, and the server uses an emotion engine to analyze the user's emotions based on the received data and facial expressions.

[0381] Step 10:

[0382] The server uses the text generation means based on the analysis results to generate an appropriate response text according to the user's utterance and emotion. The generated response text is sent from the server to the terminal.

[0383] Step 11:

[0384] The terminal transmits the received response text to the voice generation means, which generates voice data for the character to speak, and plays back the generated voice data so that the character can speak it.

[0385] Step 12:

[0386] Once the conversation is over, the server analyzes the entire conversation data and the emotional data obtained from the emotion engine, evaluates the user's comments and responses, and generates feedback including the evaluation results and advice.

[0387] Step 13:

[0388] The server transmits the generated feedback data to the terminal, which displays the feedback to the user.

[0389] Step 14:

[0390] The user can choose to end the session and decide whether to save the conversation data and feedback. If the user chooses to save, the device sends the session data to the server, which then stores the data.

[0391] Example 2

[0392] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0393] Current systems for improving communication skills have the problem that it is difficult for users to receive training in an environment that closely resembles a real communication situation. Furthermore, there is a lack of systems that can analyze not only the content of a user's speech but also their emotions and provide appropriate feedback based on that. This makes it difficult to simultaneously improve both a user's language ability and their emotion recognition ability.

[0394] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0395] In this invention, the server includes means for a user to select a predetermined scene, means for generating a character image that matches the predetermined scene using an image generation means, means for generating a voice based on the character image using a voice generation means, means for generating a conversation content according to the scene using a text generation means, means for transcribing the user's speech in real time, means for generating a response based on the generated transcription, means for analyzing the user's speech using an emotion analysis engine and generating an appropriate response, and means for displaying the generated character image and voice in an augmented reality environment. This allows the user to train in an environment that resembles an actual communication situation, thereby simultaneously improving both language ability and emotion recognition ability.

[0396] A "user" is an individual or corporation that uses this system to achieve a specific purpose.

[0397] A "scene" is a specific situation or environment selected by the user, which is the subject of training or simulation.

[0398] "Image generation means" refers to technology or equipment for generating character images that match a specific scene.

[0399] "Character images" are image data of people or objects generated according to the scene.

[0400] "Sound generation means" refers to technology or devices that generate sounds based on character images.

[0401] "Text generation means" refers to a technology or device that generates conversation content appropriate for a specific scene.

[0402] "Transcription" is the process of converting a user's speech from audio to text in real time.

[0403] "Response generation means" refers to technology or devices that generate appropriate responses based on the generated transcription or other data.

[0404] An "emotion analysis engine" refers to technology or equipment for analyzing emotions from a user's speech or behavior.

[0405] "Augmented reality environment" refers to technology that displays virtual information overlaid on the real environment, and the environment that uses it.

[0406] This invention is a system for improving a user's communication skills, which provides training by selecting a specific scene and having a conversation with a character. This system is configured to include the following means.

[0407] First, a user launches a system-specific application using a device such as a smartphone or tablet. On the initial screen of the application, the user selects a desired scene from multiple scenes (e.g., a university club meeting, a company meeting, etc.).

[0408] The selected scene information is transmitted from the terminal to the server. The server uses an image generation means to generate a character image that matches the selected scene. For example, a deep learning model can be used for this image generation. The generated character image data is transmitted from the server to the terminal.

[0409] Next, the terminal sends a request to the server to generate a voice for the character. The server uses a voice generation means to generate a voice based on the character image. A text-to-speech engine can be used for generating the voice. The generated voice data is sent to the terminal, and the terminal displays the character image and voice in combination.

[0410] A text generation means within the server is used to generate the conversation scenario. The text generation means uses, for example, a generative AI model that makes full use of natural language processing technology to generate conversation content that corresponds to the scene selected by the user. The generated text data is sent to the terminal, where it is converted into audio data via a text-to-speech engine and played back as the character's speech.

[0411] The user listens to the character's speech through earphones and a microphone and expresses their opinions and thoughts. The device records the user's speech in real time and converts it into text (transcription). The transcribed data is then sent to the server.

[0412] The server uses an emotion analysis engine to analyze the user's speech data and identify emotions. It then uses a text generation means to generate an appropriate response based on the user's speech content and emotions. This response data is sent to the terminal, converted into voice data, and played as the character's speech.

[0413] As a concrete example, if you select the "University Club Meeting" scene, the system will behave as follows:

[0414] 1. The user launches the application and selects the "University Club Meeting" scene.

[0415] 2. The device sends the selected scene ID to the server.

[0416] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[0417] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[0418] 5. The device displays the character image and sound in the AR environment.

[0419] 6. The user begins to interact with the character, and the device transcribes what is said and sends it to the server.

[0420] 7. The server analyzes the user's speech and emotions, generates an appropriate response, and sends it to the device.

[0421] 8. The device plays back the response as if it were being spoken by a character.

[0422] 9. After the conversation ends, the server generates feedback based on the emotion data and sends it to the device.

[0423] 10. The device displays feedback to the user.

[0424] An example prompt is:

[0425] You have selected a university club meeting scene. The server and device interact, generating character images and voices. The following conversation unfolds between you and the character:

[0426] "Today I would like to discuss a new project. I'd like to hear your thoughts."

[0427] How would you respond to this statement?

[0428] In this way, users can receive training in an environment that is close to a real communication situation, improving both their language skills and their emotion recognition abilities.

[0429] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0430] Step 1:

[0431] The user launches a dedicated application on the device and selects a specific scene. At this time, scene information (e.g., scene ID) is provided as input. The device accepts the user's selection and sends the scene information to the server. The output at this stage is that the scene information is sent to the server.

[0432] Step 2:

[0433] The device sends scene information to the server and requests the server to generate a character image. The server receives the scene information as input and generates a character image using an image generation means. Specifically, it uses an AI model to design a character image that fits the scene and generates its data. This image data is generated as output and sent from the server to the device.

[0434] Step 3:

[0435] The terminal receives the character image sent from the server and prepares to display it. It then sends a request to the server again to generate voice. This request includes information about the character image, which is passed to the server as input. The server uses a voice generation means to generate voice for the character. At this time, it uses a text-to-speech engine to convert the specified text into voice data. The generated voice data is sent from the server to the terminal.

[0436] Step 4:

[0437] The server uses a text generation means to generate dialogue content corresponding to the scene selected by the user. Scene information and prompt sentences are used as input, and natural language processing technology is used to generate appropriate dialogue text. This dialogue text is generated as output and sent to the terminal. The terminal converts this text into audio data and plays it back as if the character were speaking.

[0438] Step 5:

[0439] The user listens to the character's speech through headphones and expresses their own opinions and thoughts through a microphone. The user's speech is input, and the device records and transcribes it in real time. This transcription data is output and sent to the server.

[0440] Step 6:

[0441] The server uses a sentiment analysis engine to analyze the user's speech data. It receives the transcription data as input and analyzes it using a machine learning model to identify the user's emotions. The analysis results are output and used as input data for generating an appropriate response text. It uses a text generation means to generate a response text based on the user's speech content and emotions. The generated response text is sent to the device.

[0442] Step 7:

[0443] The device converts the received response text into audio data and plays it as if the character were speaking.The device converts the text into audio using a voice generation means and displays it as if the character is speaking in the AR environment.

[0444] Step 8:

[0445] After the conversation ends, the server analyzes all conversation content and emotional data and generates feedback for the user. The emotional data and spoken content are used as inputs for this analysis. The generated feedback information is output and sent to the device. The device displays the feedback to the user, allowing the user to improve themselves based on the results.

[0446] (Application example 2)

[0447] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0448] There is a need to efficiently and effectively improve the customer service skills of employees in brick-and-mortar stores. However, traditional training methods have difficulty providing an environment similar to actual customer interactions, limiting their effectiveness in improving communication skills. Furthermore, because training feedback is not provided in real time, it is difficult for employees to quickly understand and correct areas for improvement. Another issue is the lack of a system for analyzing employees' emotions and attitudes and providing appropriate advice.

[0449] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for allowing a user to select a predetermined scene, means for generating a person image matching the predetermined scene using an image generation means, means for generating a voice based on the person image using a voice generation means, means for generating a dialogue content appropriate to the scene using a text generation means, means for transcribing user utterances in real time, means for generating a response based on the generated transcription, emotion analysis means for analyzing the transcribed user utterances and emotions, means for displaying the generated response and analysis results in a substitute real environment, and means for providing feedback based on the response. This makes it possible to conduct training while receiving real-time feedback in an environment that resembles an actual customer service situation, thereby effectively improving employees' communication skills and customer service skills.

[0450] The "means for allowing the user to select a predetermined scene" refers to a means for providing an interface that allows the user to select a specific scenario or situation in advance.

[0451] "Image generation means" refers to an algorithm or program for automatically generating a person image that matches a scene selected by a user.

[0452] The "audio generating means" refers to a system or program for generating appropriate audio data based on the generated person image.

[0453] "Text generation means" refers to software or algorithms that automatically generate dialogue content appropriate for a specific situation.

[0454] "Means for transcribing user utterances in real time" refers to technology or devices that convert user speech into text in real time.

[0455] "Means for generating a response based on the generated transcription" refers to a mechanism for analyzing the transcribed content of a user's speech and automatically generating an appropriate corresponding response.

[0456] "Emotion analysis means" refers to technology that analyzes a user's comments and the emotions they express at the time, and provides appropriate actions and feedback based on the results.

[0457] "Means for displaying in a virtual reality environment" refers to a system or device for displaying the generated human image and audio in the virtual or alternate reality space experienced by the user.

[0458] "Means for providing feedback" refers to a mechanism that provides information for improvement and evaluation in real time based on the results of users' actions and comments.

[0459] This invention is a system for improving customer service skills in brick-and-mortar stores. The system aims to provide training that is similar to actual customer service situations by having users interact with customers in a virtual environment.

[0460] System Overview

[0461] The system includes the following main components:

[0462] 1. A means by which the user can select a given scene:

[0463] It provides an interface that allows users (store employees) to select a scene (e.g., introducing a new product, working at the cash register, etc.).

[0464] For example, by selecting a menu on the smart glasses display, the user can easily select the appropriate scene.

[0465] 2. Image generation means:

[0466] An algorithm or program for generating a person image that matches a selected scene.

[0467] Here, we use technology such as DALL-E from OpenAI (registered trademark) as image generation technology.

[0468] 3. Sound Generation Means:

[0469] A technology that generates appropriate voice data based on the generated person image.

[0470] As an example, we use Azure® Cognitive Services to generate a virtual voice of the customer.

[0471] 4. Text generation methods:

[0472] A technology that automatically generates dialogue content based on the selected situation.

[0473] Here, we use a text generation model (e.g., GPT-3) to generate scenario-based questions and answers.

[0474] 5. Means of transcribing user comments in real time:

[0475] A technology that converts user speech into text in real time.

[0476] It uses Google Speech-to-Text technology to instantly transcribe what users say.

[0477] 6. How to generate a response based on the generated transcript:

[0478] A system that automatically generates appropriate responses based on transcription data.

[0479] Again, a text generation model (e.g., GPT-3) is used to generate responses.

[0480] 7. Emotion analysis means:

[0481] A technology that analyzes a user's comments and their emotions at the time, and provides appropriate actions and feedback based on the results.

[0482] IBM Watson (registered trademark) is used as the sentiment analysis engine.

[0483] 8. Means of display in virtual reality environment:

[0484] A technology that displays generated human images and audio in a virtual or augmented reality environment.

[0485] Smart glasses (e.g., Microsoft® HoloLens®) are used to provide visual and auditory information to the user.

[0486] 9. Ways to provide feedback:

[0487] A system that provides real-time feedback based on user actions and comments.

[0488] Based on the analysis results, appropriate evaluations and improvement advice are provided to users in real time.

[0489] Specific examples

[0490] For example, in a scenario where a new product is being introduced, a user (employee) can wear smart glasses and undergo training. The user is asked questions such as, "What are the features of the new product?" and as the user responds, the system performs real-time transcription and sentiment analysis. As a result, feedback is provided on the accuracy of the response and the employee's attitude.

[0491] Prompt Sentence Examples

[0492] An example prompt for a new product introduction scene is as follows:

[0493] "Please tell me the features of this product."

[0494] Points to consider: Product features, benefits, user expectations, etc.

[0495] In this way, the system helps store employees to receive training in an environment that closely resembles an actual customer service situation, effectively improving their communication and customer service skills.

[0496] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0497] Step 1:

[0498] The user puts on the smart glasses and launches the application. Through the interface, the user selects a specific training scenario (e.g., a new product introduction) based on a real customer service scene. Scene information is selected as input, and the selected information is sent to the device as output.

[0499] Step 2:

[0500] The device sends the selected scene information to the server, which receives this information and uses an image generation model (e.g., OpenAI DALL-E) to generate a virtual customer persona that fits the scene. The scene information is given as input, and the generated persona is sent as output to the device.

[0501] Step 3:

[0502] The device sends a voice generation request to the server based on the received person image. The server uses a voice generation model (e.g., Azure Cognitive Services) to generate voice data that matches the scene in the voice of the virtual customer. The person image is given as input, and the voice data is sent to the device as output.

[0503] Step 4:

[0504] The device displays and plays the received image and audio data. The virtual customer is displayed on the smart glasses' display and the corresponding audio is played. By providing the user with visual and audio information, a realistic customer service scene is simulated.

[0505] Step 5:

[0506] The server uses a text generation model (e.g., GPT-3) to generate dialogue content based on the selected scene. Scene information is given as input, and appropriate questions and comments are sent as output to the device as text data.

[0507] Step 6:

[0508] The device converts the received text data into voice data using a speech generation model and plays it back as if the virtual customer were speaking it, for example, asking a question such as, "What are the features of this product?"

[0509] Step 7:

[0510] The user speaks their opinions and answers to the questions posed by the virtual customer through the microphone in the smart glasses, and the voice data is sent to the terminal as input.

[0511] Step 8:

[0512] The device uses speech recognition technology (e.g., Google Speech-to-Text) to transcribe and convert user voice data into text in real time. Voice data is given as input and text data is generated as output.

[0513] Step 9:

[0514] The device sends the generated text data to a server, which uses an emotion analysis engine (e.g., IBM Watson) to analyze the user's speech and the emotions expressed at the time. The text data is given as input, and the emotion analysis results are obtained as output.

[0515] Step 10:

[0516] The server uses a text generation model to generate an appropriate response text based on the user's utterances and emotions. The analysis results and the user's utterances are given as input, and the response text is generated as output.

[0517] Step 11:

[0518] The terminal converts the response text into speech data and plays it back as if the virtual customer were speaking the response, again using a speech model, and provides the generated speech data to the user.

[0519] Step 12:

[0520] The server generates feedback based on all conversation data and sentiment analysis results. Using a generative AI model, the server provides feedback to users, including areas for improvement and evaluations. Conversation data and analysis results are given as input, and feedback is generated as output.

[0521] Step 13:

[0522] The device then displays the generated feedback to the user, who is then shown on the smart glasses display, providing the user with specific points to improve and a visual assessment.

[0523] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0524] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0525] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0526] [Second embodiment]

[0527] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0528] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0529] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0530] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0531] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0532] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0533] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0534] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0535] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0536] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0537] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0538] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0539] The verbalization ability development system of the present invention is equipped with multiple generation means and a real-time feedback function to efficiently improve the user's communication ability. This system is composed of an image generation means, a voice generation means, a text generation means, and a real-time transcription means, which work together to provide the user with an environment that is similar to an actual communication situation.

[0540] Overview of the program

[0541] 1. Launch the application and select a scene

[0542] User: Launches the dedicated application on a smartphone and selects a specific scene (e.g., a university club meeting, a company meeting, etc.).

[0543] Terminal: The initial application screen is displayed and the user's selection is accepted.

[0544] 2. Character image and voice generation

[0545] Terminal: Sends information about the scene selected by the user to the server and makes a request to generate a character image.

[0546] Server: Uses the image generation means to generate a character image suitable for the specified scene and sends the image data to the terminal.

[0547] Terminal: Next, the terminal uses the voice generating means to send a request to the server to generate a voice corresponding to the character image.

[0548] Server: Using the voice generation means, generates a voice corresponding to the specified character and sends the voice data to the terminal.

[0549] Device: Combines the received character image and audio and displays them in the user's AR environment.

[0550] 3. Conversation scenario generation

[0551] Server: Generates conversation content using a text generation means appropriate for the scene, and sends the text data to the terminal.

[0552] Terminal: The received text data is converted into voice data using a voice generation means and played back as the character's speech.

[0553] 4. User utterances and response generation

[0554] User: Listen to the character's speech through earphones and express their own thoughts and opinions.

[0555] Terminal: Receives user speech and transcribes it in real time.

[0556] Terminal: Sends the text of the utterance to the server and requests that a response be generated.

[0557] Server: Uses the text generation means to generate an appropriate response text in response to the user's utterance and sends that data to the terminal.

[0558] Terminal: The received response text is converted into audio data using a voice generation means, and played back as if spoken by a character.

[0559] 5. Providing Feedback

[0560] Server: Analyzes the data from the entire conversation and evaluates the user's comments and responses.

[0561] Server: Uses a text generation means to generate feedback including evaluation results and advice, and sends the data to the terminal.

[0562] Terminal: Display received feedback to the user.

[0563] Specific examples

[0564] Example 1: University club meeting

[0565] 1. The user launches the application and selects the "University Club Meeting" scene.

[0566] 2. The device sends the scene information to the server and makes a request to generate an appropriate character image.

[0567] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[0568] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[0569] 5. The device displays the received character image and audio in the AR environment.

[0570] 6. The user begins a conversation with the character, and the device transcribes what is said and sends it to the server.

[0571] 7. The server generates an appropriate response and sends it to the terminal.

[0572] 8. The device plays the character's response as if it were speaking it.

[0573] 9. After the conversation is over, the server generates feedback and sends it to the device.

[0574] 10. The device displays feedback to the user.

[0575] In this way, the system allows users to practice in an environment that is similar to a real communication situation, and can efficiently improve their verbalization ability.

[0576] The processing flow will be explained below.

[0577] Step 1:

[0578] The user launches the dedicated application on their smartphone. The initial screen of the application is displayed, and they select a specific scene (e.g., a university club meeting, a company meeting, etc.).

[0579] Step 2:

[0580] The terminal receives the user's selection and sends scene information based on the selection to the server as a request, the request including detailed information about the selected scene.

[0581] Step 3:

[0582] The server analyzes the received request and sends instructions to the image generation means to generate a character image suitable for the scene. The image generation means generates a character image based on the specified features and returns the generated image data to the server.

[0583] Step 4:

[0584] The server receives the generated character image data and then transmits an instruction to generate a voice for the character to the voice generation means. The voice generation means generates a voice based on the specified characteristics and returns the generated voice data to the server.

[0585] Step 5:

[0586] The server then sends the received character image and audio data to the device, which then displays the character image in the AR environment and plays the audio based on the received data.

[0587] Step 6:

[0588] At the same time, the server uses a text generation means to generate conversation content according to the scene. The conversation text data generated based on the scene and character settings is transmitted from the server to the terminal.

[0589] Step 7:

[0590] The terminal transmits the received conversation text to the voice generation means, which generates voice data for the character to speak, and plays back the generated voice data so that the character can speak.

[0591] Step 8:

[0592] The user listens to the character's speech through earphones and expresses their own opinions and thoughts. The device receives the user's speech in real time and converts it into text (transcription).

[0593] Step 9:

[0594] The device transmits the transcribed user speech data in real time to the server, which then uses a text generation means to generate an appropriate response based on the received data.

[0595] Step 10:

[0596] The server transmits the generated response text data to the terminal. The terminal transmits the response text data to the voice generation means, which generates voice data for the character to speak. The voice data is then played back so that it sounds like the character is speaking.

[0597] Step 11:

[0598] After the conversation ends, the server analyzes the entire conversation data, evaluates the user's comments and responses, and generates feedback including the evaluation results and advice.

[0599] Step 12:

[0600] The server transmits the generated feedback data to the terminal, which displays the feedback to the user.

[0601] Step 13:

[0602] The user can choose to end the session and decide whether to save the conversation data and feedback. If the user chooses to save, the device sends the session data to the server, which then saves the data.

[0603] Example 1

[0604] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0605] Conventional communication skills development systems have had the problem of making it difficult for users to practice in an environment that closely resembles a real-life communication situation. In particular, the lack of real-time feedback makes it difficult for users to immediately evaluate the accuracy and appropriateness of their own speech, hindering efficient skill improvement.

[0606] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0607] In this invention, the server includes means for allowing a user to select a predetermined scene, means for generating a character image matching the predetermined scene using an image generation means, means for generating a voice based on the character image using a voice generation means, means for generating a conversation content corresponding to the scene using a text generation means, means for transcribing the user's utterance in real time, means for generating a response based on the generated transcription, means for displaying the generated character image and voice in an AR environment, means for generating and playing a response in real time in response to the user's utterance, and means for analyzing the generated conversation content and the user's utterance and providing appropriate feedback. This allows the user to efficiently train in an environment similar to an actual communication situation and instantly evaluate and improve the accuracy and appropriateness of their utterance.

[0608] "User" means an individual or organization that uses the system to improve their communication skills.

[0609] A "scene" refers to a specific situation or occasion selected by the user (e.g., a university club meeting, a company meeting, etc.).

[0610] "Image generation means" refers to a technology or device that enables a computer to generate corresponding images based on specified scenes or conditions, including, for example, deep learning models.

[0611] "Character image" refers to an image of a virtual character that is appropriate for the scene.

[0612] "Speech generation means" refers to technology or devices that synthesize speech based on text or digital data, including, for example, a speech synthesis engine.

[0613] "Text generation means" refers to a technology or device that generates natural language sentences based on input data, including, for example, a natural language processing model.

[0614] "Real-time transcription means" refers to technology or devices that convert user speech into text in real time, including, for example, speech recognition APIs.

[0615] The "means for generating a response" refers to a technology or device for generating an appropriate response text in response to a user's utterance.

[0616] An "AR environment" refers to a technological environment that displays virtual objects and information superimposed on the real environment.

[0617] "Means for providing feedback" refers to a technology or device that analyzes the content of a conversation that a user has had and provides advice or evaluation based on the results.

[0618] The verbalization ability development system of the present invention is designed to efficiently improve a user's communication ability and is equipped with multiple generation means and a real-time feedback function. This system is composed of an image generation means, a voice generation means, a text generation means, and a real-time transcription means, which work together to provide the user with an environment that is similar to an actual communication situation.

[0619] 1. Launching the application and selecting a scene

[0620] A user starts a dedicated application using a device such as a smartphone. On the initial screen, the user selects a specific scene (e.g., a university club meeting, a company meeting, etc.). The device then sends this scene information to the server.

[0621] 2. Character image and voice generation

[0622] The device sends a request to the server to generate a character image. The server uses an image generation means (e.g., a deep learning-based model) to generate a character image suitable for the specified scene and sends the data to the device. Next, the device sends a request to the server to generate audio based on the character image. The server uses an audio generation means (e.g., a speech synthesis engine) to generate audio corresponding to the specified character and sends this audio data to the device. The device combines these pieces of data and displays them in the user's AR environment.

[0623] 3. Conversation scenario generation

[0624] The device uses a text generation means to send a request to the server to generate conversation content appropriate for the scene. The server uses a text generation model (e.g., GPT-4) to generate conversation text appropriate for the scene and sends this text data to the device. The device then converts the received text data into audio data and plays it back as the character's speech.

[0625] 4. User utterances and response generation

[0626] The user listens to the character's speech through earphones and expresses their own thoughts and opinions. The device transcribes the user's speech in real time using the Google Speech-to-Text API and sends this text data to the server. The server uses a text generation means to generate an appropriate response text to the user's speech and sends this data to the device. The device converts the received response text into audio data and plays it back as if the character were speaking the response.

[0627] 5. Providing Feedback

[0628] The server analyzes the data from the entire conversation and evaluates the user's comments and responses. Based on the analysis results, the server uses a text generation means to generate feedback including the evaluation results and advice, and sends this data to the terminal. The terminal displays the received feedback to the user. The user can then identify areas for improvement based on the feedback and use them in the next conversation.

[0629] Specific examples

[0630] For example, if the user selects the scene "University Club Meeting," the following steps are taken:

[0631] 1. The user launches the application and selects the "University Club Meeting" scene.

[0632] 2. The device sends the scene information to the server and makes a request to generate an appropriate character image.

[0633] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[0634] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[0635] 5. The device displays the received character image and audio in the AR environment.

[0636] 6. The user begins a conversation with the character, and the device transcribes what is said and sends it to the server.

[0637] 7. The server generates an appropriate response and sends it to the terminal.

[0638] 8. The device plays the character's response as if it were speaking it.

[0639] 9. After the conversation is over, the server generates feedback and sends it to the device.

[0640] 10. The device displays feedback to the user.

[0641] In this way, the system allows users to train in an environment that is close to a real communication situation, and can efficiently improve their verbalization ability. In this system, the following are examples of prompt sentences when using a generative AI model:

[0642] ---

[0643] Example prompt sentence:

[0644] "Imagine a scene at a university club meeting where an idea for a new project is proposed, and generate how the characters would react."

[0645] ---

[0646] This allows users to hone their practical communication skills through conversations tailored to specific situations.

[0647] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0648] Step 1: Launch the application and select a scene

[0649] 1. The user taps the dedicated application on their smartphone to launch the app.

[0650] Input: Tap the app icon.

[0651] Output: The initial screen of the application is displayed.

[0652] 2. The terminal displays the initial screen and waits for user operation.

[0653] What it does: Draws the UI components on the initial screen.

[0654] 3. On the initial screen that appears, the user selects a scene such as a "university club meeting" or a "company meeting."

[0655] Input: Tap to select a scene.

[0656] Output: The information of the selected scene is saved to the device.

[0657] Step 2: Generate character images and sounds

[0658] 1. The device sends information about the scene selected by the user to the server and makes a request to generate a character image.

[0659] Input: Scene information.

[0660] Output: Request sent.

[0661] What it does: Creates and sends an HTTP request.

[0662] 2. The server receives the request and uses the image generation means to generate a character image suitable for the specified scene.

[0663] Input: A request for scene information.

[0664] Output: Character image data.

[0665] Action: Running an image generation model.

[0666] 3. The server sends the generated character image data to the terminal.

[0667] Input: Character image data.

[0668] Output: Sending image data.

[0669] What it does: Generates and sends an HTTP response.

[0670] 4. The device receives the character image and stores it in memory.

[0671] Input: Character image data.

[0672] Output: Saving image data.

[0673] Action: Processing data storage.

[0674] 5. The device sends a request to the server to generate audio based on the character image.

[0675] Input: Character image information.

[0676] Output: Request sent.

[0677] What it does: Creates and sends an HTTP request.

[0678] 6. The server uses the voice generation means to generate a voice corresponding to the specified character.

[0679] Input: Request for character information.

[0680] Output: Character voice data.

[0681] What it does: Runs the speech synthesis engine.

[0682] 7. The server sends the generated voice data to the terminal.

[0683] Input: Character voice data.

[0684] Output: Sending audio data.

[0685] What it does: Generates and sends an HTTP response.

[0686] 8. The device combines the character image and audio data and prepares to display them in the user's AR environment.

[0687] Input: Character image and audio data.

[0688] Output: AR display preparation.

[0689] What it does: Synchronize and prepare data.

[0690] Step 3: Generate conversation scenarios

[0691] 1. The device sends a request to the server to generate conversation content appropriate for the scene.

[0692] Input: Scene information.

[0693] Output: Request sent.

[0694] What it does: Creates and sends an HTTP request.

[0695] 2. The server uses a text generation means to generate conversation content appropriate for the scene.

[0696] Input: A request for scene information.

[0697] Output: Text data of the conversation scenario.

[0698] What it does: Run a text generation model.

[0699] 3. The server sends the text data of the generated conversation scenario to the terminal.

[0700] Input: Text data of a conversation scenario.

[0701] Output: Sending text data.

[0702] What it does: Generates and sends an HTTP response.

[0703] 4. The device converts the received text data into audio data and plays it back as the character's speech.

[0704] Input: Text data of a conversation scenario.

[0705] Output: Generate audio data.

[0706] What it does: Performs text-to-speech (TTS).

[0707] Step 4: User utterances and response generation

[0708] 1. The device plays the character's voice.

[0709] Input: Audio data.

[0710] Output: Audio playback.

[0711] Action: Performs audio playback function.

[0712] 2. The user listens to the character's speech and expresses their own thoughts and opinions.

[0713] Input: Character speech.

[0714] Output: User utterance.

[0715] Action: User's auditory comprehension and speech.

[0716] 3. The device collects the user's speech and transcribes it in real time.

[0717] Input: User speech.

[0718] Output: Text data.

[0719] What it does: Transcription using the Google Speech-to-Text API.

[0720] 4. The device sends the transcribed text data to the server and makes a request to generate a response.

[0721] Input: Speech text data.

[0722] Output: Request sent.

[0723] What it does: Creates and sends an HTTP request.

[0724] 5. The server uses a text generation means to generate an appropriate response based on the user's utterance.

[0725] Input: User utterance text data.

[0726] Output: Response text data.

[0727] What it does: Run a text generation model.

[0728] 6. The server sends the response text data to the terminal.

[0729] Input: Response text data.

[0730] Output: Sending text data.

[0731] What it does: Generates and sends an HTTP response.

[0732] 7. The device converts the received response text into audio data and plays it back as if the character were speaking the response.

[0733] Input: Response text data.

[0734] Output: Generation and playback of audio data.

[0735] What it does: Performs text-to-speech (TTS) and plays audio.

[0736] Step 5: Provide feedback

[0737] 1. The server analyzes the data from the entire conversation and evaluates the user's comments and responses.

[0738] Input: Utterance and response data.

[0739] Output: Evaluation results.

[0740] Action: Run the data analysis model.

[0741] 2. The server generates feedback based on the evaluation results using a text generation means.

[0742] Input: Analysis results.

[0743] Output: Feedback text data.

[0744] What it does: Run a text generation model.

[0745] 3. The server sends the feedback data to the terminal.

[0746] Input: Feedback text data.

[0747] Output: Sending text data.

[0748] What it does: Generates and sends an HTTP response.

[0749] 4. The device displays the received feedback to the user.

[0750] Input: Feedback text data.

[0751] Output: Display feedback.

[0752] Action: Drawing a UI component.

[0753] (Application example 1)

[0754] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0755] There is a need to provide a training environment in brick-and-mortar stores that allows store employees to efficiently improve their customer service skills. In particular, it is important to improve conversation skills in specific situations, such as explaining new products or handling complaints. However, because training that replicates real-world situations is time-consuming and costly, there is a need to develop an easily accessible simulation tool.

[0756] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0757] In this invention, the server includes means for allowing a user to select a predetermined scene, means for generating a character image that matches the predetermined scene using an image generation means, means for generating a voice based on the character image using a voice generation means, means for generating a conversation content corresponding to the scene using a text generation means, means for transcribing utterances from the user in real time, means for generating a response based on the generated transcription, and means for generating a conversation scenario corresponding to a specific situation in customer service work. This provides a simulation environment that is close to a real customer service situation, enabling store clerks to efficiently improve their customer service skills.

[0758] "Means for allowing the user to select a specific scene" refers to a function that provides an interface for the user to use the application to select a specific scene or situation (e.g., explaining a new product, handling a complaint, etc.).

[0759] "Image generation means" refers to a technology for generating character images that match a specific scene, and is a means for generating high-quality images using machine learning and deep learning.

[0760] The "voice generation means" is a technology for generating appropriate voice based on the generated character image, and is a means for converting text data into voice data.

[0761] The "text generation means" is a technology for generating conversation content appropriate to the scene, and is a means for generating appropriate text data using natural language processing technology.

[0762] "Means for transcribing user speech in real time" refers to a technology for instantly converting what a user says into text data, and is a means that uses voice recognition technology.

[0763] The "means for generating a response based on the generated transcription" is a technology for receiving a user's speech as text data and generating an appropriate response in response to it.

[0764] "Means for generating conversation scenarios according to specific situations in customer service work" refers to technology for generating scenarios in specific situations related to customer service work, and is a means for automatically generating conversation content according to user selections.

[0765] An "augmented reality environment" is a technology that displays virtual information overlaid on the real world, and is an environment that integrates the real environment with virtual information using multimedia content such as 3D models, videos, and audio.

[0766] This invention provides a system for efficiently training in customer service work. The system has a function that allows a user to select a predetermined scene and experience a virtual customer service situation based on that scene. The following describes in detail how this system can be realized.

[0767] Program Structure

[0768] The system consists of the following main components:

[0769] 1. A means for allowing the user to select a specific scene: A user interface is provided that allows the user to select a scene, such as explaining a new product or handling a complaint.

[0770] 2. Image generation method: Use a deep learning model to generate character images suitable for a given scene. For example, we use TensorFlow.

[0771] 3. Voice generation method: Google Cloud Text-to-Speech is used to generate voice based on the generated character image.

[0772] 4. Text generation: Use natural language processing models such as GPT-3 to generate conversational content appropriate to the situation.

[0773] 5. A way to transcribe user speech in real time: Use Google Cloud Speech-to-Text.

[0774] 6. Generating responses based on the generated transcription: The GPT-3 model is used to generate appropriate responses to the user's utterances.

[0775] 7. A means of generating conversation scenarios for specific situations in customer service: Automatically create conversation scenarios for specific situations based on user selections.

[0776] Hardware and Software Configuration

[0777] Hardware: Smartphone (iOS or Android), earphones

[0778] software:

[0779] TensorFlow: Image Generation

[0780] Google Cloud Text-to-Speech: Audio Generation

[0781] Google Cloud Speech-to-Text: Real-time transcription

[0782] GPT-3: Text and response generation

[0783] Specific examples

[0784] Example: New product description

[0785] The user launches the application and selects the "Explanation of a new product" scene. The application sends the selection information to the server and requests the generation of an appropriate character image and sound. The server generates a character image and sound using the image generation means and sound generation means, and sends them to the smartphone. The smartphone displays the received character image and sound in the augmented reality environment, and the user begins a virtual customer service interaction with the character.

[0786] When a user says, "Today I'll talk about our new product," the smartphone transcribes the speech in real time and sends it to the server. The server then generates an appropriate response, such as repeating what the user said or asking follow-up questions, and sends it to the smartphone. The character then replies, "Can you tell me about the features of our new product?" and the conversation continues.

[0787] Prompt Sentence Examples

[0788] Below are some example inputs for a generative AI model:

[0789] Please generate a conversation scenario with questions and answers from customers when explaining a new product. The specific product name is Smartwatch Z. Please create the scenario by including points that customers are likely to have questions about.

[0790] In this way, the present invention provides a system for efficiently conducting customer service training in a real store, allowing users to train in a realistic simulation environment.

[0791] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0792] Step 1:

[0793] The user selects a given scene

[0794] The user launches the customer service trainer app on their smartphone and selects a specific scene, such as "explaining a new product" or "handling a complaint," from the scene selection screen. The smartphone then sends the information about the selected scene to the server.

[0795] Input: User scene selection

[0796] Output: Send scene selection information to the server

[0797] Step 2:

[0798] Character image generation

[0799] Based on the received scene selection information, the server uses a deep learning model (e.g., TensorFlow) to generate a character image suitable for the specified scene. The generated character image is then sent from the server to the smartphone.

[0800] Input: Scene selection information

[0801] Output: Character image

[0802] Step 3:

[0803] Audio generation

[0804] The server uses Google Cloud Text-to-Speech to generate voice for the character based on the generated character image, and the generated voice is sent from the server to the smartphone.

[0805] Input: Character image

[0806] Output: Character voice

[0807] Step 4:

[0808] Conversation scenario generation using text generation

[0809] The server uses GPT-3 to generate conversation content based on the scene selected by the user. The conversation scenario is sent as text data from the server to the smartphone, which then converts it into audio data and plays it back.

[0810] Input: Scene selection information

[0811] Output: Conversation scenario (text data)

[0812] Step 5:

[0813] Transcription of user utterances

[0814] Users listen to the character's speech through headphones and express their thoughts and opinions. The smartphone then transcribes the speech in real time using Google Cloud Speech-to-Text and sends the data to a server.

[0815] Input: User utterance

[0816] Output: Transcription data

[0817] Step 6:

[0818] Generating a response

[0819] The server analyzes the text data using the GPT-3 model and generates an appropriate response, which is then sent to the smartphone, where it is converted into audio and played back.

[0820] Input: Transcription data

[0821] Output: Response text data

[0822] Step 7:

[0823] Providing feedback

[0824] The server analyzes the entire conversation data, evaluates the user's comments and responses, and generates appropriate feedback. The feedback is sent as text data from the server to the smartphone and displayed to the user.

[0825] Input: Data for the entire conversation

[0826] Output: Feedback data

[0827] In this way, the system allows users to efficiently improve their customer service skills in a realistic simulation environment.

[0828] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0829] The verbalization ability development system of the present invention is equipped with multiple generation means and a real-time feedback function to efficiently improve the user's communication ability. This system is composed of an image generation means, a voice generation means, a text generation means, an emotion engine, and a real-time transcription means, which work together to provide the user with an environment that is similar to an actual communication situation.

[0830] Overview of the program

[0831] 1. Launch the application and select a scene

[0832] User: Launches the dedicated application on a smartphone and selects a specific scene (e.g., a university club meeting, a company meeting, etc.).

[0833] Terminal: The initial application screen is displayed and the user's selection is accepted.

[0834] 2. Character image and voice generation

[0835] Terminal: Sends information about the scene selected by the user to the server and makes a request to generate a character image.

[0836] Server: Uses the image generation means to generate a character image suitable for the specified scene and sends the image data to the terminal.

[0837] Terminal: Next, the terminal uses the voice generating means to send a request to the server to generate a voice corresponding to the character image.

[0838] Server: Using the voice generation means, generates a voice corresponding to the specified character and sends the voice data to the terminal.

[0839] Device: Combines the received character image and audio and displays them in the user's AR environment.

[0840] 3. Conversation scenario generation

[0841] Server: Generates conversation content using a text generation means appropriate for the scene, and sends the text data to the terminal.

[0842] Terminal: The received text data is converted into voice data using a voice generation means and played back as the character's speech.

[0843] 4. User utterances and response generation

[0844] User: Listens to the character's speech through earphones and expresses his or her own opinions and thoughts. The device receives the user's speech in real time and converts it into text (transcription).

[0845] Device: The transcribed user speech data is sent to the server in real time. The server then analyzes the user's emotions using the received data and the emotion engine.

[0846] Server: Uses a text generation means to generate an appropriate response text based on the user's utterance and emotions, and sends the data to the terminal.

[0847] Terminal: The received response text is converted into audio data using a voice generation means, and played back as if spoken by a character.

[0848] 5. Providing feedback and using emotions

[0849] Server: Analyzes the overall conversation data and the emotional data obtained from the emotion engine, and evaluates the user's comments and responses.

[0850] Server: Generates feedback including evaluation results and advice that take emotions into account, and sends this data to the device.

[0851] Terminal: Display received feedback to the user.

[0852] Specific examples

[0853] Example 1: University club meeting

[0854] 1. The user launches the application and selects the "University Club Meeting" scene.

[0855] 2. The device sends the scene information to the server and makes a request to generate an appropriate character image.

[0856] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[0857] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[0858] 5. The device displays the received character image and audio in the AR environment.

[0859] 6. The user begins a conversation with the character, and the device transcribes what is said and sends it to the server.

[0860] 7. The server generates a response based on the user's speech and the emotion engine and sends it to the device.

[0861] 8. The device plays the character's response as if it were speaking it.

[0862] 9. After the conversation ends, the server generates a rating based on the emotional data along with feedback and sends it to the device.

[0863] 10. The device displays feedback to the user.

[0864] In this way, the system allows users to train in an environment that is similar to a real communication situation, and can efficiently improve their verbalization and emotion recognition abilities.

[0865] The processing flow will be explained below.

[0866] Step 1:

[0867] The user launches the dedicated application on their smartphone. The initial screen of the application is displayed, and they select a specific scene (e.g., a university club meeting, a company meeting, etc.).

[0868] Step 2:

[0869] The terminal receives the user's selection and sends scene information based on the selection to the server as a request, the request including detailed information about the selected scene.

[0870] Step 3:

[0871] The server analyzes the received request and sends instructions to the image generation means to generate a character image suitable for the scene. The image generation means generates a character image based on the specified features and returns the generated image data to the server.

[0872] Step 4:

[0873] The server receives the generated character image data and then transmits an instruction to generate a voice for the character to the voice generation means. The voice generation means generates a voice based on the specified characteristics and returns the generated voice data to the server.

[0874] Step 5:

[0875] The server then sends the received character image and audio data to the device, which then displays the character image in the AR environment and plays the audio based on the received data.

[0876] Step 6:

[0877] At the same time, the server uses a text generation means to generate conversation content according to the scene. The conversation text data generated based on the scene and character settings is transmitted from the server to the terminal.

[0878] Step 7:

[0879] The terminal transmits the received conversation text to the voice generation means, which generates voice data for the character to speak, and plays back the generated voice data so that the character can speak.

[0880] Step 8:

[0881] The user listens to the character's speech through earphones and expresses their own opinions and thoughts. The device receives the user's speech in real time and converts the speech into text (transcription).

[0882] Step 9:

[0883] The device transmits the transcribed user speech data to the server in real time, and the server uses an emotion engine to analyze the user's emotions based on the received data and facial expressions.

[0884] Step 10:

[0885] The server uses the text generation means based on the analysis results to generate an appropriate response text according to the user's utterance and emotion. The generated response text is sent from the server to the terminal.

[0886] Step 11:

[0887] The terminal transmits the received response text to the voice generation means, which generates voice data for the character to speak, and plays back the generated voice data so that the character can speak it.

[0888] Step 12:

[0889] Once the conversation is over, the server analyzes the entire conversation data and the emotional data obtained from the emotion engine, evaluates the user's comments and responses, and generates feedback including the evaluation results and advice.

[0890] Step 13:

[0891] The server transmits the generated feedback data to the terminal, which displays the feedback to the user.

[0892] Step 14:

[0893] The user can choose to end the session and decide whether to save the conversation data and feedback. If the user chooses to save, the device sends the session data to the server, which then stores the data.

[0894] Example 2

[0895] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0896] Current systems for improving communication skills have the problem that it is difficult for users to receive training in an environment that closely resembles a real communication situation. Furthermore, there is a lack of systems that can analyze not only the content of a user's speech but also their emotions and provide appropriate feedback based on that. This makes it difficult to simultaneously improve both a user's language ability and their emotion recognition ability.

[0897] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0898] In this invention, the server includes means for a user to select a predetermined scene, means for generating a character image that matches the predetermined scene using an image generation means, means for generating a voice based on the character image using a voice generation means, means for generating a conversation content according to the scene using a text generation means, means for transcribing the user's speech in real time, means for generating a response based on the generated transcription, means for analyzing the user's speech using an emotion analysis engine and generating an appropriate response, and means for displaying the generated character image and voice in an augmented reality environment. This allows the user to train in an environment that resembles an actual communication situation, thereby simultaneously improving both language ability and emotion recognition ability.

[0899] A "user" is an individual or corporation that uses this system to achieve a specific purpose.

[0900] A "scene" is a specific situation or environment selected by the user, which is the subject of training or simulation.

[0901] "Image generation means" refers to technology or equipment for generating character images that match a specific scene.

[0902] "Character images" are image data of people or objects generated according to the scene.

[0903] "Sound generation means" refers to technology or devices that generate sounds based on character images.

[0904] "Text generation means" refers to a technology or device that generates conversation content appropriate for a specific scene.

[0905] "Transcription" is the process of converting a user's speech from audio to text in real time.

[0906] "Response generation means" refers to technology or devices that generate appropriate responses based on the generated transcription or other data.

[0907] An "emotion analysis engine" refers to technology or equipment for analyzing emotions from a user's speech or behavior.

[0908] "Augmented reality environment" refers to technology that displays virtual information overlaid on the real environment, and the environment that uses it.

[0909] This invention is a system for improving a user's communication skills, which provides training by selecting a specific scene and having a conversation with a character. This system is configured to include the following means.

[0910] First, a user launches a system-specific application using a device such as a smartphone or tablet. On the initial screen of the application, the user selects a desired scene from multiple scenes (e.g., a university club meeting, a company meeting, etc.).

[0911] The selected scene information is transmitted from the terminal to the server. The server uses an image generation means to generate a character image that matches the selected scene. For example, a deep learning model can be used for this image generation. The generated character image data is transmitted from the server to the terminal.

[0912] Next, the terminal sends a request to the server to generate a voice for the character. The server uses a voice generation means to generate a voice based on the character image. A text-to-speech engine can be used for generating the voice. The generated voice data is sent to the terminal, and the terminal displays the character image and voice in combination.

[0913] A text generation means within the server is used to generate the conversation scenario. The text generation means uses, for example, a generative AI model that makes full use of natural language processing technology to generate conversation content that corresponds to the scene selected by the user. The generated text data is sent to the terminal, where it is converted into audio data via a text-to-speech engine and played back as the character's speech.

[0914] The user listens to the character's speech through earphones and a microphone and expresses their opinions and thoughts. The device records the user's speech in real time and converts it into text (transcription). The transcribed data is then sent to the server.

[0915] The server uses an emotion analysis engine to analyze the user's speech data and identify emotions. It then uses a text generation means to generate an appropriate response based on the user's speech content and emotions. This response data is sent to the terminal, converted into voice data, and played as the character's speech.

[0916] As a concrete example, if you select the "University Club Meeting" scene, the system will behave as follows:

[0917] 1. The user launches the application and selects the "University Club Meeting" scene.

[0918] 2. The device sends the selected scene ID to the server.

[0919] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[0920] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[0921] 5. The device displays the character image and sound in the AR environment.

[0922] 6. The user begins to interact with the character, and the device transcribes what is said and sends it to the server.

[0923] 7. The server analyzes the user's speech and emotions, generates an appropriate response, and sends it to the device.

[0924] 8. The device plays back the response as if it were being spoken by a character.

[0925] 9. After the conversation ends, the server generates feedback based on the emotion data and sends it to the device.

[0926] 10. The device displays feedback to the user.

[0927] An example prompt is:

[0928] You have selected a university club meeting scene. The server and device interact, generating character images and voices. The following conversation unfolds between you and the character:

[0929] "Today I would like to discuss a new project. I'd like to hear your thoughts."

[0930] How would you respond to this statement?

[0931] In this way, users can receive training in an environment that is close to a real communication situation, improving both their language skills and their emotion recognition abilities.

[0932] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0933] Step 1:

[0934] The user launches a dedicated application on the device and selects a specific scene. At this time, scene information (e.g., scene ID) is provided as input. The device accepts the user's selection and sends the scene information to the server. The output at this stage is that the scene information is sent to the server.

[0935] Step 2:

[0936] The device sends scene information to the server and requests the server to generate a character image. The server receives the scene information as input and generates a character image using an image generation means. Specifically, it uses an AI model to design a character image that fits the scene and generates its data. This image data is generated as output and sent from the server to the device.

[0937] Step 3:

[0938] The terminal receives the character image sent from the server and prepares to display it. It then sends a request to the server again to generate voice. This request includes information about the character image, which is passed to the server as input. The server uses a voice generation means to generate voice for the character. At this time, it uses a text-to-speech engine to convert the specified text into voice data. The generated voice data is sent from the server to the terminal.

[0939] Step 4:

[0940] The server uses a text generation means to generate dialogue content corresponding to the scene selected by the user. Scene information and prompt sentences are used as input, and natural language processing technology is used to generate appropriate dialogue text. This dialogue text is generated as output and sent to the terminal. The terminal converts this text into audio data and plays it back as if the character were speaking.

[0941] Step 5:

[0942] The user listens to the character's speech through headphones and expresses their own opinions and thoughts through a microphone. The user's speech is input, and the device records and transcribes it in real time. This transcription data is output and sent to the server.

[0943] Step 6:

[0944] The server uses a sentiment analysis engine to analyze the user's speech data. It receives the transcription data as input and analyzes it using a machine learning model to identify the user's emotions. The analysis results are output and used as input data for generating an appropriate response text. It uses a text generation means to generate a response text based on the user's speech content and emotions. The generated response text is sent to the device.

[0945] Step 7:

[0946] The device converts the received response text into audio data and plays it as if the character were speaking.The device converts the text into audio using a voice generation means and displays it as if the character is speaking in the AR environment.

[0947] Step 8:

[0948] After the conversation ends, the server analyzes all conversation content and emotional data and generates feedback for the user. The emotional data and spoken content are used as inputs for this analysis. The generated feedback information is output and sent to the device. The device displays the feedback to the user, allowing the user to improve themselves based on the results.

[0949] (Application example 2)

[0950] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0951] There is a need to efficiently and effectively improve the customer service skills of employees in brick-and-mortar stores. However, traditional training methods have difficulty providing an environment similar to actual customer interactions, limiting their effectiveness in improving communication skills. Furthermore, because training feedback is not provided in real time, it is difficult for employees to quickly understand and correct areas for improvement. Another issue is the lack of a system for analyzing employees' emotions and attitudes and providing appropriate advice.

[0952] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for allowing a user to select a predetermined scene, means for generating a person image matching the predetermined scene using an image generation means, means for generating a voice based on the person image using a voice generation means, means for generating a dialogue content appropriate to the scene using a text generation means, means for transcribing user utterances in real time, means for generating a response based on the generated transcription, emotion analysis means for analyzing the transcribed user utterances and emotions, means for displaying the generated response and analysis results in a substitute real environment, and means for providing feedback based on the response. This makes it possible to conduct training while receiving real-time feedback in an environment that resembles an actual customer service situation, thereby effectively improving employees' communication skills and customer service skills.

[0953] The "means for allowing the user to select a predetermined scene" refers to a means for providing an interface that allows the user to select a specific scenario or situation in advance.

[0954] "Image generation means" refers to an algorithm or program for automatically generating a person image that matches a scene selected by a user.

[0955] The "audio generating means" refers to a system or program for generating appropriate audio data based on the generated person image.

[0956] "Text generation means" refers to software or algorithms that automatically generate dialogue content appropriate for a specific situation.

[0957] "Means for transcribing user utterances in real time" refers to technology or devices that convert user speech into text in real time.

[0958] "Means for generating a response based on the generated transcription" refers to a mechanism for analyzing the transcribed content of a user's speech and automatically generating an appropriate corresponding response.

[0959] "Emotion analysis means" refers to technology that analyzes a user's comments and the emotions they express at the time, and provides appropriate actions and feedback based on the results.

[0960] "Means for displaying in a virtual reality environment" refers to a system or device for displaying the generated human image and audio in the virtual or alternate reality space experienced by the user.

[0961] "Means for providing feedback" refers to a mechanism that provides information for improvement and evaluation in real time based on the results of users' actions and comments.

[0962] This invention is a system for improving customer service skills in brick-and-mortar stores. The system aims to provide training that is similar to actual customer service situations by having users interact with customers in a virtual environment.

[0963] System Overview

[0964] The system includes the following main components:

[0965] 1. A means by which the user can select a given scene:

[0966] It provides an interface that allows users (store employees) to select a scene (e.g., introducing a new product, working at the cash register, etc.).

[0967] For example, by selecting a menu on the smart glasses display, the user can easily select the appropriate scene.

[0968] 2. Image generation means:

[0969] An algorithm or program for generating a person image that matches a selected scene.

[0970] Here, we use technologies such as OpenAI's DALL-E as image generation technology.

[0971] 3. Sound Generation Means:

[0972] A technology that generates appropriate voice data based on the generated person image.

[0973] For example, we use Azure Cognitive Services to generate a virtual customer voice.

[0974] 4. Text generation methods:

[0975] A technology that automatically generates dialogue content based on the selected situation.

[0976] Here, we use a text generation model (e.g., GPT-3) to generate scenario-based questions and answers.

[0977] 5. Means of transcribing user comments in real time:

[0978] A technology that converts user speech into text in real time.

[0979] It uses Google Speech-to-Text technology to instantly transcribe what users say.

[0980] 6. How to generate a response based on the generated transcript:

[0981] A system that automatically generates appropriate responses based on transcription data.

[0982] Again, a text generation model (e.g., GPT-3) is used to generate responses.

[0983] 7. Emotion analysis means:

[0984] A technology that analyzes a user's comments and their emotions at the time, and provides appropriate actions and feedback based on the results.

[0985] IBM Watson is used as the sentiment analysis engine.

[0986] 8. Means of display in virtual reality environment:

[0987] A technology that displays generated human images and audio in a virtual or augmented reality environment.

[0988] Smart glasses (e.g., Microsoft HoloLens) are used to provide visual and auditory information to users.

[0989] 9. Ways to provide feedback:

[0990] A system that provides real-time feedback based on user actions and comments.

[0991] Based on the analysis results, appropriate evaluations and improvement advice are provided to users in real time.

[0992] Specific examples

[0993] For example, in a scenario where a new product is being introduced, a user (employee) can wear smart glasses and undergo training. The user is asked questions such as, "What are the features of the new product?" and as the user responds, the system performs real-time transcription and sentiment analysis. As a result, feedback is provided on the accuracy of the response and the employee's attitude.

[0994] Prompt Sentence Examples

[0995] An example prompt for a new product introduction scene is as follows:

[0996] "Please tell me the features of this product."

[0997] Points to consider: Product features, benefits, user expectations, etc.

[0998] In this way, the system helps store employees to receive training in an environment that closely resembles an actual customer service situation, effectively improving their communication and customer service skills.

[0999] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1000] Step 1:

[1001] The user puts on the smart glasses and launches the application. Through the interface, the user selects a specific training scenario (e.g., a new product introduction) based on a real customer service scene. Scene information is selected as input, and the selected information is sent to the device as output.

[1002] Step 2:

[1003] The device sends the selected scene information to the server, which receives this information and uses an image generation model (e.g., OpenAI DALL-E) to generate a virtual customer persona that fits the scene. The scene information is given as input, and the generated persona is sent as output to the device.

[1004] Step 3:

[1005] The device sends a voice generation request to the server based on the received person image. The server uses a voice generation model (e.g., Azure Cognitive Services) to generate voice data that matches the scene in the voice of the virtual customer. The person image is given as input, and the voice data is sent to the device as output.

[1006] Step 4:

[1007] The device displays and plays the received image and audio data. The virtual customer is displayed on the smart glasses' display and the corresponding audio is played. By providing the user with visual and audio information, a realistic customer service scene is simulated.

[1008] Step 5:

[1009] The server uses a text generation model (e.g., GPT-3) to generate dialogue content based on the selected scene. Scene information is given as input, and appropriate questions and comments are sent as output to the device as text data.

[1010] Step 6:

[1011] The device converts the received text data into voice data using a speech generation model and plays it back as if the virtual customer were speaking it, for example, asking a question such as, "What are the features of this product?"

[1012] Step 7:

[1013] The user speaks their opinions and answers to the questions posed by the virtual customer through the microphone in the smart glasses, and the voice data is sent to the terminal as input.

[1014] Step 8:

[1015] The device uses speech recognition technology (e.g., Google Speech-to-Text) to transcribe and convert user voice data into text in real time. Voice data is given as input and text data is generated as output.

[1016] Step 9:

[1017] The device sends the generated text data to a server, which uses an emotion analysis engine (e.g., IBM Watson) to analyze the user's speech and the emotions expressed at the time. The text data is given as input, and the emotion analysis results are obtained as output.

[1018] Step 10:

[1019] The server uses a text generation model to generate an appropriate response text based on the user's utterances and emotions. The analysis results and the user's utterances are given as input, and the response text is generated as output.

[1020] Step 11:

[1021] The terminal converts the response text into speech data and plays it back as if the virtual customer were speaking the response, again using a speech model, and provides the generated speech data to the user.

[1022] Step 12:

[1023] The server generates feedback based on all conversation data and sentiment analysis results. Using a generative AI model, the server provides feedback to users, including areas for improvement and evaluations. Conversation data and analysis results are given as input, and feedback is generated as output.

[1024] Step 13:

[1025] The device then displays the generated feedback to the user, who is then shown on the smart glasses display, providing the user with specific points to improve and a visual assessment.

[1026] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1027] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1028] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1029] [Third embodiment]

[1030] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1031] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1033] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1034] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1035] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1037] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1038] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1040] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1041] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1042] The verbalization ability development system of the present invention is equipped with multiple generation means and a real-time feedback function to efficiently improve the user's communication ability. This system is composed of an image generation means, a voice generation means, a text generation means, and a real-time transcription means, which work together to provide the user with an environment that is similar to an actual communication situation.

[1043] Overview of the program

[1044] 1. Launch the application and select a scene

[1045] User: Launches the dedicated application on a smartphone and selects a specific scene (e.g., a university club meeting, a company meeting, etc.).

[1046] Terminal: The initial application screen is displayed and the user's selection is accepted.

[1047] 2. Character image and voice generation

[1048] Terminal: Sends information about the scene selected by the user to the server and makes a request to generate a character image.

[1049] Server: Uses the image generation means to generate a character image suitable for the specified scene and sends the image data to the terminal.

[1050] Terminal: Next, the terminal uses the voice generating means to send a request to the server to generate a voice corresponding to the character image.

[1051] Server: Using the voice generation means, generates a voice corresponding to the specified character and sends the voice data to the terminal.

[1052] Device: Combines the received character image and audio and displays them in the user's AR environment.

[1053] 3. Conversation scenario generation

[1054] Server: Generates conversation content using a text generation means appropriate for the scene, and sends the text data to the terminal.

[1055] Terminal: The received text data is converted into voice data using a voice generation means and played back as the character's speech.

[1056] 4. User utterances and response generation

[1057] User: Listen to the character's speech through earphones and express their own thoughts and opinions.

[1058] Terminal: Receives user speech and transcribes it in real time.

[1059] Terminal: Sends the text of the utterance to the server and requests that a response be generated.

[1060] Server: Uses the text generation means to generate an appropriate response text in response to the user's utterance and sends that data to the terminal.

[1061] Terminal: The received response text is converted into audio data using a voice generation means, and played back as if spoken by a character.

[1062] 5. Providing Feedback

[1063] Server: Analyzes the data from the entire conversation and evaluates the user's comments and responses.

[1064] Server: Uses a text generation means to generate feedback including evaluation results and advice, and sends the data to the terminal.

[1065] Terminal: Display received feedback to the user.

[1066] Specific examples

[1067] Example 1: University club meeting

[1068] 1. The user launches the application and selects the "University Club Meeting" scene.

[1069] 2. The device sends the scene information to the server and makes a request to generate an appropriate character image.

[1070] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[1071] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[1072] 5. The device displays the received character image and audio in the AR environment.

[1073] 6. The user begins a conversation with the character, and the device transcribes what is said and sends it to the server.

[1074] 7. The server generates an appropriate response and sends it to the terminal.

[1075] 8. The device plays the character's response as if it were speaking it.

[1076] 9. After the conversation is over, the server generates feedback and sends it to the device.

[1077] 10. The device displays feedback to the user.

[1078] In this way, the system allows users to practice in an environment that is similar to a real communication situation, and can efficiently improve their verbalization ability.

[1079] The processing flow will be explained below.

[1080] Step 1:

[1081] The user launches the dedicated application on their smartphone. The initial screen of the application is displayed, and they select a specific scene (e.g., a university club meeting, a company meeting, etc.).

[1082] Step 2:

[1083] The terminal receives the user's selection and sends scene information based on the selection to the server as a request, the request including detailed information about the selected scene.

[1084] Step 3:

[1085] The server analyzes the received request and sends instructions to the image generation means to generate a character image suitable for the scene. The image generation means generates a character image based on the specified features and returns the generated image data to the server.

[1086] Step 4:

[1087] The server receives the generated character image data and then transmits an instruction to generate a voice for the character to the voice generation means. The voice generation means generates a voice based on the specified characteristics and returns the generated voice data to the server.

[1088] Step 5:

[1089] The server then sends the received character image and audio data to the device, which then displays the character image in the AR environment and plays the audio based on the received data.

[1090] Step 6:

[1091] At the same time, the server uses a text generation means to generate conversation content according to the scene. The conversation text data generated based on the scene and character settings is transmitted from the server to the terminal.

[1092] Step 7:

[1093] The terminal transmits the received conversation text to the voice generation means, which generates voice data for the character to speak, and plays back the generated voice data so that the character can speak.

[1094] Step 8:

[1095] The user listens to the character's speech through earphones and expresses their own opinions and thoughts. The device receives the user's speech in real time and converts it into text (transcription).

[1096] Step 9:

[1097] The device transmits the transcribed user speech data in real time to the server, which then uses a text generation means to generate an appropriate response based on the received data.

[1098] Step 10:

[1099] The server transmits the generated response text data to the terminal. The terminal transmits the response text data to the voice generation means, which generates voice data for the character to speak. The voice data is then played back so that it sounds like the character is speaking.

[1100] Step 11:

[1101] After the conversation ends, the server analyzes the entire conversation data, evaluates the user's comments and responses, and generates feedback including the evaluation results and advice.

[1102] Step 12:

[1103] The server transmits the generated feedback data to the terminal, which displays the feedback to the user.

[1104] Step 13:

[1105] The user can choose to end the session and decide whether to save the conversation data and feedback. If the user chooses to save, the device sends the session data to the server, which then saves the data.

[1106] Example 1

[1107] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1108] Conventional communication skills development systems have had the problem of making it difficult for users to practice in an environment that closely resembles a real-life communication situation. In particular, the lack of real-time feedback makes it difficult for users to immediately evaluate the accuracy and appropriateness of their own speech, hindering efficient skill improvement.

[1109] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1110] In this invention, the server includes means for allowing a user to select a predetermined scene, means for generating a character image matching the predetermined scene using an image generation means, means for generating a voice based on the character image using a voice generation means, means for generating a conversation content corresponding to the scene using a text generation means, means for transcribing the user's utterance in real time, means for generating a response based on the generated transcription, means for displaying the generated character image and voice in an AR environment, means for generating and playing a response in real time in response to the user's utterance, and means for analyzing the generated conversation content and the user's utterance and providing appropriate feedback. This allows the user to efficiently train in an environment similar to an actual communication situation and instantly evaluate and improve the accuracy and appropriateness of their utterance.

[1111] "User" means an individual or organization that uses the system to improve their communication skills.

[1112] A "scene" refers to a specific situation or occasion selected by the user (e.g., a university club meeting, a company meeting, etc.).

[1113] "Image generation means" refers to a technology or device that enables a computer to generate corresponding images based on specified scenes or conditions, including, for example, deep learning models.

[1114] "Character image" refers to an image of a virtual character that is appropriate for the scene.

[1115] "Speech generation means" refers to technology or devices that synthesize speech based on text or digital data, including, for example, a speech synthesis engine.

[1116] "Text generation means" refers to a technology or device that generates natural language sentences based on input data, including, for example, a natural language processing model.

[1117] "Real-time transcription means" refers to technology or devices that convert user speech into text in real time, including, for example, speech recognition APIs.

[1118] The "means for generating a response" refers to a technology or device for generating an appropriate response text in response to a user's utterance.

[1119] An "AR environment" refers to a technological environment that displays virtual objects and information superimposed on the real environment.

[1120] "Means for providing feedback" refers to a technology or device that analyzes the content of a conversation that a user has had and provides advice or evaluation based on the results.

[1121] The verbalization ability development system of the present invention is designed to efficiently improve a user's communication ability and is equipped with multiple generation means and a real-time feedback function. This system is composed of an image generation means, a voice generation means, a text generation means, and a real-time transcription means, which work together to provide the user with an environment that is similar to an actual communication situation.

[1122] 1. Launching the application and selecting a scene

[1123] A user starts a dedicated application using a device such as a smartphone. On the initial screen, the user selects a specific scene (e.g., a university club meeting, a company meeting, etc.). The device then sends this scene information to the server.

[1124] 2. Character image and voice generation

[1125] The device sends a request to the server to generate a character image. The server uses an image generation means (e.g., a deep learning-based model) to generate a character image suitable for the specified scene and sends the data to the device. Next, the device sends a request to the server to generate audio based on the character image. The server uses an audio generation means (e.g., a speech synthesis engine) to generate audio corresponding to the specified character and sends this audio data to the device. The device combines these pieces of data and displays them in the user's AR environment.

[1126] 3. Conversation scenario generation

[1127] The device uses a text generation means to send a request to the server to generate conversation content appropriate for the scene. The server uses a text generation model (e.g., GPT-4) to generate conversation text appropriate for the scene and sends this text data to the device. The device then converts the received text data into audio data and plays it back as the character's speech.

[1128] 4. User utterances and response generation

[1129] The user listens to the character's speech through earphones and expresses their own thoughts and opinions. The device transcribes the user's speech in real time using the Google Speech-to-Text API and sends this text data to the server. The server uses a text generation means to generate an appropriate response text to the user's speech and sends this data to the device. The device converts the received response text into audio data and plays it back as if the character were speaking the response.

[1130] 5. Providing Feedback

[1131] The server analyzes the data from the entire conversation and evaluates the user's comments and responses. Based on the analysis results, the server uses a text generation means to generate feedback including the evaluation results and advice, and sends this data to the terminal. The terminal displays the received feedback to the user. The user can then identify areas for improvement based on the feedback and use them in the next conversation.

[1132] Specific examples

[1133] For example, if the user selects the scene "University Club Meeting," the following steps are taken:

[1134] 1. The user launches the application and selects the "University Club Meeting" scene.

[1135] 2. The device sends the scene information to the server and makes a request to generate an appropriate character image.

[1136] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[1137] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[1138] 5. The device displays the received character image and audio in the AR environment.

[1139] 6. The user begins a conversation with the character, and the device transcribes what is said and sends it to the server.

[1140] 7. The server generates an appropriate response and sends it to the terminal.

[1141] 8. The device plays the character's response as if it were speaking it.

[1142] 9. After the conversation is over, the server generates feedback and sends it to the device.

[1143] 10. The device displays feedback to the user.

[1144] In this way, the system allows users to train in an environment that is close to a real communication situation, and can efficiently improve their verbalization ability. In this system, the following are examples of prompt sentences when using a generative AI model:

[1145] ---

[1146] Example prompt sentence:

[1147] "Imagine a scene at a university club meeting where an idea for a new project is proposed, and generate how the characters would react."

[1148] ---

[1149] This allows users to hone their practical communication skills through conversations tailored to specific situations.

[1150] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1151] Step 1: Launch the application and select a scene

[1152] 1. The user taps the dedicated application on their smartphone to launch the app.

[1153] Input: Tap the app icon.

[1154] Output: The initial screen of the application is displayed.

[1155] 2. The terminal displays the initial screen and waits for user operation.

[1156] What it does: Draws the UI components on the initial screen.

[1157] 3. On the initial screen that appears, the user selects a scene such as a "university club meeting" or a "company meeting."

[1158] Input: Tap to select a scene.

[1159] Output: The information of the selected scene is saved to the device.

[1160] Step 2: Generate character images and sounds

[1161] 1. The device sends information about the scene selected by the user to the server and makes a request to generate a character image.

[1162] Input: Scene information.

[1163] Output: Request sent.

[1164] What it does: Creates and sends an HTTP request.

[1165] 2. The server receives the request and uses the image generation means to generate a character image suitable for the specified scene.

[1166] Input: A request for scene information.

[1167] Output: Character image data.

[1168] Action: Running an image generation model.

[1169] 3. The server sends the generated character image data to the terminal.

[1170] Input: Character image data.

[1171] Output: Sending image data.

[1172] What it does: Generates and sends an HTTP response.

[1173] 4. The device receives the character image and stores it in memory.

[1174] Input: Character image data.

[1175] Output: Saving image data.

[1176] Action: Processing data storage.

[1177] 5. The device sends a request to the server to generate audio based on the character image.

[1178] Input: Character image information.

[1179] Output: Request sent.

[1180] What it does: Creates and sends an HTTP request.

[1181] 6. The server uses the voice generation means to generate a voice corresponding to the specified character.

[1182] Input: Request for character information.

[1183] Output: Character voice data.

[1184] What it does: Runs the speech synthesis engine.

[1185] 7. The server sends the generated voice data to the terminal.

[1186] Input: Character voice data.

[1187] Output: Sending audio data.

[1188] What it does: Generates and sends an HTTP response.

[1189] 8. The device combines the character image and audio data and prepares to display them in the user's AR environment.

[1190] Input: Character image and audio data.

[1191] Output: AR display preparation.

[1192] What it does: Synchronize and prepare data.

[1193] Step 3: Generate conversation scenarios

[1194] 1. The device sends a request to the server to generate conversation content appropriate for the scene.

[1195] Input: Scene information.

[1196] Output: Request sent.

[1197] What it does: Creates and sends an HTTP request.

[1198] 2. The server uses a text generation means to generate conversation content appropriate for the scene.

[1199] Input: A request for scene information.

[1200] Output: Text data of the conversation scenario.

[1201] What it does: Run a text generation model.

[1202] 3. The server sends the text data of the generated conversation scenario to the terminal.

[1203] Input: Text data of a conversation scenario.

[1204] Output: Sending text data.

[1205] What it does: Generates and sends an HTTP response.

[1206] 4. The device converts the received text data into audio data and plays it back as the character's speech.

[1207] Input: Text data of a conversation scenario.

[1208] Output: Generate audio data.

[1209] What it does: Performs text-to-speech (TTS).

[1210] Step 4: User utterances and response generation

[1211] 1. The device plays the character's voice.

[1212] Input: Audio data.

[1213] Output: Audio playback.

[1214] Action: Performs audio playback function.

[1215] 2. The user listens to the character's speech and expresses their own thoughts and opinions.

[1216] Input: Character speech.

[1217] Output: User utterance.

[1218] Action: User's auditory comprehension and speech.

[1219] 3. The device collects the user's speech and transcribes it in real time.

[1220] Input: User speech.

[1221] Output: Text data.

[1222] What it does: Transcription using the Google Speech-to-Text API.

[1223] 4. The device sends the transcribed text data to the server and makes a request to generate a response.

[1224] Input: Speech text data.

[1225] Output: Request sent.

[1226] What it does: Creates and sends an HTTP request.

[1227] 5. The server uses a text generation means to generate an appropriate response based on the user's utterance.

[1228] Input: User utterance text data.

[1229] Output: Response text data.

[1230] What it does: Run a text generation model.

[1231] 6. The server sends the response text data to the terminal.

[1232] Input: Response text data.

[1233] Output: Sending text data.

[1234] What it does: Generates and sends an HTTP response.

[1235] 7. The device converts the received response text into audio data and plays it back as if the character were speaking the response.

[1236] Input: Response text data.

[1237] Output: Generation and playback of audio data.

[1238] What it does: Performs text-to-speech (TTS) and plays audio.

[1239] Step 5: Provide feedback

[1240] 1. The server analyzes the data from the entire conversation and evaluates the user's comments and responses.

[1241] Input: Utterance and response data.

[1242] Output: Evaluation results.

[1243] Action: Run the data analysis model.

[1244] 2. The server generates feedback based on the evaluation results using a text generation means.

[1245] Input: Analysis results.

[1246] Output: Feedback text data.

[1247] What it does: Run a text generation model.

[1248] 3. The server sends the feedback data to the terminal.

[1249] Input: Feedback text data.

[1250] Output: Sending text data.

[1251] What it does: Generates and sends an HTTP response.

[1252] 4. The device displays the received feedback to the user.

[1253] Input: Feedback text data.

[1254] Output: Display feedback.

[1255] Action: Drawing a UI component.

[1256] (Application example 1)

[1257] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1258] There is a need to provide a training environment in brick-and-mortar stores that allows store employees to efficiently improve their customer service skills. In particular, it is important to improve conversation skills in specific situations, such as explaining new products or handling complaints. However, because training that replicates real-world situations is time-consuming and costly, there is a need to develop an easily accessible simulation tool.

[1259] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1260] In this invention, the server includes means for allowing a user to select a predetermined scene, means for generating a character image that matches the predetermined scene using an image generation means, means for generating a voice based on the character image using a voice generation means, means for generating a conversation content corresponding to the scene using a text generation means, means for transcribing utterances from the user in real time, means for generating a response based on the generated transcription, and means for generating a conversation scenario corresponding to a specific situation in customer service work. This provides a simulation environment that is close to a real customer service situation, enabling store clerks to efficiently improve their customer service skills.

[1261] "Means for allowing the user to select a specific scene" refers to a function that provides an interface for the user to use the application to select a specific scene or situation (e.g., explaining a new product, handling a complaint, etc.).

[1262] "Image generation means" refers to a technology for generating character images that match a specific scene, and is a means for generating high-quality images using machine learning and deep learning.

[1263] The "voice generation means" is a technology for generating appropriate voice based on the generated character image, and is a means for converting text data into voice data.

[1264] The "text generation means" is a technology for generating conversation content appropriate to the scene, and is a means for generating appropriate text data using natural language processing technology.

[1265] "Means for transcribing user speech in real time" refers to a technology for instantly converting what a user says into text data, and is a means that uses voice recognition technology.

[1266] The "means for generating a response based on the generated transcription" is a technology for receiving a user's speech as text data and generating an appropriate response in response to it.

[1267] "Means for generating conversation scenarios according to specific situations in customer service work" refers to technology for generating scenarios in specific situations related to customer service work, and is a means for automatically generating conversation content according to user selections.

[1268] An "augmented reality environment" is a technology that displays virtual information overlaid on the real world, and is an environment that integrates the real environment with virtual information using multimedia content such as 3D models, videos, and audio.

[1269] This invention provides a system for efficiently training in customer service work. The system has a function that allows a user to select a predetermined scene and experience a virtual customer service situation based on that scene. The following describes in detail how this system can be realized.

[1270] Program Structure

[1271] The system consists of the following main components:

[1272] 1. A means for allowing the user to select a specific scene: A user interface is provided that allows the user to select a scene, such as explaining a new product or handling a complaint.

[1273] 2. Image generation method: Use a deep learning model to generate character images suitable for a given scene. For example, we use TensorFlow.

[1274] 3. Voice generation method: Google Cloud Text-to-Speech is used to generate voice based on the generated character image.

[1275] 4. Text generation: Use natural language processing models such as GPT-3 to generate conversational content appropriate to the situation.

[1276] 5. A way to transcribe user speech in real time: Use Google Cloud Speech-to-Text.

[1277] 6. Generating responses based on the generated transcription: The GPT-3 model is used to generate appropriate responses to the user's utterances.

[1278] 7. A means of generating conversation scenarios for specific situations in customer service: Automatically create conversation scenarios for specific situations based on user selections.

[1279] Hardware and Software Configuration

[1280] Hardware: Smartphone (iOS or Android), earphones

[1281] software:

[1282] TensorFlow: Image Generation

[1283] Google Cloud Text-to-Speech: Audio Generation

[1284] Google Cloud Speech-to-Text: Real-time transcription

[1285] GPT-3: Text and response generation

[1286] Specific examples

[1287] Example: New product description

[1288] The user launches the application and selects the "Explanation of a new product" scene. The application sends the selection information to the server and requests the generation of an appropriate character image and sound. The server generates a character image and sound using the image generation means and sound generation means, and sends them to the smartphone. The smartphone displays the received character image and sound in the augmented reality environment, and the user begins a virtual customer service interaction with the character.

[1289] When a user says, "Today I'll talk about our new product," the smartphone transcribes the speech in real time and sends it to the server. The server then generates an appropriate response, such as repeating what the user said or asking follow-up questions, and sends it to the smartphone. The character then replies, "Can you tell me about the features of our new product?" and the conversation continues.

[1290] Prompt Sentence Examples

[1291] Below are some example inputs for a generative AI model:

[1292] Please generate a conversation scenario with questions and answers from customers when explaining a new product. The specific product name is Smartwatch Z. Please create the scenario by including points that customers are likely to have questions about.

[1293] In this way, the present invention provides a system for efficiently conducting customer service training in a real store, allowing users to train in a realistic simulation environment.

[1294] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1295] Step 1:

[1296] The user selects a given scene

[1297] The user launches the customer service trainer app on their smartphone and selects a specific scene, such as "explaining a new product" or "handling a complaint," from the scene selection screen. The smartphone then sends the information about the selected scene to the server.

[1298] Input: User scene selection

[1299] Output: Send scene selection information to the server

[1300] Step 2:

[1301] Character image generation

[1302] Based on the received scene selection information, the server uses a deep learning model (e.g., TensorFlow) to generate a character image suitable for the specified scene. The generated character image is then sent from the server to the smartphone.

[1303] Input: Scene selection information

[1304] Output: Character image

[1305] Step 3:

[1306] Audio generation

[1307] The server uses Google Cloud Text-to-Speech to generate voice for the character based on the generated character image, and the generated voice is sent from the server to the smartphone.

[1308] Input: Character image

[1309] Output: Character voice

[1310] Step 4:

[1311] Conversation scenario generation using text generation

[1312] The server uses GPT-3 to generate conversation content based on the scene selected by the user. The conversation scenario is sent as text data from the server to the smartphone, which then converts it into audio data and plays it back.

[1313] Input: Scene selection information

[1314] Output: Conversation scenario (text data)

[1315] Step 5:

[1316] Transcription of user utterances

[1317] Users listen to the character's speech through headphones and express their thoughts and opinions. The smartphone then transcribes the speech in real time using Google Cloud Speech-to-Text and sends the data to a server.

[1318] Input: User utterance

[1319] Output: Transcription data

[1320] Step 6:

[1321] Generating a response

[1322] The server analyzes the text data using the GPT-3 model and generates an appropriate response, which is then sent to the smartphone, where it is converted into audio and played back.

[1323] Input: Transcription data

[1324] Output: Response text data

[1325] Step 7:

[1326] Providing feedback

[1327] The server analyzes the entire conversation data, evaluates the user's comments and responses, and generates appropriate feedback. The feedback is sent as text data from the server to the smartphone and displayed to the user.

[1328] Input: Data for the entire conversation

[1329] Output: Feedback data

[1330] In this way, the system allows users to efficiently improve their customer service skills in a realistic simulation environment.

[1331] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1332] The verbalization ability development system of the present invention is equipped with multiple generation means and a real-time feedback function to efficiently improve the user's communication ability. This system is composed of an image generation means, a voice generation means, a text generation means, an emotion engine, and a real-time transcription means, which work together to provide the user with an environment that is similar to an actual communication situation.

[1333] Overview of the program

[1334] 1. Launch the application and select a scene

[1335] User: Launches the dedicated application on a smartphone and selects a specific scene (e.g., a university club meeting, a company meeting, etc.).

[1336] Terminal: The initial application screen is displayed and the user's selection is accepted.

[1337] 2. Character image and voice generation

[1338] Terminal: Sends information about the scene selected by the user to the server and makes a request to generate a character image.

[1339] Server: Uses the image generation means to generate a character image suitable for the specified scene and sends the image data to the terminal.

[1340] Terminal: Next, the terminal uses the voice generating means to send a request to the server to generate a voice corresponding to the character image.

[1341] Server: Using the voice generation means, generates a voice corresponding to the specified character and sends the voice data to the terminal.

[1342] Device: Combines the received character image and audio and displays them in the user's AR environment.

[1343] 3. Conversation scenario generation

[1344] Server: Generates conversation content using a text generation means appropriate for the scene, and sends the text data to the terminal.

[1345] Terminal: The received text data is converted into voice data using a voice generation means and played back as the character's speech.

[1346] 4. User utterances and response generation

[1347] User: Listens to the character's speech through earphones and expresses his or her own opinions and thoughts. The device receives the user's speech in real time and converts it into text (transcription).

[1348] Device: The transcribed user speech data is sent to the server in real time. The server then analyzes the user's emotions using the received data and the emotion engine.

[1349] Server: Uses a text generation means to generate an appropriate response text based on the user's utterance and emotions, and sends the data to the terminal.

[1350] Terminal: The received response text is converted into audio data using a voice generation means, and played back as if spoken by a character.

[1351] 5. Providing feedback and using emotions

[1352] Server: Analyzes the overall conversation data and the emotional data obtained from the emotion engine, and evaluates the user's comments and responses.

[1353] Server: Generates feedback including evaluation results and advice that take emotions into account, and sends this data to the device.

[1354] Terminal: Display received feedback to the user.

[1355] Specific examples

[1356] Example 1: University club meeting

[1357] 1. The user launches the application and selects the "University Club Meeting" scene.

[1358] 2. The device sends the scene information to the server and makes a request to generate an appropriate character image.

[1359] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[1360] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[1361] 5. The device displays the received character image and audio in the AR environment.

[1362] 6. The user begins a conversation with the character, and the device transcribes what is said and sends it to the server.

[1363] 7. The server generates a response based on the user's speech and the emotion engine and sends it to the device.

[1364] 8. The device plays the character's response as if it were speaking it.

[1365] 9. After the conversation ends, the server generates a rating based on the emotional data along with feedback and sends it to the device.

[1366] 10. The device displays feedback to the user.

[1367] In this way, the system allows users to train in an environment that is similar to a real communication situation, and can efficiently improve their verbalization and emotion recognition abilities.

[1368] The processing flow will be explained below.

[1369] Step 1:

[1370] The user launches the dedicated application on their smartphone. The initial screen of the application is displayed, and they select a specific scene (e.g., a university club meeting, a company meeting, etc.).

[1371] Step 2:

[1372] The terminal receives the user's selection and sends scene information based on the selection to the server as a request, the request including detailed information about the selected scene.

[1373] Step 3:

[1374] The server analyzes the received request and sends instructions to the image generation means to generate a character image suitable for the scene. The image generation means generates a character image based on the specified features and returns the generated image data to the server.

[1375] Step 4:

[1376] The server receives the generated character image data and then transmits an instruction to generate a voice for the character to the voice generation means. The voice generation means generates a voice based on the specified characteristics and returns the generated voice data to the server.

[1377] Step 5:

[1378] The server then sends the received character image and audio data to the device, which then displays the character image in the AR environment and plays the audio based on the received data.

[1379] Step 6:

[1380] At the same time, the server uses a text generation means to generate conversation content according to the scene. The conversation text data generated based on the scene and character settings is transmitted from the server to the terminal.

[1381] Step 7:

[1382] The terminal transmits the received conversation text to the voice generation means, which generates voice data for the character to speak, and plays back the generated voice data so that the character can speak.

[1383] Step 8:

[1384] The user listens to the character's speech through earphones and expresses their own opinions and thoughts. The device receives the user's speech in real time and converts the speech into text (transcription).

[1385] Step 9:

[1386] The device transmits the transcribed user speech data to the server in real time, and the server uses an emotion engine to analyze the user's emotions based on the received data and facial expressions.

[1387] Step 10:

[1388] The server uses the text generation means based on the analysis results to generate an appropriate response text according to the user's utterance and emotion. The generated response text is sent from the server to the terminal.

[1389] Step 11:

[1390] The terminal transmits the received response text to the voice generation means, which generates voice data for the character to speak, and plays back the generated voice data so that the character can speak it.

[1391] Step 12:

[1392] Once the conversation is over, the server analyzes the entire conversation data and the emotional data obtained from the emotion engine, evaluates the user's comments and responses, and generates feedback including the evaluation results and advice.

[1393] Step 13:

[1394] The server transmits the generated feedback data to the terminal, which displays the feedback to the user.

[1395] Step 14:

[1396] The user can choose to end the session and decide whether to save the conversation data and feedback. If the user chooses to save, the device sends the session data to the server, which then stores the data.

[1397] Example 2

[1398] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1399] Current systems for improving communication skills have the problem that it is difficult for users to receive training in an environment that closely resembles a real communication situation. Furthermore, there is a lack of systems that can analyze not only the content of a user's speech but also their emotions and provide appropriate feedback based on that. This makes it difficult to simultaneously improve both a user's language ability and their emotion recognition ability.

[1400] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1401] In this invention, the server includes means for a user to select a predetermined scene, means for generating a character image that matches the predetermined scene using an image generation means, means for generating a voice based on the character image using a voice generation means, means for generating a conversation content according to the scene using a text generation means, means for transcribing the user's speech in real time, means for generating a response based on the generated transcription, means for analyzing the user's speech using an emotion analysis engine and generating an appropriate response, and means for displaying the generated character image and voice in an augmented reality environment. This allows the user to train in an environment that resembles an actual communication situation, thereby simultaneously improving both language ability and emotion recognition ability.

[1402] A "user" is an individual or corporation that uses this system to achieve a specific purpose.

[1403] A "scene" is a specific situation or environment selected by the user, which is the subject of training or simulation.

[1404] "Image generation means" refers to technology or equipment for generating character images that match a specific scene.

[1405] "Character images" are image data of people or objects generated according to the scene.

[1406] "Sound generation means" refers to technology or devices that generate sounds based on character images.

[1407] "Text generation means" refers to a technology or device that generates conversation content appropriate for a specific scene.

[1408] "Transcription" is the process of converting a user's speech from audio to text in real time.

[1409] "Response generation means" refers to technology or devices that generate appropriate responses based on the generated transcription or other data.

[1410] An "emotion analysis engine" refers to technology or equipment for analyzing emotions from a user's speech or behavior.

[1411] "Augmented reality environment" refers to technology that displays virtual information overlaid on the real environment, and the environment that uses it.

[1412] This invention is a system for improving a user's communication skills, which provides training by selecting a specific scene and having a conversation with a character. This system is configured to include the following means.

[1413] First, a user launches a system-specific application using a device such as a smartphone or tablet. On the initial screen of the application, the user selects a desired scene from multiple scenes (e.g., a university club meeting, a company meeting, etc.).

[1414] The selected scene information is transmitted from the terminal to the server. The server uses an image generation means to generate a character image that matches the selected scene. For example, a deep learning model can be used for this image generation. The generated character image data is transmitted from the server to the terminal.

[1415] Next, the terminal sends a request to the server to generate a voice for the character. The server uses a voice generation means to generate a voice based on the character image. A text-to-speech engine can be used for generating the voice. The generated voice data is sent to the terminal, and the terminal displays the character image and voice in combination.

[1416] A text generation means within the server is used to generate the conversation scenario. The text generation means uses, for example, a generative AI model that makes full use of natural language processing technology to generate conversation content that corresponds to the scene selected by the user. The generated text data is sent to the terminal, where it is converted into audio data via a text-to-speech engine and played back as the character's speech.

[1417] The user listens to the character's speech through earphones and a microphone and expresses their opinions and thoughts. The device records the user's speech in real time and converts it into text (transcription). The transcribed data is then sent to the server.

[1418] The server uses an emotion analysis engine to analyze the user's speech data and identify emotions. It then uses a text generation means to generate an appropriate response based on the user's speech content and emotions. This response data is sent to the terminal, converted into voice data, and played as the character's speech.

[1419] As a concrete example, if you select the "University Club Meeting" scene, the system will behave as follows:

[1420] 1. The user launches the application and selects the "University Club Meeting" scene.

[1421] 2. The device sends the selected scene ID to the server.

[1422] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[1423] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[1424] 5. The device displays the character image and sound in the AR environment.

[1425] 6. The user begins to interact with the character, and the device transcribes what is said and sends it to the server.

[1426] 7. The server analyzes the user's speech and emotions, generates an appropriate response, and sends it to the device.

[1427] 8. The device plays back the response as if it were being spoken by a character.

[1428] 9. After the conversation ends, the server generates feedback based on the emotion data and sends it to the device.

[1429] 10. The device displays feedback to the user.

[1430] An example prompt is:

[1431] You have selected a university club meeting scene. The server and device interact, generating character images and voices. The following conversation unfolds between you and the character:

[1432] "Today I would like to discuss a new project. I'd like to hear your thoughts."

[1433] How would you respond to this statement?

[1434] In this way, users can receive training in an environment that is close to a real communication situation, improving both their language skills and their emotion recognition abilities.

[1435] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1436] Step 1:

[1437] The user launches a dedicated application on the device and selects a specific scene. At this time, scene information (e.g., scene ID) is provided as input. The device accepts the user's selection and sends the scene information to the server. The output at this stage is that the scene information is sent to the server.

[1438] Step 2:

[1439] The device sends scene information to the server and requests the server to generate a character image. The server receives the scene information as input and generates a character image using an image generation means. Specifically, it uses an AI model to design a character image that fits the scene and generates its data. This image data is generated as output and sent from the server to the device.

[1440] Step 3:

[1441] The terminal receives the character image sent from the server and prepares to display it. It then sends a request to the server again to generate voice. This request includes information about the character image, which is passed to the server as input. The server uses a voice generation means to generate voice for the character. At this time, it uses a text-to-speech engine to convert the specified text into voice data. The generated voice data is sent from the server to the terminal.

[1442] Step 4:

[1443] The server uses a text generation means to generate dialogue content corresponding to the scene selected by the user. Scene information and prompt sentences are used as input, and natural language processing technology is used to generate appropriate dialogue text. This dialogue text is generated as output and sent to the terminal. The terminal converts this text into audio data and plays it back as if the character were speaking.

[1444] Step 5:

[1445] The user listens to the character's speech through headphones and expresses their own opinions and thoughts through a microphone. The user's speech is input, and the device records and transcribes it in real time. This transcription data is output and sent to the server.

[1446] Step 6:

[1447] The server uses a sentiment analysis engine to analyze the user's speech data. It receives the transcription data as input and analyzes it using a machine learning model to identify the user's emotions. The analysis results are output and used as input data for generating an appropriate response text. It uses a text generation means to generate a response text based on the user's speech content and emotions. The generated response text is sent to the device.

[1448] Step 7:

[1449] The device converts the received response text into audio data and plays it as if the character were speaking.The device converts the text into audio using a voice generation means and displays it as if the character is speaking in the AR environment.

[1450] Step 8:

[1451] After the conversation ends, the server analyzes all conversation content and emotional data and generates feedback for the user. The emotional data and spoken content are used as inputs for this analysis. The generated feedback information is output and sent to the device. The device displays the feedback to the user, allowing the user to improve themselves based on the results.

[1452] (Application example 2)

[1453] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1454] There is a need to efficiently and effectively improve the customer service skills of employees in brick-and-mortar stores. However, traditional training methods have difficulty providing an environment similar to actual customer interactions, limiting their effectiveness in improving communication skills. Furthermore, because training feedback is not provided in real time, it is difficult for employees to quickly understand and correct areas for improvement. Another issue is the lack of a system for analyzing employees' emotions and attitudes and providing appropriate advice.

[1455] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for allowing a user to select a predetermined scene, means for generating a person image matching the predetermined scene using an image generation means, means for generating a voice based on the person image using a voice generation means, means for generating a dialogue content appropriate to the scene using a text generation means, means for transcribing user utterances in real time, means for generating a response based on the generated transcription, emotion analysis means for analyzing the transcribed user utterances and emotions, means for displaying the generated response and analysis results in a substitute real environment, and means for providing feedback based on the response. This makes it possible to conduct training while receiving real-time feedback in an environment that resembles an actual customer service situation, thereby effectively improving employees' communication skills and customer service skills.

[1456] The "means for allowing the user to select a predetermined scene" refers to a means for providing an interface that allows the user to select a specific scenario or situation in advance.

[1457] "Image generation means" refers to an algorithm or program for automatically generating a person image that matches a scene selected by a user.

[1458] The "audio generating means" refers to a system or program for generating appropriate audio data based on the generated person image.

[1459] "Text generation means" refers to software or algorithms that automatically generate dialogue content appropriate for a specific situation.

[1460] "Means for transcribing user utterances in real time" refers to technology or devices that convert user speech into text in real time.

[1461] "Means for generating a response based on the generated transcription" refers to a mechanism for analyzing the transcribed content of a user's speech and automatically generating an appropriate corresponding response.

[1462] "Emotion analysis means" refers to technology that analyzes a user's comments and the emotions they express at the time, and provides appropriate actions and feedback based on the results.

[1463] "Means for displaying in a virtual reality environment" refers to a system or device for displaying the generated human image and audio in the virtual or alternate reality space experienced by the user.

[1464] "Means for providing feedback" refers to a mechanism that provides information for improvement and evaluation in real time based on the results of users' actions and comments.

[1465] This invention is a system for improving customer service skills in brick-and-mortar stores. The system aims to provide training that is similar to actual customer service situations by having users interact with customers in a virtual environment.

[1466] System Overview

[1467] The system includes the following main components:

[1468] 1. A means by which the user can select a given scene:

[1469] It provides an interface that allows users (store employees) to select a scene (e.g., introducing a new product, working at the cash register, etc.).

[1470] For example, by selecting a menu on the smart glasses display, the user can easily select the appropriate scene.

[1471] 2. Image generation means:

[1472] An algorithm or program for generating a person image that matches a selected scene.

[1473] Here, we use technologies such as OpenAI's DALL-E as image generation technology.

[1474] 3. Sound Generation Means:

[1475] A technology that generates appropriate voice data based on the generated person image.

[1476] For example, we use Azure Cognitive Services to generate a virtual customer voice.

[1477] 4. Text generation methods:

[1478] A technology that automatically generates dialogue content based on the selected situation.

[1479] Here, we use a text generation model (e.g., GPT-3) to generate scenario-based questions and answers.

[1480] 5. Means of transcribing user comments in real time:

[1481] A technology that converts user speech into text in real time.

[1482] It uses Google Speech-to-Text technology to instantly transcribe what users say.

[1483] 6. How to generate a response based on the generated transcript:

[1484] A system that automatically generates appropriate responses based on transcription data.

[1485] Again, a text generation model (e.g., GPT-3) is used to generate responses.

[1486] 7. Emotion analysis means:

[1487] A technology that analyzes a user's comments and their emotions at the time, and provides appropriate actions and feedback based on the results.

[1488] IBM Watson is used as the sentiment analysis engine.

[1489] 8. Means of display in virtual reality environment:

[1490] A technology that displays generated human images and audio in a virtual or augmented reality environment.

[1491] Smart glasses (e.g., Microsoft HoloLens) are used to provide visual and auditory information to users.

[1492] 9. Ways to provide feedback:

[1493] A system that provides real-time feedback based on user actions and comments.

[1494] Based on the analysis results, appropriate evaluations and improvement advice are provided to users in real time.

[1495] Specific examples

[1496] For example, in a scenario where a new product is being introduced, a user (employee) can wear smart glasses and undergo training. The user is asked questions such as, "What are the features of the new product?" and as the user responds, the system performs real-time transcription and sentiment analysis. As a result, feedback is provided on the accuracy of the response and the employee's attitude.

[1497] Prompt Sentence Examples

[1498] An example prompt for a new product introduction scene is as follows:

[1499] "Please tell me the features of this product."

[1500] Points to consider: Product features, benefits, user expectations, etc.

[1501] In this way, the system helps store employees to receive training in an environment that closely resembles an actual customer service situation, effectively improving their communication and customer service skills.

[1502] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1503] Step 1:

[1504] The user puts on the smart glasses and launches the application. Through the interface, the user selects a specific training scenario (e.g., a new product introduction) based on a real customer service scene. Scene information is selected as input, and the selected information is sent to the device as output.

[1505] Step 2:

[1506] The device sends the selected scene information to the server, which receives this information and uses an image generation model (e.g., OpenAI DALL-E) to generate a virtual customer persona that fits the scene. The scene information is given as input, and the generated persona is sent as output to the device.

[1507] Step 3:

[1508] The device sends a voice generation request to the server based on the received person image. The server uses a voice generation model (e.g., Azure Cognitive Services) to generate voice data that matches the scene in the voice of the virtual customer. The person image is given as input, and the voice data is sent to the device as output.

[1509] Step 4:

[1510] The device displays and plays the received image and audio data. The virtual customer is displayed on the smart glasses' display and the corresponding audio is played. By providing the user with visual and audio information, a realistic customer service scene is simulated.

[1511] Step 5:

[1512] The server uses a text generation model (e.g., GPT-3) to generate dialogue content based on the selected scene. Scene information is given as input, and appropriate questions and comments are sent as output to the device as text data.

[1513] Step 6:

[1514] The device converts the received text data into voice data using a speech generation model and plays it back as if the virtual customer were speaking it, for example, asking a question such as, "What are the features of this product?"

[1515] Step 7:

[1516] The user speaks their opinions and answers to the questions posed by the virtual customer through the microphone in the smart glasses, and the voice data is sent to the terminal as input.

[1517] Step 8:

[1518] The device uses speech recognition technology (e.g., Google Speech-to-Text) to transcribe and convert user voice data into text in real time. Voice data is given as input and text data is generated as output.

[1519] Step 9:

[1520] The device sends the generated text data to a server, which uses an emotion analysis engine (e.g., IBM Watson) to analyze the user's speech and the emotions expressed at the time. The text data is given as input, and the emotion analysis results are obtained as output.

[1521] Step 10:

[1522] The server uses a text generation model to generate an appropriate response text based on the user's utterances and emotions. The analysis results and the user's utterances are given as input, and the response text is generated as output.

[1523] Step 11:

[1524] The terminal converts the response text into speech data and plays it back as if the virtual customer were speaking the response, again using a speech model, and provides the generated speech data to the user.

[1525] Step 12:

[1526] The server generates feedback based on all conversation data and sentiment analysis results. Using a generative AI model, the server provides feedback to users, including areas for improvement and evaluations. Conversation data and analysis results are given as input, and feedback is generated as output.

[1527] Step 13:

[1528] The device then displays the generated feedback to the user, who is then shown on the smart glasses display, providing the user with specific points to improve and a visual assessment.

[1529] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1530] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1531] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1532] [Fourth embodiment]

[1533] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1534] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1535] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1536] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1537] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1538] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1539] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1540] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1541] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1542] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1543] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1544] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1545] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1546] The verbalization ability development system of the present invention is equipped with multiple generation means and a real-time feedback function to efficiently improve the user's communication ability. This system is composed of an image generation means, a voice generation means, a text generation means, and a real-time transcription means, which work together to provide the user with an environment that is similar to an actual communication situation.

[1547] Overview of the program

[1548] 1. Launch the application and select a scene

[1549] User: Launches the dedicated application on a smartphone and selects a specific scene (e.g., a university club meeting, a company meeting, etc.).

[1550] Terminal: The initial application screen is displayed and the user's selection is accepted.

[1551] 2. Character image and voice generation

[1552] Terminal: Sends information about the scene selected by the user to the server and makes a request to generate a character image.

[1553] Server: Uses the image generation means to generate a character image suitable for the specified scene and sends the image data to the terminal.

[1554] Terminal: Next, the terminal uses the voice generating means to send a request to the server to generate a voice corresponding to the character image.

[1555] Server: Using the voice generation means, generates a voice corresponding to the specified character and sends the voice data to the terminal.

[1556] Device: Combines the received character image and audio and displays them in the user's AR environment.

[1557] 3. Conversation scenario generation

[1558] Server: Generates conversation content using a text generation means appropriate for the scene, and sends the text data to the terminal.

[1559] Terminal: The received text data is converted into voice data using a voice generation means and played back as the character's speech.

[1560] 4. User utterances and response generation

[1561] User: Listen to the character's speech through earphones and express their own thoughts and opinions.

[1562] Terminal: Receives user speech and transcribes it in real time.

[1563] Terminal: Sends the text of the utterance to the server and requests that a response be generated.

[1564] Server: Uses the text generation means to generate an appropriate response text in response to the user's utterance and sends that data to the terminal.

[1565] Terminal: The received response text is converted into audio data using a voice generation means, and played back as if spoken by a character.

[1566] 5. Providing Feedback

[1567] Server: Analyzes the data from the entire conversation and evaluates the user's comments and responses.

[1568] Server: Uses a text generation means to generate feedback including evaluation results and advice, and sends the data to the terminal.

[1569] Terminal: Display received feedback to the user.

[1570] Specific examples

[1571] Example 1: University club meeting

[1572] 1. The user launches the application and selects the "University Club Meeting" scene.

[1573] 2. The device sends the scene information to the server and makes a request to generate an appropriate character image.

[1574] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[1575] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[1576] 5. The device displays the received character image and audio in the AR environment.

[1577] 6. The user begins a conversation with the character, and the device transcribes what is said and sends it to the server.

[1578] 7. The server generates an appropriate response and sends it to the terminal.

[1579] 8. The device plays the character's response as if it were speaking it.

[1580] 9. After the conversation is over, the server generates feedback and sends it to the device.

[1581] 10. The device displays feedback to the user.

[1582] In this way, the system allows users to practice in an environment that is similar to a real communication situation, and can efficiently improve their verbalization ability.

[1583] The processing flow will be explained below.

[1584] Step 1:

[1585] The user launches the dedicated application on their smartphone. The initial screen of the application is displayed, and they select a specific scene (e.g., a university club meeting, a company meeting, etc.).

[1586] Step 2:

[1587] The terminal receives the user's selection and sends scene information based on the selection to the server as a request, the request including detailed information about the selected scene.

[1588] Step 3:

[1589] The server analyzes the received request and sends instructions to the image generation means to generate a character image suitable for the scene. The image generation means generates a character image based on the specified features and returns the generated image data to the server.

[1590] Step 4:

[1591] The server receives the generated character image data and then transmits an instruction to generate a voice for the character to the voice generation means. The voice generation means generates a voice based on the specified characteristics and returns the generated voice data to the server.

[1592] Step 5:

[1593] The server then sends the received character image and audio data to the device, which then displays the character image in the AR environment and plays the audio based on the received data.

[1594] Step 6:

[1595] At the same time, the server uses a text generation means to generate conversation content according to the scene. The conversation text data generated based on the scene and character settings is transmitted from the server to the terminal.

[1596] Step 7:

[1597] The terminal transmits the received conversation text to the voice generation means, which generates voice data for the character to speak, and plays back the generated voice data so that the character can speak.

[1598] Step 8:

[1599] The user listens to the character's speech through earphones and expresses their own opinions and thoughts. The device receives the user's speech in real time and converts it into text (transcription).

[1600] Step 9:

[1601] The device transmits the transcribed user speech data in real time to the server, which then uses a text generation means to generate an appropriate response based on the received data.

[1602] Step 10:

[1603] The server transmits the generated response text data to the terminal. The terminal transmits the response text data to the voice generation means, which generates voice data for the character to speak. The voice data is then played back so that it sounds like the character is speaking.

[1604] Step 11:

[1605] After the conversation ends, the server analyzes the entire conversation data, evaluates the user's comments and responses, and generates feedback including the evaluation results and advice.

[1606] Step 12:

[1607] The server transmits the generated feedback data to the terminal, which displays the feedback to the user.

[1608] Step 13:

[1609] The user can choose to end the session and decide whether to save the conversation data and feedback. If the user chooses to save, the device sends the session data to the server, which then saves the data.

[1610] Example 1

[1611] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1612] Conventional communication skills development systems have had the problem of making it difficult for users to practice in an environment that closely resembles a real-life communication situation. In particular, the lack of real-time feedback makes it difficult for users to immediately evaluate the accuracy and appropriateness of their own speech, hindering efficient skill improvement.

[1613] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1614] In this invention, the server includes means for allowing a user to select a predetermined scene, means for generating a character image matching the predetermined scene using an image generation means, means for generating a voice based on the character image using a voice generation means, means for generating a conversation content corresponding to the scene using a text generation means, means for transcribing the user's utterance in real time, means for generating a response based on the generated transcription, means for displaying the generated character image and voice in an AR environment, means for generating and playing a response in real time in response to the user's utterance, and means for analyzing the generated conversation content and the user's utterance and providing appropriate feedback. This allows the user to efficiently train in an environment similar to an actual communication situation and instantly evaluate and improve the accuracy and appropriateness of their utterance.

[1615] "User" means an individual or organization that uses the system to improve their communication skills.

[1616] A "scene" refers to a specific situation or occasion selected by the user (e.g., a university club meeting, a company meeting, etc.).

[1617] "Image generation means" refers to a technology or device that enables a computer to generate corresponding images based on specified scenes or conditions, including, for example, deep learning models.

[1618] "Character image" refers to an image of a virtual character that is appropriate for the scene.

[1619] "Speech generation means" refers to technology or devices that synthesize speech based on text or digital data, including, for example, a speech synthesis engine.

[1620] "Text generation means" refers to a technology or device that generates natural language sentences based on input data, including, for example, a natural language processing model.

[1621] "Real-time transcription means" refers to technology or devices that convert user speech into text in real time, including, for example, speech recognition APIs.

[1622] The "means for generating a response" refers to a technology or device for generating an appropriate response text in response to a user's utterance.

[1623] An "AR environment" refers to a technological environment that displays virtual objects and information superimposed on the real environment.

[1624] "Means for providing feedback" refers to a technology or device that analyzes the content of a conversation that a user has had and provides advice or evaluation based on the results.

[1625] The verbalization ability development system of the present invention is designed to efficiently improve a user's communication ability and is equipped with multiple generation means and a real-time feedback function. This system is composed of an image generation means, a voice generation means, a text generation means, and a real-time transcription means, which work together to provide the user with an environment that is similar to an actual communication situation.

[1626] 1. Launching the application and selecting a scene

[1627] A user starts a dedicated application using a device such as a smartphone. On the initial screen, the user selects a specific scene (e.g., a university club meeting, a company meeting, etc.). The device then sends this scene information to the server.

[1628] 2. Character image and voice generation

[1629] The device sends a request to the server to generate a character image. The server uses an image generation means (e.g., a deep learning-based model) to generate a character image suitable for the specified scene and sends the data to the device. Next, the device sends a request to the server to generate audio based on the character image. The server uses an audio generation means (e.g., a speech synthesis engine) to generate audio corresponding to the specified character and sends this audio data to the device. The device combines these pieces of data and displays them in the user's AR environment.

[1630] 3. Conversation scenario generation

[1631] The device uses a text generation means to send a request to the server to generate conversation content appropriate for the scene. The server uses a text generation model (e.g., GPT-4) to generate conversation text appropriate for the scene and sends this text data to the device. The device then converts the received text data into audio data and plays it back as the character's speech.

[1632] 4. User utterances and response generation

[1633] The user listens to the character's speech through earphones and expresses their own thoughts and opinions. The device transcribes the user's speech in real time using the Google Speech-to-Text API and sends this text data to the server. The server uses a text generation means to generate an appropriate response text to the user's speech and sends this data to the device. The device converts the received response text into audio data and plays it back as if the character were speaking the response.

[1634] 5. Providing Feedback

[1635] The server analyzes the data from the entire conversation and evaluates the user's comments and responses. Based on the analysis results, the server uses a text generation means to generate feedback including the evaluation results and advice, and sends this data to the terminal. The terminal displays the received feedback to the user. The user can then identify areas for improvement based on the feedback and use them in the next conversation.

[1636] Specific examples

[1637] For example, if the user selects the scene "University Club Meeting," the following steps are taken:

[1638] 1. The user launches the application and selects the "University Club Meeting" scene.

[1639] 2. The device sends the scene information to the server and makes a request to generate an appropriate character image.

[1640] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[1641] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[1642] 5. The device displays the received character image and audio in the AR environment.

[1643] 6. The user begins a conversation with the character, and the device transcribes what is said and sends it to the server.

[1644] 7. The server generates an appropriate response and sends it to the terminal.

[1645] 8. The device plays the character's response as if it were speaking it.

[1646] 9. After the conversation is over, the server generates feedback and sends it to the device.

[1647] 10. The device displays feedback to the user.

[1648] In this way, the system allows users to train in an environment that is close to a real communication situation, and can efficiently improve their verbalization ability. In this system, the following are examples of prompt sentences when using a generative AI model:

[1649] ---

[1650] Example prompt sentence:

[1651] "Imagine a scene at a university club meeting where an idea for a new project is proposed, and generate how the characters would react."

[1652] ---

[1653] This allows users to hone their practical communication skills through conversations tailored to specific situations.

[1654] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1655] Step 1: Launch the application and select a scene

[1656] 1. The user taps the dedicated application on their smartphone to launch the app.

[1657] Input: Tap the app icon.

[1658] Output: The initial screen of the application is displayed.

[1659] 2. The terminal displays the initial screen and waits for user operation.

[1660] What it does: Draws the UI components on the initial screen.

[1661] 3. On the initial screen that appears, the user selects a scene such as a "university club meeting" or a "company meeting."

[1662] Input: Tap to select a scene.

[1663] Output: The information of the selected scene is saved to the device.

[1664] Step 2: Generate character images and sounds

[1665] 1. The device sends information about the scene selected by the user to the server and makes a request to generate a character image.

[1666] Input: Scene information.

[1667] Output: Request sent.

[1668] What it does: Creates and sends an HTTP request.

[1669] 2. The server receives the request and uses the image generation means to generate a character image suitable for the specified scene.

[1670] Input: A request for scene information.

[1671] Output: Character image data.

[1672] Action: Running an image generation model.

[1673] 3. The server sends the generated character image data to the terminal.

[1674] Input: Character image data.

[1675] Output: Sending image data.

[1676] What it does: Generates and sends an HTTP response.

[1677] 4. The device receives the character image and stores it in memory.

[1678] Input: Character image data.

[1679] Output: Saving image data.

[1680] Action: Processing data storage.

[1681] 5. The device sends a request to the server to generate audio based on the character image.

[1682] Input: Character image information.

[1683] Output: Request sent.

[1684] What it does: Creates and sends an HTTP request.

[1685] 6. The server uses the voice generation means to generate a voice corresponding to the specified character.

[1686] Input: Request for character information.

[1687] Output: Character voice data.

[1688] What it does: Runs the speech synthesis engine.

[1689] 7. The server sends the generated voice data to the terminal.

[1690] Input: Character voice data.

[1691] Output: Sending audio data.

[1692] What it does: Generates and sends an HTTP response.

[1693] 8. The device combines the character image and audio data and prepares to display them in the user's AR environment.

[1694] Input: Character image and audio data.

[1695] Output: AR display preparation.

[1696] What it does: Synchronize and prepare data.

[1697] Step 3: Generate conversation scenarios

[1698] 1. The device sends a request to the server to generate conversation content appropriate for the scene.

[1699] Input: Scene information.

[1700] Output: Request sent.

[1701] What it does: Creates and sends an HTTP request.

[1702] 2. The server uses a text generation means to generate conversation content appropriate for the scene.

[1703] Input: A request for scene information.

[1704] Output: Text data of the conversation scenario.

[1705] What it does: Run a text generation model.

[1706] 3. The server sends the text data of the generated conversation scenario to the terminal.

[1707] Input: Text data of a conversation scenario.

[1708] Output: Sending text data.

[1709] What it does: Generates and sends an HTTP response.

[1710] 4. The device converts the received text data into audio data and plays it back as the character's speech.

[1711] Input: Text data of a conversation scenario.

[1712] Output: Generate audio data.

[1713] What it does: Performs text-to-speech (TTS).

[1714] Step 4: User utterances and response generation

[1715] 1. The device plays the character's voice.

[1716] Input: Audio data.

[1717] Output: Audio playback.

[1718] Action: Performs audio playback function.

[1719] 2. The user listens to the character's speech and expresses their own thoughts and opinions.

[1720] Input: Character speech.

[1721] Output: User utterance.

[1722] Action: User's auditory comprehension and speech.

[1723] 3. The device collects the user's speech and transcribes it in real time.

[1724] Input: User speech.

[1725] Output: Text data.

[1726] What it does: Transcription using the Google Speech-to-Text API.

[1727] 4. The device sends the transcribed text data to the server and makes a request to generate a response.

[1728] Input: Speech text data.

[1729] Output: Request sent.

[1730] What it does: Creates and sends an HTTP request.

[1731] 5. The server uses a text generation means to generate an appropriate response based on the user's utterance.

[1732] Input: User utterance text data.

[1733] Output: Response text data.

[1734] What it does: Run a text generation model.

[1735] 6. The server sends the response text data to the terminal.

[1736] Input: Response text data.

[1737] Output: Sending text data.

[1738] What it does: Generates and sends an HTTP response.

[1739] 7. The device converts the received response text into audio data and plays it back as if the character were speaking the response.

[1740] Input: Response text data.

[1741] Output: Generation and playback of audio data.

[1742] What it does: Performs text-to-speech (TTS) and plays audio.

[1743] Step 5: Provide feedback

[1744] 1. The server analyzes the data from the entire conversation and evaluates the user's comments and responses.

[1745] Input: Utterance and response data.

[1746] Output: Evaluation results.

[1747] Action: Run the data analysis model.

[1748] 2. The server generates feedback based on the evaluation results using a text generation means.

[1749] Input: Analysis results.

[1750] Output: Feedback text data.

[1751] What it does: Run a text generation model.

[1752] 3. The server sends the feedback data to the terminal.

[1753] Input: Feedback text data.

[1754] Output: Sending text data.

[1755] What it does: Generates and sends an HTTP response.

[1756] 4. The device displays the received feedback to the user.

[1757] Input: Feedback text data.

[1758] Output: Display feedback.

[1759] Action: Drawing a UI component.

[1760] (Application example 1)

[1761] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1762] There is a need to provide a training environment in brick-and-mortar stores that allows store employees to efficiently improve their customer service skills. In particular, it is important to improve conversation skills in specific situations, such as explaining new products or handling complaints. However, because training that replicates real-world situations is time-consuming and costly, there is a need to develop an easily accessible simulation tool.

[1763] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1764] In this invention, the server includes means for allowing a user to select a predetermined scene, means for generating a character image that matches the predetermined scene using an image generation means, means for generating a voice based on the character image using a voice generation means, means for generating a conversation content corresponding to the scene using a text generation means, means for transcribing utterances from the user in real time, means for generating a response based on the generated transcription, and means for generating a conversation scenario corresponding to a specific situation in customer service work. This provides a simulation environment that is close to a real customer service situation, enabling store clerks to efficiently improve their customer service skills.

[1765] "Means for allowing the user to select a specific scene" refers to a function that provides an interface for the user to use the application to select a specific scene or situation (e.g., explaining a new product, handling a complaint, etc.).

[1766] "Image generation means" refers to a technology for generating character images that match a specific scene, and is a means for generating high-quality images using machine learning and deep learning.

[1767] The "voice generation means" is a technology for generating appropriate voice based on the generated character image, and is a means for converting text data into voice data.

[1768] The "text generation means" is a technology for generating conversation content appropriate to the scene, and is a means for generating appropriate text data using natural language processing technology.

[1769] "Means for transcribing user speech in real time" refers to a technology for instantly converting what a user says into text data, and is a means that uses voice recognition technology.

[1770] The "means for generating a response based on the generated transcription" is a technology for receiving a user's speech as text data and generating an appropriate response in response to it.

[1771] "Means for generating conversation scenarios according to specific situations in customer service work" refers to technology for generating scenarios in specific situations related to customer service work, and is a means for automatically generating conversation content according to user selections.

[1772] An "augmented reality environment" is a technology that displays virtual information overlaid on the real world, and is an environment that integrates the real environment with virtual information using multimedia content such as 3D models, videos, and audio.

[1773] This invention provides a system for efficiently training in customer service work. The system has a function that allows a user to select a predetermined scene and experience a virtual customer service situation based on that scene. The following describes in detail how this system can be realized.

[1774] Program Structure

[1775] The system consists of the following main components:

[1776] 1. A means for allowing the user to select a specific scene: A user interface is provided that allows the user to select a scene, such as explaining a new product or handling a complaint.

[1777] 2. Image generation method: Use a deep learning model to generate character images suitable for a given scene. For example, we use TensorFlow.

[1778] 3. Voice generation method: Google Cloud Text-to-Speech is used to generate voice based on the generated character image.

[1779] 4. Text generation: Use natural language processing models such as GPT-3 to generate conversational content appropriate to the situation.

[1780] 5. A way to transcribe user speech in real time: Use Google Cloud Speech-to-Text.

[1781] 6. Generating responses based on the generated transcription: The GPT-3 model is used to generate appropriate responses to the user's utterances.

[1782] 7. A means of generating conversation scenarios for specific situations in customer service: Automatically create conversation scenarios for specific situations based on user selections.

[1783] Hardware and Software Configuration

[1784] Hardware: Smartphone (iOS or Android), earphones

[1785] software:

[1786] TensorFlow: Image Generation

[1787] Google Cloud Text-to-Speech: Audio Generation

[1788] Google Cloud Speech-to-Text: Real-time transcription

[1789] GPT-3: Text and response generation

[1790] Specific examples

[1791] Example: New product description

[1792] The user launches the application and selects the "Explanation of a new product" scene. The application sends the selection information to the server and requests the generation of an appropriate character image and sound. The server generates a character image and sound using the image generation means and sound generation means, and sends them to the smartphone. The smartphone displays the received character image and sound in the augmented reality environment, and the user begins a virtual customer service interaction with the character.

[1793] When a user says, "Today I'll talk about our new product," the smartphone transcribes the speech in real time and sends it to the server. The server then generates an appropriate response, such as repeating what the user said or asking follow-up questions, and sends it to the smartphone. The character then replies, "Can you tell me about the features of our new product?" and the conversation continues.

[1794] Prompt Sentence Examples

[1795] Below are some example inputs for a generative AI model:

[1796] Please generate a conversation scenario with questions and answers from customers when explaining a new product. The specific product name is Smartwatch Z. Please create the scenario by including points that customers are likely to have questions about.

[1797] In this way, the present invention provides a system for efficiently conducting customer service training in a real store, allowing users to train in a realistic simulation environment.

[1798] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1799] Step 1:

[1800] The user selects a given scene

[1801] The user launches the customer service trainer app on their smartphone and selects a specific scene, such as "explaining a new product" or "handling a complaint," from the scene selection screen. The smartphone then sends the information about the selected scene to the server.

[1802] Input: User scene selection

[1803] Output: Send scene selection information to the server

[1804] Step 2:

[1805] Character image generation

[1806] Based on the received scene selection information, the server uses a deep learning model (e.g., TensorFlow) to generate a character image suitable for the specified scene. The generated character image is then sent from the server to the smartphone.

[1807] Input: Scene selection information

[1808] Output: Character image

[1809] Step 3:

[1810] Audio generation

[1811] The server uses Google Cloud Text-to-Speech to generate voice for the character based on the generated character image, and the generated voice is sent from the server to the smartphone.

[1812] Input: Character image

[1813] Output: Character voice

[1814] Step 4:

[1815] Conversation scenario generation using text generation

[1816] The server uses GPT-3 to generate conversation content based on the scene selected by the user. The conversation scenario is sent as text data from the server to the smartphone, which then converts it into audio data and plays it back.

[1817] Input: Scene selection information

[1818] Output: Conversation scenario (text data)

[1819] Step 5:

[1820] Transcription of user utterances

[1821] Users listen to the character's speech through headphones and express their thoughts and opinions. The smartphone then transcribes the speech in real time using Google Cloud Speech-to-Text and sends the data to a server.

[1822] Input: User utterance

[1823] Output: Transcription data

[1824] Step 6:

[1825] Generating a response

[1826] The server analyzes the text data using the GPT-3 model and generates an appropriate response, which is then sent to the smartphone, where it is converted into audio and played back.

[1827] Input: Transcription data

[1828] Output: Response text data

[1829] Step 7:

[1830] Providing feedback

[1831] The server analyzes the entire conversation data, evaluates the user's comments and responses, and generates appropriate feedback. The feedback is sent as text data from the server to the smartphone and displayed to the user.

[1832] Input: Data for the entire conversation

[1833] Output: Feedback data

[1834] In this way, the system allows users to efficiently improve their customer service skills in a realistic simulation environment.

[1835] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1836] The verbalization ability development system of the present invention is equipped with multiple generation means and a real-time feedback function to efficiently improve the user's communication ability. This system is composed of an image generation means, a voice generation means, a text generation means, an emotion engine, and a real-time transcription means, which work together to provide the user with an environment that is similar to an actual communication situation.

[1837] Overview of the program

[1838] 1. Launch the application and select a scene

[1839] User: Launches the dedicated application on a smartphone and selects a specific scene (e.g., a university club meeting, a company meeting, etc.).

[1840] Terminal: The initial application screen is displayed and the user's selection is accepted.

[1841] 2. Character image and voice generation

[1842] Terminal: Sends information about the scene selected by the user to the server and makes a request to generate a character image.

[1843] Server: Uses the image generation means to generate a character image suitable for the specified scene and sends the image data to the terminal.

[1844] Terminal: Next, the terminal uses the voice generating means to send a request to the server to generate a voice corresponding to the character image.

[1845] Server: Using the voice generation means, generates a voice corresponding to the specified character and sends the voice data to the terminal.

[1846] Device: Combines the received character image and audio and displays them in the user's AR environment.

[1847] 3. Conversation scenario generation

[1848] Server: Generates conversation content using a text generation means appropriate for the scene, and sends the text data to the terminal.

[1849] Terminal: The received text data is converted into voice data using a voice generation means and played back as the character's speech.

[1850] 4. User utterances and response generation

[1851] User: Listens to the character's speech through earphones and expresses his or her own opinions and thoughts. The device receives the user's speech in real time and converts it into text (transcription).

[1852] Device: The transcribed user speech data is sent to the server in real time. The server then analyzes the user's emotions using the received data and the emotion engine.

[1853] Server: Uses a text generation means to generate an appropriate response text based on the user's utterance and emotions, and sends the data to the terminal.

[1854] Terminal: The received response text is converted into audio data using a voice generation means, and played back as if spoken by a character.

[1855] 5. Providing feedback and using emotions

[1856] Server: Analyzes the overall conversation data and the emotional data obtained from the emotion engine, and evaluates the user's comments and responses.

[1857] Server: Generates feedback including evaluation results and advice that take emotions into account, and sends this data to the device.

[1858] Terminal: Display received feedback to the user.

[1859] Specific examples

[1860] Example 1: University club meeting

[1861] 1. The user launches the application and selects the "University Club Meeting" scene.

[1862] 2. The device sends the scene information to the server and makes a request to generate an appropriate character image.

[1863] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[1864] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[1865] 5. The device displays the received character image and audio in the AR environment.

[1866] 6. The user begins a conversation with the character, and the device transcribes what is said and sends it to the server.

[1867] 7. The server generates a response based on the user's speech and the emotion engine and sends it to the device.

[1868] 8. The device plays the character's response as if it were speaking it.

[1869] 9. After the conversation ends, the server generates a rating based on the emotional data along with feedback and sends it to the device.

[1870] 10. The device displays feedback to the user.

[1871] In this way, the system allows users to train in an environment that is similar to a real communication situation, and can efficiently improve their verbalization and emotion recognition abilities.

[1872] The processing flow will be explained below.

[1873] Step 1:

[1874] The user launches the dedicated application on their smartphone. The initial screen of the application is displayed, and they select a specific scene (e.g., a university club meeting, a company meeting, etc.).

[1875] Step 2:

[1876] The terminal receives the user's selection and sends scene information based on the selection to the server as a request, the request including detailed information about the selected scene.

[1877] Step 3:

[1878] The server analyzes the received request and sends instructions to the image generation means to generate a character image suitable for the scene. The image generation means generates a character image based on the specified features and returns the generated image data to the server.

[1879] Step 4:

[1880] The server receives the generated character image data and then transmits an instruction to generate a voice for the character to the voice generation means. The voice generation means generates a voice based on the specified characteristics and returns the generated voice data to the server.

[1881] Step 5:

[1882] The server then sends the received character image and audio data to the device, which then displays the character image in the AR environment and plays the audio based on the received data.

[1883] Step 6:

[1884] At the same time, the server uses a text generation means to generate conversation content according to the scene. The conversation text data generated based on the scene and character settings is transmitted from the server to the terminal.

[1885] Step 7:

[1886] The terminal transmits the received conversation text to the voice generation means, which generates voice data for the character to speak, and plays back the generated voice data so that the character can speak.

[1887] Step 8:

[1888] The user listens to the character's speech through earphones and expresses their own opinions and thoughts. The device receives the user's speech in real time and converts the speech into text (transcription).

[1889] Step 9:

[1890] The device transmits the transcribed user speech data to the server in real time, and the server uses an emotion engine to analyze the user's emotions based on the received data and facial expressions.

[1891] Step 10:

[1892] The server uses the text generation means based on the analysis results to generate an appropriate response text according to the user's utterance and emotion. The generated response text is sent from the server to the terminal.

[1893] Step 11:

[1894] The terminal transmits the received response text to the voice generation means, which generates voice data for the character to speak, and plays back the generated voice data so that the character can speak it.

[1895] Step 12:

[1896] Once the conversation is over, the server analyzes the entire conversation data and the emotional data obtained from the emotion engine, evaluates the user's comments and responses, and generates feedback including the evaluation results and advice.

[1897] Step 13:

[1898] The server transmits the generated feedback data to the terminal, which displays the feedback to the user.

[1899] Step 14:

[1900] The user can choose to end the session and decide whether to save the conversation data and feedback. If the user chooses to save, the device sends the session data to the server, which then stores the data.

[1901] Example 2

[1902] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1903] Current systems for improving communication skills have the problem that it is difficult for users to receive training in an environment that closely resembles a real communication situation. Furthermore, there is a lack of systems that can analyze not only the content of a user's speech but also their emotions and provide appropriate feedback based on that. This makes it difficult to simultaneously improve both a user's language ability and their emotion recognition ability.

[1904] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1905] In this invention, the server includes means for a user to select a predetermined scene, means for generating a character image that matches the predetermined scene using an image generation means, means for generating a voice based on the character image using a voice generation means, means for generating a conversation content according to the scene using a text generation means, means for transcribing the user's speech in real time, means for generating a response based on the generated transcription, means for analyzing the user's speech using an emotion analysis engine and generating an appropriate response, and means for displaying the generated character image and voice in an augmented reality environment. This allows the user to train in an environment that resembles an actual communication situation, thereby simultaneously improving both language ability and emotion recognition ability.

[1906] A "user" is an individual or corporation that uses this system to achieve a specific purpose.

[1907] A "scene" is a specific situation or environment selected by the user, which is the subject of training or simulation.

[1908] "Image generation means" refers to technology or equipment for generating character images that match a specific scene.

[1909] "Character images" are image data of people or objects generated according to the scene.

[1910] "Sound generation means" refers to technology or devices that generate sounds based on character images.

[1911] "Text generation means" refers to a technology or device that generates conversation content appropriate for a specific scene.

[1912] "Transcription" is the process of converting a user's speech from audio to text in real time.

[1913] "Response generation means" refers to technology or devices that generate appropriate responses based on the generated transcription or other data.

[1914] An "emotion analysis engine" refers to technology or equipment for analyzing emotions from a user's speech or behavior.

[1915] "Augmented reality environment" refers to technology that displays virtual information overlaid on the real environment, and the environment that uses it.

[1916] This invention is a system for improving a user's communication skills, which provides training by selecting a specific scene and having a conversation with a character. This system is configured to include the following means.

[1917] First, a user launches a system-specific application using a device such as a smartphone or tablet. On the initial screen of the application, the user selects a desired scene from multiple scenes (e.g., a university club meeting, a company meeting, etc.).

[1918] The selected scene information is transmitted from the terminal to the server. The server uses an image generation means to generate a character image that matches the selected scene. For example, a deep learning model can be used for this image generation. The generated character image data is transmitted from the server to the terminal.

[1919] Next, the terminal sends a request to the server to generate a voice for the character. The server uses a voice generation means to generate a voice based on the character image. A text-to-speech engine can be used for generating the voice. The generated voice data is sent to the terminal, and the terminal displays the character image and voice in combination.

[1920] A text generation means within the server is used to generate the conversation scenario. The text generation means uses, for example, a generative AI model that makes full use of natural language processing technology to generate conversation content that corresponds to the scene selected by the user. The generated text data is sent to the terminal, where it is converted into audio data via a text-to-speech engine and played back as the character's speech.

[1921] The user listens to the character's speech through earphones and a microphone and expresses their opinions and thoughts. The device records the user's speech in real time and converts it into text (transcription). The transcribed data is then sent to the server.

[1922] The server uses an emotion analysis engine to analyze the user's speech data and identify emotions. It then uses a text generation means to generate an appropriate response based on the user's speech content and emotions. This response data is sent to the terminal, converted into voice data, and played as the character's speech.

[1923] As a concrete example, if you select the "University Club Meeting" scene, the system will behave as follows:

[1924] 1. The user launches the application and selects the "University Club Meeting" scene.

[1925] 2. The device sends the selected scene ID to the server.

[1926] 3. The server uses the image generation means to generate a character image and transmits it to the terminal.

[1927] 4. The server uses a voice generation means to generate the character's voice and transmits it to the terminal.

[1928] 5. The device displays the character image and sound in the AR environment.

[1929] 6. The user begins to interact with the character, and the device transcribes what is said and sends it to the server.

[1930] 7. The server analyzes the user's speech and emotions, generates an appropriate response, and sends it to the device.

[1931] 8. The device plays back the response as if it were being spoken by a character.

[1932] 9. After the conversation ends, the server generates feedback based on the emotion data and sends it to the device.

[1933] 10. The device displays feedback to the user.

[1934] An example prompt is:

[1935] You have selected a university club meeting scene. The server and device interact, generating character images and voices. The following conversation unfolds between you and the character:

[1936] "Today I would like to discuss a new project. I'd like to hear your thoughts."

[1937] How would you respond to this statement?

[1938] In this way, users can receive training in an environment that is close to a real communication situation, improving both their language skills and their emotion recognition abilities.

[1939] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1940] Step 1:

[1941] The user launches a dedicated application on the device and selects a specific scene. At this time, scene information (e.g., scene ID) is provided as input. The device accepts the user's selection and sends the scene information to the server. The output at this stage is that the scene information is sent to the server.

[1942] Step 2:

[1943] The device sends scene information to the server and requests the server to generate a character image. The server receives the scene information as input and generates a character image using an image generation means. Specifically, it uses an AI model to design a character image that fits the scene and generates its data. This image data is generated as output and sent from the server to the device.

[1944] Step 3:

[1945] The terminal receives the character image sent from the server and prepares to display it. It then sends a request to the server again to generate voice. This request includes information about the character image, which is passed to the server as input. The server uses a voice generation means to generate voice for the character. At this time, it uses a text-to-speech engine to convert the specified text into voice data. The generated voice data is sent from the server to the terminal.

[1946] Step 4:

[1947] The server uses a text generation means to generate dialogue content corresponding to the scene selected by the user. Scene information and prompt sentences are used as input, and natural language processing technology is used to generate appropriate dialogue text. This dialogue text is generated as output and sent to the terminal. The terminal converts this text into audio data and plays it back as if the character were speaking.

[1948] Step 5:

[1949] The user listens to the character's speech through headphones and expresses their own opinions and thoughts through a microphone. The user's speech is input, and the device records and transcribes it in real time. This transcription data is output and sent to the server.

[1950] Step 6:

[1951] The server uses a sentiment analysis engine to analyze the user's speech data. It receives the transcription data as input and analyzes it using a machine learning model to identify the user's emotions. The analysis results are output and used as input data for generating an appropriate response text. It uses a text generation means to generate a response text based on the user's speech content and emotions. The generated response text is sent to the device.

[1952] Step 7:

[1953] The device converts the received response text into audio data and plays it as if the character were speaking.The device converts the text into audio using a voice generation means and displays it as if the character is speaking in the AR environment.

[1954] Step 8:

[1955] After the conversation ends, the server analyzes all conversation content and emotional data and generates feedback for the user. The emotional data and spoken content are used as inputs for this analysis. The generated feedback information is output and sent to the device. The device displays the feedback to the user, allowing the user to improve themselves based on the results.

[1956] (Application example 2)

[1957] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1958] There is a need to efficiently and effectively improve the customer service skills of employees in brick-and-mortar stores. However, traditional training methods have difficulty providing an environment similar to actual customer interactions, limiting their effectiveness in improving communication skills. Furthermore, because training feedback is not provided in real time, it is difficult for employees to quickly understand and correct areas for improvement. Another issue is the lack of a system for analyzing employees' emotions and attitudes and providing appropriate advice.

[1959] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for allowing a user to select a predetermined scene, means for generating a person image matching the predetermined scene using an image generation means, means for generating a voice based on the person image using a voice generation means, means for generating a dialogue content appropriate to the scene using a text generation means, means for transcribing user utterances in real time, means for generating a response based on the generated transcription, emotion analysis means for analyzing the transcribed user utterances and emotions, means for displaying the generated response and analysis results in a substitute real environment, and means for providing feedback based on the response. This makes it possible to conduct training while receiving real-time feedback in an environment that resembles an actual customer service situation, thereby effectively improving employees' communication skills and customer service skills.

[1960] The "means for allowing the user to select a predetermined scene" refers to a means for providing an interface that allows the user to select a specific scenario or situation in advance.

[1961] "Image generation means" refers to an algorithm or program for automatically generating a person image that matches a scene selected by a user.

[1962] The "audio generating means" refers to a system or program for generating appropriate audio data based on the generated person image.

[1963] "Text generation means" refers to software or algorithms that automatically generate dialogue content appropriate for a specific situation.

[1964] "Means for transcribing user utterances in real time" refers to technology or devices that convert user speech into text in real time.

[1965] "Means for generating a response based on the generated transcription" refers to a mechanism for analyzing the transcribed content of a user's speech and automatically generating an appropriate corresponding response.

[1966] "Emotion analysis means" refers to technology that analyzes a user's comments and the emotions they express at the time, and provides appropriate actions and feedback based on the results.

[1967] "Means for displaying in a virtual reality environment" refers to a system or device for displaying the generated human image and audio in the virtual or alternate reality space experienced by the user.

[1968] "Means for providing feedback" refers to a mechanism that provides information for improvement and evaluation in real time based on the results of users' actions and comments.

[1969] This invention is a system for improving customer service skills in brick-and-mortar stores. The system aims to provide training that is similar to actual customer service situations by having users interact with customers in a virtual environment.

[1970] System Overview

[1971] The system includes the following main components:

[1972] 1. A means by which the user can select a given scene:

[1973] It provides an interface that allows users (store employees) to select a scene (e.g., introducing a new product, working at the cash register, etc.).

[1974] For example, by selecting a menu on the smart glasses display, the user can easily select the appropriate scene.

[1975] 2. Image generation means:

[1976] An algorithm or program for generating a person image that matches a selected scene.

[1977] Here, we use technologies such as OpenAI's DALL-E as image generation technology.

[1978] 3. Sound Generation Means:

[1979] A technology that generates appropriate voice data based on the generated person image.

[1980] For example, we use Azure Cognitive Services to generate a virtual customer voice.

[1981] 4. Text generation methods:

[1982] A technology that automatically generates dialogue content based on the selected situation.

[1983] Here, we use a text generation model (e.g., GPT-3) to generate scenario-based questions and answers.

[1984] 5. Means of transcribing user comments in real time:

[1985] A technology that converts user speech into text in real time.

[1986] It uses Google Speech-to-Text technology to instantly transcribe what users say.

[1987] 6. How to generate a response based on the generated transcript:

[1988] A system that automatically generates appropriate responses based on transcription data.

[1989] Again, a text generation model (e.g., GPT-3) is used to generate responses.

[1990] 7. Emotion analysis means:

[1991] A technology that analyzes a user's comments and their emotions at the time, and provides appropriate actions and feedback based on the results.

[1992] IBM Watson is used as the sentiment analysis engine.

[1993] 8. Means of display in virtual reality environment:

[1994] A technology that displays generated human images and audio in a virtual or augmented reality environment.

[1995] Smart glasses (e.g., Microsoft HoloLens) are used to provide visual and auditory information to users.

[1996] 9. Ways to provide feedback:

[1997] A system that provides real-time feedback based on user actions and comments.

[1998] Based on the analysis results, appropriate evaluations and improvement advice are provided to users in real time.

[1999] Specific examples

[2000] For example, in a scenario where a new product is being introduced, a user (employee) can wear smart glasses and undergo training. The user is asked questions such as, "What are the features of the new product?" and as the user responds, the system performs real-time transcription and sentiment analysis. As a result, feedback is provided on the accuracy of the response and the employee's attitude.

[2001] Prompt Sentence Examples

[2002] An example prompt for a new product introduction scene is as follows:

[2003] "Please tell me the features of this product."

[2004] Points to consider: Product features, benefits, user expectations, etc.

[2005] In this way, the system helps store employees to receive training in an environment that closely resembles an actual customer service situation, effectively improving their communication and customer service skills.

[2006] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2007] Step 1:

[2008] The user puts on the smart glasses and launches the application. Through the interface, the user selects a specific training scenario (e.g., a new product introduction) based on a real customer service scene. Scene information is selected as input, and the selected information is sent to the device as output.

[2009] Step 2:

[2010] The device sends the selected scene information to the server, which receives this information and uses an image generation model (e.g., OpenAI DALL-E) to generate a virtual customer persona that fits the scene. The scene information is given as input, and the generated persona is sent as output to the device.

[2011] Step 3:

[2012] The device sends a voice generation request to the server based on the received person image. The server uses a voice generation model (e.g., Azure Cognitive Services) to generate voice data that matches the scene in the voice of the virtual customer. The person image is given as input, and the voice data is sent to the device as output.

[2013] Step 4:

[2014] The device displays and plays the received image and audio data. The virtual customer is displayed on the smart glasses' display and the corresponding audio is played. By providing the user with visual and audio information, a realistic customer service scene is simulated.

[2015] Step 5:

[2016] The server uses a text generation model (e.g., GPT-3) to generate dialogue content based on the selected scene. Scene information is given as input, and appropriate questions and comments are sent as output to the device as text data.

[2017] Step 6:

[2018] The device converts the received text data into voice data using a speech generation model and plays it back as if the virtual customer were speaking it, for example, asking a question such as, "What are the features of this product?"

[2019] Step 7:

[2020] The user speaks their opinions and answers to the questions posed by the virtual customer through the microphone in the smart glasses, and the voice data is sent to the terminal as input.

[2021] Step 8:

[2022] The device uses speech recognition technology (e.g., Google Speech-to-Text) to transcribe and convert user voice data into text in real time. Voice data is given as input and text data is generated as output.

[2023] Step 9:

[2024] The device sends the generated text data to a server, which uses an emotion analysis engine (e.g., IBM Watson) to analyze the user's speech and the emotions expressed at the time. The text data is given as input, and the emotion analysis results are obtained as output.

[2025] Step 10:

[2026] The server uses a text generation model to generate an appropriate response text based on the user's utterances and emotions. The analysis results and the user's utterances are given as input, and the response text is generated as output.

[2027] Step 11:

[2028] The terminal converts the response text into speech data and plays it back as if the virtual customer were speaking the response, again using a speech model, and provides the generated speech data to the user.

[2029] Step 12:

[2030] The server generates feedback based on all conversation data and sentiment analysis results. Using a generative AI model, the server provides feedback to users, including areas for improvement and evaluations. Conversation data and analysis results are given as input, and feedback is generated as output.

[2031] Step 13:

[2032] The device then displays the generated feedback to the user, who is then shown on the smart glasses display, providing the user with specific points to improve and a visual assessment.

[2033] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2034] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2035] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2036] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2037] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2038] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2039] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2040] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2041] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2042] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2043] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2044] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2045] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2046] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2047] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2048] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2049] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2050] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2051] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2052] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2053] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2054] The following is further disclosed regarding the above embodiment.

[2055] (Claim 1)

[2056] means for allowing a user to select a predetermined scene;

[2057] means for generating a character image that matches a predetermined scene using an image generation means;

[2058] a means for generating a sound based on the character image using a sound generating means;

[2059] a means for generating conversation content according to a scene using a text generation means;

[2060] A means for transcribing user utterances in real time;

[2061] The system includes means for generating a response based on the generated transcription.

[2062] (Claim 2)

[2063] The system of claim 1, further comprising means for analyzing the generated conversation content and the user's utterance and providing appropriate feedback.

[2064] (Claim 3)

[2065] The system of claim 1 , further comprising: means for displaying the generated character image and sound in an AR environment.

[2066] "Example 1"

[2067] (Claim 1)

[2068] means for allowing a user to select a predetermined scene;

[2069] means for generating a character image that matches a predetermined scene using an image generation means;

[2070] a means for genera...

Claims

1. means for allowing a user to select a predetermined scene; means for generating a character image that matches a predetermined scene using an image generation means; a means for generating a sound based on the character image using a sound generating means; a means for generating conversation content according to a scene using a text generation means; A means for transcribing user utterances in real time; The system includes means for generating a response based on the generated transcription.

2. The system of claim 1 , further comprising means for analyzing the generated conversation content and the user's utterance and providing appropriate feedback.

3. The system of claim 1 , further comprising: means for displaying the generated character image and sound in an AR environment.

Images