system

Abstract

A system is provided.SOLUTION: A system, comprising: means for configuring a VR environment; means for capturing audio and motion of a user in real-time; means including an artificial intelligence model for analyzing the captured audio and motion data and generating feedback; means for displaying the feedback to the user; and means for storing the generated feedback and an assessment score.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's world, online interactions such as chat and phone calls are increasingly common when communicating with loved ones or the opposite sex, reducing the opportunities to see each other face-to-face. This makes it difficult to see the other person's facial expressions and gestures, making it difficult to propose to a loved one or confess one's feelings to a loved one. Even if one actually does so, there is a concern that it may end in failure. Therefore, there is a need for a way for users to practice their words and behavior to ensure the ideal proposal or confession is successful. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following system. First, by including a means for setting up a VR environment, the user can practice in a virtual space. Second, by including a means for capturing the user's voice and movements in real time, it is possible to practice in a situation close to the real thing. Furthermore, by including a means for analyzing the captured voice and movement data and including an artificial intelligence model for generating feedback, it is possible to obtain an evaluation similar to that of the real thing. Finally, by including a means for presenting and saving the generated feedback and evaluation score to the user, the user can check their progress and identify areas for improvement. In this way, the user can improve their confession and proposal skills.

[0006] A "VR environment" is a virtual space in which users can act and interact as if in reality.

[0007] "User voice and movements" refers to the voice and body movements made by the user within the VR environment, and these are captured to enhance the realism of the simulation.

[0008] "Real-time capture means" refers to technology that instantly captures a user's voice and movements as digital data, making that data immediately available for analysis and display.

[0009] "Captured voice and movement data" means voices uttered and movements made by a user that are digitally recorded for subsequent analysis and evaluation.

[0010] "Artificial intelligence model for analysis and feedback generation" refers to machine learning algorithms and other AI technologies that analyze a user's voice and movement data and provide suggestions for improvement and evaluation based on the analysis results.

[0011] The "means for displaying feedback to the user" refers to an interface or method that allows the user to visually or audibly confirm the analysis results and the provided feedback.

[0012] "Means for storing evaluation scores" refers to technology that records the generated evaluation scores in a user profile or database so that they can be referenced later.

[0013] "Confession and proposal simulation" is a series of processes in which users recreate and practice situations in which they express their romantic or marriage intentions to a partner in a virtual environment.

[0014] "Specific situations and locations" refer to specific situations (e.g., restaurants, parks) or places where users want to practice, and the VR environment will be built around these.

[0015] "Text conversion" refers to the process of converting audio data into text data, which can then be used for analysis and feedback generation.

[0016] "Server" means a central computer system that receives, processes, stores and provides User Data. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a system that allows a user to simulate a confession or proposal in a virtual reality (VR) environment, analyzes and evaluates the process using artificial intelligence (AI), and provides feedback. Hereinafter, specific embodiments of the present invention will be described in detail.

[0039] System Overview

[0040] User Registration and Login

[0041] A user starts an application and registers a new account. The user enters the required information (e.g., email address, username, password) and sends it to the server. The server creates a user account based on the received information and saves it in the database. A user with an existing account enters their email address and password on the login screen, is authenticated by the server, and logs into the system.

[0042] Setting the VR situation

[0043] After logging in, users can select a "confession" or "proposal" situation from the menu and then specify a specific location (e.g., a park, a restaurant, or their own home). Based on this, the server sends the relevant VR data to the device, which then displays the VR environment to the user.

[0044] Start the conversation simulation

[0045] The user operates an avatar in the VR space and begins to interact with the AI ​​character. The user's voice and actions are captured in real time by the device and sent to the server. The server converts the voice data into text and uses it for analysis along with the movement data.

[0046] AI analysis and feedback

[0047] The server analyzes the received voice and movement data using an AI model. This analysis evaluates the user's language, facial expressions, and behavior. Based on this, the AI ​​generates specific feedback and calculates an evaluation score. Feedback may include advice such as "Speak a little louder" or "Look at the other person's face more."

[0048] View and save feedback

[0049] The generated feedback and evaluation score are sent from the server to the device and displayed to the user. Based on this, the user can check their own performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later. This allows the user to check their progress and use the information to improve their next simulation.

[0050] Specific examples

[0051] For example, if a user wants to simulate a marriage proposal at a restaurant, they first launch the application and log in. After selecting "Proposal" and "Restaurant" from the menu, the device displays the corresponding VR environment. The user puts on VR goggles and controls an avatar in the VR environment to begin a conversation with the AI ​​character. When the user utters the proposal phrase, "Will you marry me?", the device captures the audio and sends it to the server. The server converts the audio into text and analyzes it using an AI model. Based on the results, the AI ​​generates feedback such as "Speak with more confidence," calculates an evaluation score, and presents it to the user. The user can then repeat the simulation based on this feedback to improve their skills.

[0052] Thus, the present invention provides an effective and practical system that allows users to practice confession and proposal situations with a high success rate.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user launches the application and is presented with a new registration or login screen.

[0056] Step 2:

[0057] The user selects new registration and enters the required information (email address, username, password).

[0058] The terminal transmits the input information to the server.

[0059] Step 3:

[0060] The server stores the received information in a database and notifies the terminal that the account creation is complete.

[0061] Step 4:

[0062] The user enters their email address and password on the login screen and clicks "Login."

[0063] The terminal transmits the input information to the server.

[0064] Step 5:

[0065] The server checks the login information, and if authentication is successful, starts a session and notifies the terminal.

[0066] The device displays a menu screen.

[0067] Step 6:

[0068] The user selects a "confession" or "proposal" situation from the menu, and then selects a specific location (e.g., park, restaurant, home).

[0069] Step 7:

[0070] The terminal transmits the selected situation and location data to the server.

[0071] The server sends the relevant VR data back to the device.

[0072] Step 8:

[0073] The device creates a VR environment based on the returned data and displays it to the user, who then puts on the VR goggles and begins the simulation.

[0074] Step 9:

[0075] Users control an avatar in a VR environment and begin interacting with AI characters, with their voices and actions captured in real time.

[0076] Step 10:

[0077] The device converts the captured audio into text in real time and sends it to the server along with the motion data.

[0078] Step 11:

[0079] The server then uses an AI model to analyze the received voice and movement data, including the user's speech, facial expressions, and body movements.

[0080] Step 12:

[0081] The server generates specific feedback based on the analysis results, calculates an evaluation score, and sends the generated feedback and evaluation score back to the device.

[0082] Step 13:

[0083] The device displays the feedback and evaluation score to the user, who can then review the feedback and identify areas for improvement for the next simulation.

[0084] Step 14:

[0085] The server stores the generated evaluation scores and feedback history in a database so that users can refer to them later.

[0086] Step 15:

[0087] Users can then re-run the simulation based on the feedback to improve their skills.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] In modern society, improving communication skills in situations such as confessing your feelings or proposing is important, and there is a growing demand for practice systems for this purpose. However, existing systems lack real-time feedback and have difficulty for users checking their own progress. Furthermore, there are limited means to provide a virtual reality (VR) environment that is close to the real situation and generate detailed feedback, which makes it difficult to practice effectively.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes: a means for a user to register a new account and log in; a means for setting up a VR environment; a means for constructing and displaying a VR environment based on a situation and location selected by the user; a means for capturing the user's voice and movements in real time; a means including an artificial intelligence model for converting the captured voice data into text and analyzing it together with the movement data; a means for generating feedback and an evaluation score based on the analysis results; a means for displaying the generated feedback and evaluation score to the user; and a means for saving the generated feedback and evaluation score. This allows the user to receive real-time feedback in an environment that resembles a real situation, enabling effective practice while checking their progress.

[0093] "User" refers to an individual who uses the system to simulate confessing or proposing in a VR environment.

[0094] "Account" refers to a collection of information (e.g., email address, username, password) that a user registers to access the system.

[0095] "Login" refers to the operation of enabling a user to access the system using registered account information.

[0096] "VR environment" refers to a simulation space created using virtual reality technology.

[0097] "Situation" refers to a specific scene such as a confession or proposal of marriage that the user selects.

[0098] "Location" refers to a specific place where a situation is set (e.g., a park, a restaurant, your home, etc.).

[0099] "Server" refers to the central computing unit that receives user operations and input data, constructs the VR environment, analyzes the data, generates feedback, and so on.

[0100] "Terminal" refers to the device operated by the user (e.g., smartphone, tablet, VR goggles).

[0101] "Voice data" refers to data that captures what the user has said as a voice signal.

[0102] "Motion data" refers to data captured by sensors or other devices that captures the physical movements a user makes within a VR environment.

[0103] "Text conversion" refers to the process of converting audio data into text data.

[0104] "Artificial Intelligence Model" refers to the AI ​​software and algorithms used to analyze received data and generate feedback and / or evaluation scores.

[0105] "Analysis" refers to the process by which an artificial intelligence model evaluates a user's performance based on speech and motion data.

[0106] "Feedback" refers to specific advice or evaluation provided to the user based on the analysis results.

[0107] "Evaluation score" refers to the quantitative representation of a user's performance obtained by quantifying the analysis results.

[0108] "Storage" refers to the act of recording the generated feedback and rating scores in a database for future reference.

[0109] This invention is a system that allows users to simulate confessing or proposing in a virtual reality (VR) environment, and uses artificial intelligence (AI) to analyze and evaluate the process and provide feedback. Hereinafter, specific embodiments of the present invention will be described in detail.

[0110] User Registration and Login

[0111] User operations

[0112] A user starts the application and clicks the "New Registration" button to create an account. The user enters the required information (email address, username, password). Users with existing accounts can access the system by entering their email address and password on the login screen.

[0113] Device behavior

[0114] The terminal receives the user's input information and sends it to the server. In the case of login, the terminal sends the received authentication information to the server.

[0115] Server Operation

[0116] The server receives the new registration information and stores it in a database. It then checks the login information against the database and performs appropriate authentication. It then sends a response to the terminal and directs the user to the menu screen.

[0117] Setting the VR situation

[0118] User operations

[0119] The user selects a "confession" or "proposal" situation from the menu and sets a specific location (e.g., a park, restaurant, home, etc.).

[0120] Device behavior

[0121] The terminal receives the user's selection information and transmits it to the server.

[0122] Server Operation

[0123] Based on the received information, the server generates relevant VR data and sends it to the terminal.

[0124] Terminal display

[0125] The device receives the VR data and displays the virtual environment to the user through the VR goggles.

[0126] Start the conversation simulation

[0127] User operations

[0128] The user controls an avatar in the VR space and begins a conversation with the AI ​​character, for example, proposing marriage by saying, "Will you marry me?"

[0129] Terminal capture

[0130] The device captures the user's voice with a microphone and tracks their movements with sensors, which are then sent to a server in real time.

[0131] AI analysis and feedback

[0132] Server Processing

[0133] The server converts the received voice data into text using voice recognition software (e.g., Google Cloud Speech-to-Text). It also inputs the data along with movement data into an artificial intelligence model (e.g., IBM Watson, OpenAI GPT-4) for analysis. The analysis evaluates the user's language, facial expressions, and behavior.

[0134] Generate analysis results

[0135] The server generates feedback based on the analysis results and calculates an evaluation score that includes specific advice (e.g., "Speak more confidently").

[0136] View and save feedback

[0137] Server Send

[0138] The server sends the generated feedback and evaluation score to the terminal.

[0139] Terminal display

[0140] The device displays the received feedback and evaluation score to the user, who can review it and use it to improve their performance.

[0141] Save Server

[0142] The server stores the generated feedback and rating scores in a database for future reference by the user.

[0143] Specific examples

[0144] For example, if a user simulates a "proposal at a restaurant":

[0145] 1. The user launches the application and logs in.

[0146] 2. Select "Propose" from the menu and set "Restaurant."

[0147] 3. The server sends the VR data to the device, and the device displays the VR environment.

[0148] 4. The user puts on the VR goggles and proposes, "Will you marry me?"

[0149] 5. The device captures the audio and sends it to the server, which converts it into text and analyzes it using an AI model.

[0150] 6. The AI ​​model generates feedback such as "Speak more confidently" and calculates an evaluation score.

[0151] 7. Feedback and evaluation scores are displayed on the device, and the user can use them to re-run the simulation and improve their skills.

[0152] Thus, the present invention provides an effective and practical system that allows users to practice confession and proposal situations with a high success rate.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Processing Steps

[0155] Step 1:

[0156] The user launches the application and clicks the "New Registration" button, providing their email address, username, and password as input.

[0157] The terminal receives this input information and transmits it to the server.

[0158] The server creates a new account based on the received information and stores it in the database. It then sends a confirmation of success to the terminal, which displays a success message to the user.

[0159] Step 2:

[0160] The user enters an email address and password on the login screen.

[0161] The terminal receives the entered login information and sends it to the server.

[0162] The server verifies the received information against the account information stored in the database and, if authentication is successful, sends a response to the terminal to display the menu screen.

[0163] The device displays a menu screen.

[0164] Step 3:

[0165] The user selects "confession" or "proposal" from the menu screen and sets the situation and a specific location (e.g., park, restaurant, home, etc.).

[0166] The terminal receives the selection information and transmits it to the server.

[0167] Step 4:

[0168] The server uses the received information to prepare the corresponding VR data, for example, using 3D modeling software.

[0169] The generated VR data is sent to the terminal.

[0170] The device uses the received VR data to display the VR environment, and the user wears VR goggles to visually and physically experience the virtual environment.

[0171] Step 5:

[0172] The user controls an avatar in the VR space and begins a conversation with the AI ​​character, for example proposing marriage by saying, "Will you marry me?"

[0173] The device captures the user's voice with a microphone and tracks the avatar's movements with sensors, and transmits this data to a server in real time.

[0174] Step 6:

[0175] The server converts the received voice data into text using voice recognition software (e.g., Google Cloud Speech-to-Text), and also prepares the movement data for analysis.

[0176] This data is input into an artificial intelligence model (e.g., IBM Watson, OpenAI GPT-4) and analyzed, focusing on the user's language, facial expressions, and behavior.

[0177] Step 7:

[0178] The server generates specific feedback based on the analysis results of the AI ​​model, such as advice like "Speak with more confidence" and an evaluation score.

[0179] The generated feedback and evaluation score are transmitted to the terminal.

[0180] Step 8:

[0181] The device displays the received feedback and rating score to the user, allowing them to review their performance and understand areas for improvement.

[0182] The server stores the feedback and rating scores in a database for future reference by the user.

[0183] (Application example 1)

[0184] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0185] Improving service quality in customer service is a major challenge in today's industry. Training and skill development for new employees is particularly important, but on-site training has its limitations, making it difficult to find effective methods. Furthermore, there is a lack of systems for providing effective and instant feedback. To address these challenges, there is a demand for new training systems using virtual reality (VR) technology.

[0186] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0187] In this invention, the server includes a means for setting up a VR environment, a means for capturing a user's voice and movements in real time, a means including a generative AI model for analyzing the captured voice and movement data and generating feedback, a means for displaying the feedback to the user, a means for saving the generated feedback and evaluation score, and a means for the user to practice customer service in the VR environment, generate prompt sentences, and analyze the results, thereby enabling practical and effective customer service training in a virtual environment.

[0188] A "VR environment" is an environment that uses virtual reality technology to create and experience virtual spaces and situations that users can immerse themselves in.

[0189] "Capturing user voice and movements" refers to recording the voices uttered by the user and the body movements and gestures made by the user in real time.

[0190] A "generative AI model" is a model that uses artificial intelligence technology to analyze captured data and provide feedback and evaluation.

[0191] "Displaying feedback" refers to providing the generated feedback or evaluation results to the user visually or audibly.

[0192] "Storing evaluation scores" means recording the generated evaluation results in a database or storage device so that they can be referenced later.

[0193] A "prompt" is a textual question or instruction input to an AI model that guides the analysis and generation process.

[0194] "Customer service training" is a practice process for improving the skills and manners required for customer service work, in which the user simulates customer service.

[0195] "Real-time text transcription" is the process of instantly converting voice data into text.

[0196] "Setting the situation and location" means that the user selects the specific scene and location of the simulation and then constructs the VR environment based on that setting.

[0197] User Registration and Login

[0198] First, a user launches an application on a VR-compatible smartphone or head-mounted display (HMD) and creates a new account. The required information (e.g., email address, username, password) is entered and sent to the server. The server uses the received information to create a user account and stores it in a database. Users who already have an account can enter their email address and password on the login screen, receive authentication from the server, and access the system.

[0199] Setting the VR situation

[0200] After logging in, the user selects a simulation such as "Customer Service Training" from the menu. A specific scene (e.g., a store interior or counter) is set, and the server sends the relevant VR data to the device. The device displays the set VR environment to the user, and the user controls an avatar in the VR space.

[0201] Start the conversation simulation

[0202] The user controls an avatar in the VR space and starts a simulated conversation with an AI character. The user's voice and movements are captured in real time by the device and transmitted to the server.

[0203] AI analysis and feedback

[0204] The server converts the received voice data into text and analyzes it along with the behavioral data using a generative AI model. Based on the analysis results, the generative AI model generates feedback and an evaluation score. The feedback includes specific advice (e.g., "Speak with more confidence").

[0205] View and save feedback

[0206] The generated feedback and evaluation score are sent from the server to the device and displayed to the user, allowing the user to check their performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later, allowing the user to continuously improve their skills.

[0207] Program processing

[0208] The system is built using Python programs and open source libraries (SpeechRecognition, Transformers, etc.). Speech data is converted to text using the SpeechRecognition library and analyzed by a generative AI model (e.g., OpenAI's GPT series). The generated feedback and evaluation scores are displayed on the user interface. The results of the user's simulations are stored in a database.

[0209] Specific example explanation

[0210] For example, if a user wants to simulate "in-store customer service training," they launch the application and log in. By selecting "Customer Service Training" and "In-Store" from the menu, the device will display the corresponding VR environment. The user puts on the HMD and controls an avatar in the VR environment to begin a conversation with an AI character. Voice data is captured in the form of "Please tell me more about this product" and sent to the server. The server converts the voice into text and analyzes it with a generative AI model. Based on the results, feedback such as "Please explain more calmly" is generated, and an evaluation score is calculated and presented to the user. The user can then run the simulation again based on this feedback.

[0211] Prompt Sentence Examples

[0212] "Username: testuser

[0213] Scene: Customer service in store

[0214] Dialogue: Tell me more about this product."

[0215] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0216] Step 1:

[0217] User Registration and Login

[0218] A user starts the application on their device and enters their email address, username, and password on the account registration screen. The server receives this input data, creates a user account, and stores it in the database. A user with an existing account enters their email address and password on the login screen, receives authentication from the server, and accesses the system.

[0219] Input: User information (email address, username, password)

[0220] Output: Account creation or system login privileges

[0221] Step 2:

[0222] Setting the VR situation

[0223] After logging in, the user selects a simulation scene, such as "Customer Service Training," from the device's menu. The user sets a specific scene and location (e.g., inside a store or at a counter). The server sends relevant VR data based on the selected scene and location to the device, and the device uses that data to display the VR environment.

[0224] Input: Simulation scene and location selection

[0225] Output: Sending and displaying selected VR data

[0226] Step 3:

[0227] Start the conversation simulation

[0228] The user controls an avatar in the VR environment and begins to interact with the AI ​​character. The device captures the user's voice and actions in real time and sends the data to the server.

[0229] Input: User voice and movement data

[0230] Output: Sending captured data in real time

[0231] Step 4:

[0232] AI-based analysis and feedback generation

[0233] The server converts the received voice data into text and inputs it into a generative AI model along with the motion data. The generative AI model analyzes this data and generates feedback and an evaluation score. This feedback includes specific advice, such as "Speak more confidently."

[0234] Input: Voice data, motion data

[0235] Output: Feedback and rating score

[0236] Step 5:

[0237] View and save feedback

[0238] The generated feedback and evaluation score are sent from the server to the device and displayed to the user. Based on this feedback, the user can check their own performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later.

[0239] Input: Feedback and rating scores

[0240] Output: View and save feedback

[0241] Example prompt sentences:

[0242] "Username: testuser

[0243] Scene: Customer service in store

[0244] Dialogue: Tell me more about this product."

[0245] By following the steps above, a customer service training system can be implemented, which helps users improve their customer service skills in a practical environment.

[0246] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0247] The present invention is a system in which a user simulates confessing or proposing in a virtual reality (VR) environment, analyzes and evaluates the process using artificial intelligence (AI), and provides feedback by combining it with an emotion engine. A detailed description of specific embodiments of the present invention is provided below.

[0248] System Overview

[0249] User Registration and Login

[0250] A user starts an application and registers a new account. The user enters the required information (e.g., email address, username, password) and sends it to the server. The server creates a user account based on the received information and saves it in the database. A user with an existing account enters their email address and password on the login screen, is authenticated by the server, and logs into the system.

[0251] Setting the VR situation

[0252] After logging in, users can select a "confession" or "proposal" situation from the menu and then specify a specific location (e.g., a park, a restaurant, or their own home). Based on this, the server sends the relevant VR data to the device, which then displays the VR environment to the user.

[0253] Start the conversation simulation

[0254] The user operates an avatar in the VR space and begins to interact with the AI ​​character. The user's voice and actions are captured in real time by the device and sent to the server. The server converts this voice data into text and uses it for analysis along with the movement data.

[0255] Analysis and feedback by AI and emotion engine

[0256] The server analyzes the received voice and behavior data using an AI model and emotion engine. The emotion engine analyzes the user's tone of voice, facial expressions, and body movements to estimate their emotions. Based on this, the AI ​​generates feedback and calculates an evaluation score. For example, specific advice such as "It would be more effective if you smiled more" or "Speaking with confidence will help the other person understand you" is generated.

[0257] View and save feedback

[0258] The generated feedback and evaluation score are sent from the server to the device and displayed to the user. Based on this, the user can check their own performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later. This allows the user to check their progress and use the information to improve their next simulation.

[0259] Specific examples

[0260] For example, if a user wants to simulate a marriage proposal at a restaurant, they first launch the application and log in. After selecting "Proposal" and "Restaurant" from the menu, the device displays the corresponding VR environment. The user puts on VR goggles and controls an avatar in the VR environment to begin a conversation with an AI character. When the user utters the proposal phrase, "Will you marry me?", the device captures the audio and sends it to the server. The server converts the audio into text and analyzes it using an emotion engine. Based on the analysis results, feedback such as "Speak with more confidence" or "Try to look closely at the other person's face" is generated and presented to the user along with an evaluation score. The user can then repeat the simulation based on this feedback to improve their skills.

[0261] Thus, the present invention provides an effective and practical system combined with an emotion engine that allows users to practice confession and proposal situations with a high success rate.

[0262] The processing flow will be explained below.

[0263] Step 1:

[0264] The user launches the application and is presented with a new registration or login screen.

[0265] Step 2:

[0266] The user selects new registration and enters the required information (email address, username, password).

[0267] The terminal transmits the input information to the server.

[0268] Step 3:

[0269] The server stores the received information in a database and notifies the terminal that the account creation is complete.

[0270] Step 4:

[0271] The user enters their email address and password on the login screen and clicks "Login."

[0272] The terminal transmits the input information to the server.

[0273] Step 5:

[0274] The server checks the login information, and if authentication is successful, starts a session and notifies the terminal.

[0275] The device displays a menu screen.

[0276] Step 6:

[0277] The user selects a "confession" or "proposal" situation from the menu, and then selects a specific location (e.g., park, restaurant, home).

[0278] Step 7:

[0279] The terminal transmits the selected situation and location data to the server.

[0280] The server sends the relevant VR data back to the device.

[0281] Step 8:

[0282] The device creates a VR environment based on the returned data and displays it to the user, who then puts on the VR goggles and begins the simulation.

[0283] Step 9:

[0284] Users control an avatar in a VR environment and begin interacting with AI characters, with their voices and actions captured in real time.

[0285] Step 10:

[0286] The device converts the captured audio into text in real time and sends it to the server along with the motion data.

[0287] Step 11:

[0288] The server analyzes the received voice and movement data using an AI model and emotion engine, which analyzes the user's tone of voice, facial expressions, and body movements to estimate the user's emotions.

[0289] Step 12:

[0290] The server generates specific feedback based on the analysis results, calculates an evaluation score, and sends the generated feedback and evaluation score back to the device.

[0291] Step 13:

[0292] The device displays the feedback and evaluation score to the user, who can then review the feedback and identify areas for improvement for the next simulation.

[0293] Step 14:

[0294] The server stores the generated evaluation scores and feedback history in a database for users to refer to later. Users can use this stored data to prepare for marriage proposals or confessions.

[0295] Step 15:

[0296] The user then runs the simulation again based on the feedback and implements the suggested improvements. By repeating this process, the user can improve their skills.

[0297] Example 2

[0298] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0299] Conventional VR simulation systems lack the ability to capture and analyze users' voices and movements in real time, making it difficult to accurately analyze users' emotions and subtle nuances. Furthermore, the quality of the feedback is low, making it difficult for users to effectively learn and improve. Therefore, there is a demand for a system that can provide users with high-quality feedback by capturing users' voices and movements in real time and analyzing them using an emotion engine.

[0300] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for setting a VR environment, means for capturing the user's voice and movements in real time, means including an artificial intelligence model for analyzing the captured voice and movement data and generating feedback, means including an emotion engine for analyzing emotions, means for displaying feedback and an evaluation score to the user, and means for saving the generated feedback and evaluation score. This allows the user to analyze their own performance in real time and receive high-quality feedback, thereby enabling them to effectively learn and improve their skills.

[0301] A "VR environment" is a computer-generated simulation space that allows users to experience visual and auditory sensations in a virtual reality space.

[0302] "Means for capturing user voice and movements in real time" refers to devices or software that instantly record a user's voice and body movements as digital data and transmit that information to the system.

[0303] "Means including an artificial intelligence model for analyzing captured voice and movement data and generating feedback" refers to a machine learning algorithm or AI engine for processing and analyzing collected voice and movement data and providing improvement advice to the user based on the results.

[0304] "Emotion engine for analyzing emotions" refers to software and algorithms that analyze a user's tone of voice, facial expressions, body movements, etc. to infer the user's emotional state.

[0305] The "means for displaying feedback and evaluation scores to the user" refers to a device or display screen that allows the user to visually confirm the generated advice and evaluation results.

[0306] "Means for storing generated feedback and evaluation scores" refers to hardware or software that records the analysis results and feedback in a database or storage system for future reference.

[0307] "Means for configuring specific situations and locations in a VR environment" refers to the interface and menu system that allows a user to select and configure the scenarios and locations they experience in virtual reality.

[0308] "Means for creating a VR environment based on a specific situation and location and displaying it to the user" refers to software that generates a virtual reality space based on a scenario or location set by the user and outputs it to a display device so that the user can experience that environment.

[0309] "Means for converting text in real time" refers to speech recognition technology and its implementation for instantly converting a user's voice into text data.

[0310] "Means for transmitting data to a server" refers to a communications protocol and its implementation for transmitting the captured digital data over a network to a server.

[0311] "Means for analyzing and generating feedback and evaluation scores" refers to software and algorithms that analyze the received data using AI models and emotion engines to generate specific advice and evaluation results to provide to users.

[0312] The present invention is a system in which a user simulates confessing or proposing in a virtual reality (VR) environment, analyzes and evaluates the process using artificial intelligence (AI), and provides feedback by combining it with an emotion engine. A detailed description of specific embodiments of the present invention is provided below.

[0313] User Registration and Login

[0314] A user starts an application and accesses the new registration screen. The user enters their email address, username, and password, and the device sends the entered information to the server. The server creates a new user account based on the received information and saves it in the database. An existing user enters their email address and password on the login screen, and the device sends the entered authentication information to the server. The server checks the information in the database and authenticates the user. After successful authentication, the server issues a login session and sends it to the device.

[0315] Setting the VR situation

[0316] After logging in, the user selects a simulation situation such as "confession" or "proposal" from the menu. The user sets a specific location, such as a park, restaurant, or home. The device sends the selected situation information to the server. The server then sends the corresponding VR data (video, audio, 3D environment data, etc.) to the device, which receives it and displays the VR environment that the user has set.

[0317] Start the conversation simulation

[0318] The user puts on VR goggles, enters the virtual reality environment, and controls an avatar. The user begins to interact with the AI ​​character. The device captures the user's voice and actions (body movements and gestures) in real time and transmits them to the server.

[0319] Analysis and feedback by AI and emotion engine

[0320] The server converts the received voice data into text. Specifically, it uses voice recognition software (e.g., Google Cloud Speech-to-Text). The server then passes the converted text and behavioral data to an AI model for analysis. The AI ​​models used here include natural language processing (NLP) models and computer vision models. Based on the analysis results, the emotion engine analyzes the user's tone of voice, facial expressions, body movements, etc. to infer their emotions. The server combines the results of the emotion engine and the AI ​​model to generate feedback and calculate an evaluation score. Examples of generated feedback include "It would be more effective if you smile more" and "Speaking with confidence helps you reach the other person."

[0321] View and save feedback

[0322] The server sends the generated feedback and evaluation score to the terminal and displays it to the user. The user can check the feedback and use it to improve their performance. The feedback history and evaluation score are saved on the server, so the user can refer to them later to check their progress and use them for the next simulation.

[0323] Specific examples

[0324] For example, if a user wants to simulate a marriage proposal at a restaurant, they launch the application and log in. By selecting "Proposal" and "Restaurant" from the menu, the device will display the corresponding VR environment. The user puts on VR goggles and controls an avatar in the VR environment to begin a conversation with an AI character. When the user utters the proposal phrase, "Will you marry me?", the device captures the audio and sends it to the server. The server converts the audio into text and analyzes it using an emotion engine. Based on the analysis results, feedback such as "Speak with more confidence" or "Try to look closely at the other person's face" is generated and presented to the user along with an evaluation score.

[0325] Prompt Sentence Examples

[0326] "Start a simulation of proposing at a restaurant"

[0327] "I want to practice the confession scene in the park."

[0328] The present invention thus provides an effective and practical system, combined with an emotion engine, that allows users to practice confession and proposal situations with a high success rate.

[0329] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0330] Step 1:

[0331] The user launches the application and accesses the new registration screen.

[0332] Input: Enter your email address, username, and password.

[0333] The terminal transmits this input information to the server.

[0334] Output: The user information sent to the server.

[0335] Step 2:

[0336] The server creates a new user account based on the received information and stores it in the database.

[0337] Input: User information sent from the device.

[0338] Output: A confirmation message for creating a new account.

[0339] Step 3:

[0340] An existing user enters their email address and password on the login screen.

[0341] Enter your email address and password.

[0342] The terminal transmits the entered authentication information to the server.

[0343] Output: The authentication information sent to the server.

[0344] Step 4:

[0345] The server authenticates the user by checking the credentials against a database.

[0346] Input: Received credentials.

[0347] Output: Issue of login session.

[0348] Step 5:

[0349] After successful authentication, the server issues a login session and sends it to the terminal.

[0350] Input: The authentication result by the server.

[0351] Output: Login session information.

[0352] Step 6:

[0353] After the user logs in, they select a simulation situation from the menu.

[0354] Input: Simulation selection (confession, proposal, etc.) and location setting (park, restaurant, home, etc.).

[0355] The terminal transmits the selected situation information to the server.

[0356] Output: The situation information sent to the server.

[0357] Step 7:

[0358] The server sends the corresponding VR data (video, audio, 3D environment data, etc.) to the terminal.

[0359] Input: Simulation situation information.

[0360] Output: VR data sent to the device.

[0361] Step 8:

[0362] The device displays the VR environment set for the user based on the VR data received.

[0363] Input: VR data sent from the server.

[0364] Output: The VR environment displayed to the user.

[0365] Step 9:

[0366] The user puts on VR goggles and enters the virtual reality environment to control an avatar.

[0367] Input: User interaction within the VR environment.

[0368] Output: User actions with avatar.

[0369] Step 10:

[0370] The user begins a dialogue with the AI ​​character.

[0371] Input: User speech and movements.

[0372] The device captures the user's voice and movements in real time and transmits them to the server.

[0373] Output: Audio and movement data sent to the server.

[0374] Step 11:

[0375] The server converts the received voice data into text.

[0376] Input: Received audio data.

[0377] Output: Text-converted audio data.

[0378] Step 12:

[0379] The server passes the converted data and behavioral data to the AI ​​model for analysis.

[0380] Input: Textual data, behavioral data.

[0381] Output: Analysis results.

[0382] Step 13:

[0383] Based on the analysis results, the emotion engine analyzes the user's tone of voice, facial expressions, body movements, etc. to estimate their emotions.

[0384] Input: Analysis results, voice tone, facial expressions, body movements.

[0385] Output: Emotion analysis results.

[0386] Step 14:

[0387] The server combines the results of the emotion engine and AI model to generate feedback and calculate an evaluation score.

[0388] Input: Sentiment analysis results, AI model results.

[0389] Output: Feedback and rating score.

[0390] Step 15:

[0391] The server generates feedback and sends the rating score to the device.

[0392] Input: Feedback and rating score.

[0393] Output: Feedback and rating score sent to the device.

[0394] Step 16:

[0395] The device displays the received feedback and rating score to the user.

[0396] Input: Feedback and rating score sent by the server.

[0397] Output: The feedback and rating score displayed to the user.

[0398] Step 17:

[0399] Users can view feedback and use it to improve their performance.

[0400] Input: The displayed feedback and rating score.

[0401] Output: User improvement actions.

[0402] Step 18:

[0403] Feedback history and rating scores are stored on the server.

[0404] Input: Generated feedback and rating scores.

[0405] Output: Feedback history and rating scores stored on the server.

[0406] This allows users to analyze their performance in real time and receive high-quality feedback, enabling them to effectively learn and improve their skills.

[0407] (Application example 2)

[0408] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0409] Improving customer satisfaction in modern brick-and-mortar stores requires employees to have high customer service skills. However, new and inexperienced employees have limited opportunities to practice in real-life customer service situations, and nervousness and mistakes made with real customers often hinder their learning. Furthermore, traditional training methods make it difficult to provide individual feedback quickly, and there is a lack of efficient ways to promote employee skill improvement. To solve these issues, the development of a customer service training system using VR is desired.

[0410] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for setting up a VR environment, means for capturing the user's voice and movements in real time, means including an artificial intelligence model for analyzing the captured voice and movement data and generating feedback, means for displaying the feedback to the user, means for saving the generated feedback and evaluation score, and means for creating a virtual training scenario for employees of a physical store to practice their customer service skills. This allows employees to effectively practice in a virtual reality customer service scenario and improve their skills while receiving specific feedback.

[0411] A "means for setting up a VR environment" is a device or program that creates a virtual reality space and provides an environment for users to experience a specific situation or location.

[0412] "Means for capturing user voice and movements in real time" refers to a device or program that instantly records the user's voice and body movements and provides the data for subsequent analysis.

[0413] "Means including an artificial intelligence model for analyzing captured voice and movement data and generating feedback" refers to a system that uses artificial intelligence to analyze recorded voice and movement data and provides the user with guidelines for improvement based on the results.

[0414] The "means for displaying feedback to the user" refers to a device or program that presents the analysis results to the user in an appropriate form and provides information that allows the user to check and improve their own performance.

[0415] The "means for storing the generated feedback and evaluation score" refers to a device or program that records the feedback provided to the user and the evaluation results, and stores them in a form that can be referenced later.

[0416] "A means for creating virtual training scenarios for brick-and-mortar store employees to practice customer service skills" is a system that recreates customer service situations in brick-and-mortar stores in virtual reality, and designs and provides scenarios to train employees to respond appropriately in the moment.

[0417] "Means for selecting situations and locations for practicing conversations and actions in customer service situations in a physical store" refers to a device or program that allows the user to select a specific customer service situation or environment and set up an appropriate practice scenario based on that selection.

[0418] The "means for generating specific advice as feedback to improve the user's customer service skills" is a system that provides specific guidelines on how the user should improve their customer service based on analyzed data.

[0419] This invention relates to a system that uses virtual reality (VR) to train employees in customer service skills in brick-and-mortar stores. Hereinafter, specific embodiments of the invention will be described in detail.

[0420] System configuration

[0421] The system mainly consists of the following elements:

[0422] 1. How to set up a VR environment

[0423] The server uses Unity to create a virtual reality space, providing an environment in which users can practice customer service in specific situations and locations.

[0424] 2. A means of capturing the user's voice and actions in real time

[0425] It uses the camera and microphone built into the user's smartphone or VR goggles to instantly record voice and movement.

[0426] 3. Means including an artificial intelligence model for analyzing the captured audio and motion data and generating feedback.

[0427] The server uses artificial intelligence models trained in TensorFlow and OpenCV to analyze the recorded voice and movement data and generate feedback for the user.

[0428] 4. How to display feedback to the user

[0429] Feedback is displayed on the smartphone or VR goggles screen using React Native, allowing users to see the analysis results in real time.

[0430] 5. Means of storing generated feedback and rating scores

[0431] The feedback and evaluation scores are stored on a server and made available to users for future reference, allowing them to track their progress and understand areas for improvement.

[0432] 6. A way to create virtual training scenarios for brick-and-mortar store associates to practice customer service skills

[0433] The server provides scenarios that recreate specific customer service situations in a real store (e.g., greeting customers at a restaurant), allowing users to practice in situations that are close to the real situation.

[0434] System Operation

[0435] 1. Setting up the VR environment

[0436] The server uses Unity to create a virtual environment based on the situation and location selected by the user, allowing the user to begin a customer service simulation in virtual reality.

[0437] 2. Audio and motion capture

[0438] The camera and microphone built into the user's smartphone or VR goggles capture the user's voice and movements in real time and transmit them to a server.

[0439] 3. Analyzing data and generating feedback

[0440] The server analyzes the captured voice and motion data using TensorFlow and OpenCV to evaluate the user's emotions and performance, and generates specific feedback for the user based on the analysis results.

[0441] 4. Viewing Feedback

[0442] The generated feedback and rating scores are displayed on the user's smartphone or VR goggles using React Native, allowing the user to improve their performance.

[0443] 5. Data storage

[0444] Feedback and evaluation scores are stored on a server for future reference, allowing users to track their progress and continually improve their skills.

[0445] Examples of concrete examples and prompts

[0446] For example, if a user wants to simulate "greeting customers at a restaurant," the following prompt sentences are exemplified:

[0447] "Start a simulation of greeting a customer in a VR environment. The scenario is 'restaurant'."

[0448] "Hello, welcome! My name is [Name]. I'll show you to your seat."

[0449] This allows users to practice in situations that are similar to actual customer service situations and improve their skills while receiving specific feedback.

[0450] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0451] Step 1:

[0452] A user launches an application and enters their email address and password on the login screen. The input data is sent to the server, which then performs the authentication process. If authentication is successful, the user is redirected to the main menu. The input is an email address and password, and the output is the authentication result (success or failure).

[0453] Step 2:

[0454] The user selects "Customer Service Training" from the main menu. Then, the training situation and location (e.g., greeting customers at a restaurant) are set. The set situation information is sent to the server. The set situation information is the input, and the setting information is sent to the server as the output.

[0455] Step 3:

[0456] The server uses Unity to create a virtual reality (VR) environment based on the situation and location selected by the user. The created VR environment data is sent to the user's device. The input is configuration information, and the output is VR environment data that is generated and sent to the user's device.

[0457] Step 4:

[0458] The user puts on the VR goggles and starts the simulation in the VR environment. The user's movements and voice are captured in real time and sent from the device to the server. The input is the user's movements and voice, and the captured data is sent to the server as the output.

[0459] Step 5:

[0460] The server uses TensorFlow to analyze the captured voice and motion data, which generates emotion and performance data for the user. The input is the captured data, and the output is the emotion and performance data.

[0461] Step 6:

[0462] The server generates specific feedback based on the analysis results. The generated feedback and evaluation score are sent to the device. The input is emotion and performance data, and the output is feedback and evaluation score.

[0463] Step 7:

[0464] The device uses React Native to display feedback and a rating score to the user, allowing the user to see their performance and understand areas for improvement. The input is the feedback and rating score, and the output is what is displayed to the user.

[0465] Step 8:

[0466] The generated feedback and rating score are stored on the server for future reference. The input is the feedback and rating score, and the output is stored in a database.

[0467] Step 9:

[0468] Users can then replay the simulation and improve their customer service skills based on the feedback. Users can practice repeatedly at their own pace and monitor their progress while honing their skills. The input is the previous feedback and evaluation score, and the output is improved performance.

[0469] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0470] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0471] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0472] [Second embodiment]

[0473] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0474] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0475] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0476] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0477] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0478] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0479] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0480] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0481] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0482] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0483] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0484] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0485] The present invention provides a system that allows a user to simulate a confession or proposal in a virtual reality (VR) environment, analyzes and evaluates the process using artificial intelligence (AI), and provides feedback. Hereinafter, specific embodiments of the present invention will be described in detail.

[0486] System Overview

[0487] User Registration and Login

[0488] A user starts an application and registers a new account. The user enters the required information (e.g., email address, username, password) and sends it to the server. The server creates a user account based on the received information and saves it in the database. A user with an existing account enters their email address and password on the login screen, is authenticated by the server, and logs into the system.

[0489] Setting the VR situation

[0490] After logging in, users can select a "confession" or "proposal" situation from the menu and then specify a specific location (e.g., a park, a restaurant, or their own home). Based on this, the server sends the relevant VR data to the device, which then displays the VR environment to the user.

[0491] Start the conversation simulation

[0492] The user operates an avatar in the VR space and begins to interact with the AI ​​character. The user's voice and actions are captured in real time by the device and sent to the server. The server converts the voice data into text and uses it for analysis along with the movement data.

[0493] AI analysis and feedback

[0494] The server analyzes the received voice and movement data using an AI model. This analysis evaluates the user's language, facial expressions, and behavior. Based on this, the AI ​​generates specific feedback and calculates an evaluation score. Feedback may include advice such as "Speak a little louder" or "Look at the other person's face more."

[0495] View and save feedback

[0496] The generated feedback and evaluation score are sent from the server to the device and displayed to the user. Based on this, the user can check their own performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later. This allows the user to check their progress and use the information to improve their next simulation.

[0497] Specific examples

[0498] For example, if a user wants to simulate a marriage proposal at a restaurant, they first launch the application and log in. After selecting "Proposal" and "Restaurant" from the menu, the device displays the corresponding VR environment. The user puts on VR goggles and controls an avatar in the VR environment to begin a conversation with the AI ​​character. When the user utters the proposal phrase, "Will you marry me?", the device captures the audio and sends it to the server. The server converts the audio into text and analyzes it using an AI model. Based on the results, the AI ​​generates feedback such as "Speak with more confidence," calculates an evaluation score, and presents it to the user. The user can then repeat the simulation based on this feedback to improve their skills.

[0499] Thus, the present invention provides an effective and practical system that allows users to practice confession and proposal situations with a high success rate.

[0500] The processing flow will be explained below.

[0501] Step 1:

[0502] The user launches the application and is presented with a new registration or login screen.

[0503] Step 2:

[0504] The user selects new registration and enters the required information (email address, username, password).

[0505] The terminal transmits the input information to the server.

[0506] Step 3:

[0507] The server stores the received information in a database and notifies the terminal that the account creation is complete.

[0508] Step 4:

[0509] The user enters their email address and password on the login screen and clicks "Login."

[0510] The terminal transmits the input information to the server.

[0511] Step 5:

[0512] The server checks the login information, and if authentication is successful, starts a session and notifies the terminal.

[0513] The device displays a menu screen.

[0514] Step 6:

[0515] The user selects a "confession" or "proposal" situation from the menu, and then selects a specific location (e.g., park, restaurant, home).

[0516] Step 7:

[0517] The terminal transmits the selected situation and location data to the server.

[0518] The server sends the relevant VR data back to the device.

[0519] Step 8:

[0520] The device creates a VR environment based on the returned data and displays it to the user, who then puts on the VR goggles and begins the simulation.

[0521] Step 9:

[0522] Users control an avatar in a VR environment and begin interacting with AI characters, with their voices and actions captured in real time.

[0523] Step 10:

[0524] The device converts the captured audio into text in real time and sends it to the server along with the motion data.

[0525] Step 11:

[0526] The server then uses an AI model to analyze the received voice and movement data, including the user's speech, facial expressions, and body movements.

[0527] Step 12:

[0528] The server generates specific feedback based on the analysis results, calculates an evaluation score, and sends the generated feedback and evaluation score back to the device.

[0529] Step 13:

[0530] The device displays the feedback and evaluation score to the user, who can then review the feedback and identify areas for improvement for the next simulation.

[0531] Step 14:

[0532] The server stores the generated evaluation scores and feedback history in a database so that users can refer to them later.

[0533] Step 15:

[0534] Users can then re-run the simulation based on the feedback to improve their skills.

[0535] Example 1

[0536] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0537] In modern society, improving communication skills in situations such as confessing your feelings or proposing is important, and there is a growing demand for practice systems for this purpose. However, existing systems lack real-time feedback and have difficulty for users checking their own progress. Furthermore, there are limited means to provide a virtual reality (VR) environment that is close to the real situation and generate detailed feedback, which makes it difficult to practice effectively.

[0538] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0539] In this invention, the server includes: a means for a user to register a new account and log in; a means for setting up a VR environment; a means for constructing and displaying a VR environment based on a situation and location selected by the user; a means for capturing the user's voice and movements in real time; a means including an artificial intelligence model for converting the captured voice data into text and analyzing it together with the movement data; a means for generating feedback and an evaluation score based on the analysis results; a means for displaying the generated feedback and evaluation score to the user; and a means for saving the generated feedback and evaluation score. This allows the user to receive real-time feedback in an environment that resembles a real situation, enabling effective practice while checking their progress.

[0540] "User" refers to an individual who uses the system to simulate confessing or proposing in a VR environment.

[0541] "Account" refers to a collection of information (e.g., email address, username, password) that a user registers to access the system.

[0542] "Login" refers to the operation of enabling a user to access the system using registered account information.

[0543] "VR environment" refers to a simulation space created using virtual reality technology.

[0544] "Situation" refers to a specific scene such as a confession or proposal of marriage that the user selects.

[0545] "Location" refers to a specific place where a situation is set (e.g., a park, a restaurant, your home, etc.).

[0546] "Server" refers to the central computing unit that receives user operations and input data, constructs the VR environment, analyzes the data, generates feedback, and so on.

[0547] "Terminal" refers to the device operated by the user (e.g., smartphone, tablet, VR goggles).

[0548] "Voice data" refers to data that captures what the user has said as a voice signal.

[0549] "Motion data" refers to data captured by sensors or other devices that captures the physical movements a user makes within a VR environment.

[0550] "Text conversion" refers to the process of converting audio data into text data.

[0551] "Artificial Intelligence Model" refers to the AI ​​software and algorithms used to analyze received data and generate feedback and / or evaluation scores.

[0552] "Analysis" refers to the process by which an artificial intelligence model evaluates a user's performance based on speech and motion data.

[0553] "Feedback" refers to specific advice or evaluation provided to the user based on the analysis results.

[0554] "Evaluation score" refers to the quantitative representation of a user's performance obtained by quantifying the analysis results.

[0555] "Storage" refers to the act of recording the generated feedback and rating scores in a database for future reference.

[0556] This invention is a system that allows users to simulate confessing or proposing in a virtual reality (VR) environment, and uses artificial intelligence (AI) to analyze and evaluate the process and provide feedback. Hereinafter, specific embodiments of the present invention will be described in detail.

[0557] User Registration and Login

[0558] User operations

[0559] A user starts the application and clicks the "New Registration" button to create an account. The user enters the required information (email address, username, password). Users with existing accounts can access the system by entering their email address and password on the login screen.

[0560] Device behavior

[0561] The terminal receives the user's input information and sends it to the server. In the case of login, the terminal sends the received authentication information to the server.

[0562] Server Operation

[0563] The server receives the new registration information and stores it in a database. It then checks the login information against the database and performs appropriate authentication. It then sends a response to the terminal and directs the user to the menu screen.

[0564] Setting the VR situation

[0565] User operations

[0566] The user selects a "confession" or "proposal" situation from the menu and sets a specific location (e.g., a park, restaurant, home, etc.).

[0567] Device behavior

[0568] The terminal receives the user's selection information and transmits it to the server.

[0569] Server Operation

[0570] Based on the received information, the server generates relevant VR data and sends it to the terminal.

[0571] Terminal display

[0572] The device receives the VR data and displays the virtual environment to the user through the VR goggles.

[0573] Start the conversation simulation

[0574] User operations

[0575] The user controls an avatar in the VR space and begins a conversation with the AI ​​character, for example, proposing marriage by saying, "Will you marry me?"

[0576] Terminal capture

[0577] The device captures the user's voice with a microphone and tracks their movements with sensors, which are then sent to a server in real time.

[0578] AI analysis and feedback

[0579] Server Processing

[0580] The server converts the received voice data into text using voice recognition software (e.g., Google Cloud Speech-to-Text). It also inputs the data along with movement data into an artificial intelligence model (e.g., IBM Watson, OpenAI GPT-4) for analysis. The analysis evaluates the user's language, facial expressions, and behavior.

[0581] Generate analysis results

[0582] The server generates feedback based on the analysis results and calculates an evaluation score that includes specific advice (e.g., "Speak more confidently").

[0583] View and save feedback

[0584] Server Send

[0585] The server sends the generated feedback and evaluation score to the terminal.

[0586] Terminal display

[0587] The device displays the received feedback and evaluation score to the user, who can review it and use it to improve their performance.

[0588] Save Server

[0589] The server stores the generated feedback and rating scores in a database for future reference by the user.

[0590] Specific examples

[0591] For example, if a user simulates a "proposal at a restaurant":

[0592] 1. The user launches the application and logs in.

[0593] 2. Select "Propose" from the menu and set "Restaurant."

[0594] 3. The server sends the VR data to the device, and the device displays the VR environment.

[0595] 4. The user puts on the VR goggles and proposes, "Will you marry me?"

[0596] 5. The device captures the audio and sends it to the server, which converts it into text and analyzes it using an AI model.

[0597] 6. The AI ​​model generates feedback such as "Speak more confidently" and calculates an evaluation score.

[0598] 7. Feedback and evaluation scores are displayed on the device, and the user can use them to re-run the simulation and improve their skills.

[0599] Thus, the present invention provides an effective and practical system that allows users to practice confession and proposal situations with a high success rate.

[0600] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0601] Processing Steps

[0602] Step 1:

[0603] The user launches the application and clicks the "New Registration" button, providing their email address, username, and password as input.

[0604] The terminal receives this input information and transmits it to the server.

[0605] The server creates a new account based on the received information and stores it in the database. It then sends a confirmation of success to the terminal, which displays a success message to the user.

[0606] Step 2:

[0607] The user enters an email address and password on the login screen.

[0608] The terminal receives the entered login information and sends it to the server.

[0609] The server verifies the received information against the account information stored in the database and, if authentication is successful, sends a response to the terminal to display the menu screen.

[0610] The device displays a menu screen.

[0611] Step 3:

[0612] The user selects "confession" or "proposal" from the menu screen and sets the situation and a specific location (e.g., park, restaurant, home, etc.).

[0613] The terminal receives the selection information and transmits it to the server.

[0614] Step 4:

[0615] The server uses the received information to prepare the corresponding VR data, for example, using 3D modeling software.

[0616] The generated VR data is sent to the terminal.

[0617] The device uses the received VR data to display the VR environment, and the user wears VR goggles to visually and physically experience the virtual environment.

[0618] Step 5:

[0619] The user controls an avatar in the VR space and begins a conversation with the AI ​​character, for example proposing marriage by saying, "Will you marry me?"

[0620] The device captures the user's voice with a microphone and tracks the avatar's movements with sensors, and transmits this data to a server in real time.

[0621] Step 6:

[0622] The server converts the received voice data into text using voice recognition software (e.g., Google Cloud Speech-to-Text), and also prepares the movement data for analysis.

[0623] This data is input into an artificial intelligence model (e.g., IBM Watson, OpenAI GPT-4) and analyzed, focusing on the user's language, facial expressions, and behavior.

[0624] Step 7:

[0625] The server generates specific feedback based on the analysis results of the AI ​​model, such as advice like "Speak with more confidence" and an evaluation score.

[0626] The generated feedback and evaluation score are transmitted to the terminal.

[0627] Step 8:

[0628] The device displays the received feedback and rating score to the user, allowing them to review their performance and understand areas for improvement.

[0629] The server stores the feedback and rating scores in a database for future reference by the user.

[0630] (Application example 1)

[0631] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0632] Improving service quality in customer service is a major challenge in today's industry. Training and skill development for new employees is particularly important, but on-site training has its limitations, making it difficult to find effective methods. Furthermore, there is a lack of systems for providing effective and instant feedback. To address these challenges, there is a demand for new training systems using virtual reality (VR) technology.

[0633] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0634] In this invention, the server includes a means for setting up a VR environment, a means for capturing a user's voice and movements in real time, a means including a generative AI model for analyzing the captured voice and movement data and generating feedback, a means for displaying the feedback to the user, a means for saving the generated feedback and evaluation score, and a means for the user to practice customer service in the VR environment, generate prompt sentences, and analyze the results, thereby enabling practical and effective customer service training in a virtual environment.

[0635] A "VR environment" is an environment that uses virtual reality technology to create and experience virtual spaces and situations that users can immerse themselves in.

[0636] "Capturing user voice and movements" refers to recording the voices uttered by the user and the body movements and gestures made by the user in real time.

[0637] A "generative AI model" is a model that uses artificial intelligence technology to analyze captured data and provide feedback and evaluation.

[0638] "Displaying feedback" refers to providing the generated feedback or evaluation results to the user visually or audibly.

[0639] "Storing evaluation scores" means recording the generated evaluation results in a database or storage device so that they can be referenced later.

[0640] A "prompt" is a textual question or instruction input to an AI model that guides the analysis and generation process.

[0641] "Customer service training" is a practice process for improving the skills and manners required for customer service work, in which the user simulates customer service.

[0642] "Real-time text transcription" is the process of instantly converting voice data into text.

[0643] "Setting the situation and location" means that the user selects the specific scene and location of the simulation and then constructs the VR environment based on that setting.

[0644] User Registration and Login

[0645] First, a user launches an application on a VR-compatible smartphone or head-mounted display (HMD) and creates a new account. The required information (e.g., email address, username, password) is entered and sent to the server. The server uses the received information to create a user account and stores it in a database. Users who already have an account can enter their email address and password on the login screen, receive authentication from the server, and access the system.

[0646] Setting the VR situation

[0647] After logging in, the user selects a simulation such as "Customer Service Training" from the menu. A specific scene (e.g., a store interior or counter) is set, and the server sends the relevant VR data to the device. The device displays the set VR environment to the user, and the user controls an avatar in the VR space.

[0648] Start the conversation simulation

[0649] The user controls an avatar in the VR space and starts a simulated conversation with an AI character. The user's voice and movements are captured in real time by the device and transmitted to the server.

[0650] AI analysis and feedback

[0651] The server converts the received voice data into text and analyzes it along with the behavioral data using a generative AI model. Based on the analysis results, the generative AI model generates feedback and an evaluation score. The feedback includes specific advice (e.g., "Speak with more confidence").

[0652] View and save feedback

[0653] The generated feedback and evaluation score are sent from the server to the device and displayed to the user, allowing the user to check their performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later, allowing the user to continuously improve their skills.

[0654] Program processing

[0655] The system is built using Python programs and open source libraries (SpeechRecognition, Transformers, etc.). Speech data is converted to text using the SpeechRecognition library and analyzed by a generative AI model (e.g., OpenAI's GPT series). The generated feedback and evaluation scores are displayed on the user interface. The results of the user's simulations are stored in a database.

[0656] Specific example explanation

[0657] For example, if a user wants to simulate "in-store customer service training," they launch the application and log in. By selecting "Customer Service Training" and "In-Store" from the menu, the device will display the corresponding VR environment. The user puts on the HMD and controls an avatar in the VR environment to begin a conversation with an AI character. Voice data is captured in the form of "Please tell me more about this product" and sent to the server. The server converts the voice into text and analyzes it with a generative AI model. Based on the results, feedback such as "Please explain more calmly" is generated, and an evaluation score is calculated and presented to the user. The user can then run the simulation again based on this feedback.

[0658] Prompt Sentence Examples

[0659] "Username: testuser

[0660] Scene: Customer service in store

[0661] Dialogue: Tell me more about this product."

[0662] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0663] Step 1:

[0664] User Registration and Login

[0665] A user starts the application on their device and enters their email address, username, and password on the account registration screen. The server receives this input data, creates a user account, and stores it in the database. A user with an existing account enters their email address and password on the login screen, receives authentication from the server, and accesses the system.

[0666] Input: User information (email address, username, password)

[0667] Output: Account creation or system login privileges

[0668] Step 2:

[0669] Setting the VR situation

[0670] After logging in, the user selects a simulation scene, such as "Customer Service Training," from the device's menu. The user sets a specific scene and location (e.g., inside a store or at a counter). The server sends relevant VR data based on the selected scene and location to the device, and the device uses that data to display the VR environment.

[0671] Input: Simulation scene and location selection

[0672] Output: Sending and displaying selected VR data

[0673] Step 3:

[0674] Start the conversation simulation

[0675] The user controls an avatar in the VR environment and begins to interact with the AI ​​character. The device captures the user's voice and actions in real time and sends the data to the server.

[0676] Input: User voice and movement data

[0677] Output: Sending captured data in real time

[0678] Step 4:

[0679] AI-based analysis and feedback generation

[0680] The server converts the received voice data into text and inputs it into a generative AI model along with the motion data. The generative AI model analyzes this data and generates feedback and an evaluation score. This feedback includes specific advice, such as "Speak more confidently."

[0681] Input: Voice data, motion data

[0682] Output: Feedback and rating score

[0683] Step 5:

[0684] View and save feedback

[0685] The generated feedback and evaluation score are sent from the server to the device and displayed to the user. Based on this feedback, the user can check their own performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later.

[0686] Input: Feedback and rating scores

[0687] Output: View and save feedback

[0688] Example prompt sentences:

[0689] "Username: testuser

[0690] Scene: Customer service in store

[0691] Dialogue: Tell me more about this product."

[0692] By following the steps above, a customer service training system can be implemented, which helps users improve their customer service skills in a practical environment.

[0693] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0694] The present invention is a system in which a user simulates confessing or proposing in a virtual reality (VR) environment, analyzes and evaluates the process using artificial intelligence (AI), and provides feedback by combining it with an emotion engine. A detailed description of specific embodiments of the present invention is provided below.

[0695] System Overview

[0696] User Registration and Login

[0697] A user starts an application and registers a new account. The user enters the required information (e.g., email address, username, password) and sends it to the server. The server creates a user account based on the received information and saves it in the database. A user with an existing account enters their email address and password on the login screen, is authenticated by the server, and logs into the system.

[0698] Setting the VR situation

[0699] After logging in, users can select a "confession" or "proposal" situation from the menu and then specify a specific location (e.g., a park, a restaurant, or their own home). Based on this, the server sends the relevant VR data to the device, which then displays the VR environment to the user.

[0700] Start the conversation simulation

[0701] The user operates an avatar in the VR space and begins to interact with the AI ​​character. The user's voice and actions are captured in real time by the device and sent to the server. The server converts this voice data into text and uses it for analysis along with the movement data.

[0702] Analysis and feedback by AI and emotion engine

[0703] The server analyzes the received voice and behavior data using an AI model and emotion engine. The emotion engine analyzes the user's tone of voice, facial expressions, and body movements to estimate their emotions. Based on this, the AI ​​generates feedback and calculates an evaluation score. For example, specific advice such as "It would be more effective if you smiled more" or "Speaking with confidence will help the other person understand you" is generated.

[0704] View and save feedback

[0705] The generated feedback and evaluation score are sent from the server to the device and displayed to the user. Based on this, the user can check their own performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later. This allows the user to check their progress and use the information to improve their next simulation.

[0706] Specific examples

[0707] For example, if a user wants to simulate a marriage proposal at a restaurant, they first launch the application and log in. After selecting "Proposal" and "Restaurant" from the menu, the device displays the corresponding VR environment. The user puts on VR goggles and controls an avatar in the VR environment to begin a conversation with an AI character. When the user utters the proposal phrase, "Will you marry me?", the device captures the audio and sends it to the server. The server converts the audio into text and analyzes it using an emotion engine. Based on the analysis results, feedback such as "Speak with more confidence" or "Try to look closely at the other person's face" is generated and presented to the user along with an evaluation score. The user can then repeat the simulation based on this feedback to improve their skills.

[0708] Thus, the present invention provides an effective and practical system combined with an emotion engine that allows users to practice confession and proposal situations with a high success rate.

[0709] The processing flow will be explained below.

[0710] Step 1:

[0711] The user launches the application and is presented with a new registration or login screen.

[0712] Step 2:

[0713] The user selects new registration and enters the required information (email address, username, password).

[0714] The terminal transmits the input information to the server.

[0715] Step 3:

[0716] The server stores the received information in a database and notifies the terminal that the account creation is complete.

[0717] Step 4:

[0718] The user enters their email address and password on the login screen and clicks "Login."

[0719] The terminal transmits the input information to the server.

[0720] Step 5:

[0721] The server checks the login information, and if authentication is successful, starts a session and notifies the terminal.

[0722] The device displays a menu screen.

[0723] Step 6:

[0724] The user selects a "confession" or "proposal" situation from the menu, and then selects a specific location (e.g., park, restaurant, home).

[0725] Step 7:

[0726] The terminal transmits the selected situation and location data to the server.

[0727] The server sends the relevant VR data back to the device.

[0728] Step 8:

[0729] The device creates a VR environment based on the returned data and displays it to the user, who then puts on the VR goggles and begins the simulation.

[0730] Step 9:

[0731] Users control an avatar in a VR environment and begin interacting with AI characters, with their voices and actions captured in real time.

[0732] Step 10:

[0733] The device converts the captured audio into text in real time and sends it to the server along with the motion data.

[0734] Step 11:

[0735] The server analyzes the received voice and movement data using an AI model and emotion engine, which analyzes the user's tone of voice, facial expressions, and body movements to estimate the user's emotions.

[0736] Step 12:

[0737] The server generates specific feedback based on the analysis results, calculates an evaluation score, and sends the generated feedback and evaluation score back to the device.

[0738] Step 13:

[0739] The device displays the feedback and evaluation score to the user, who can then review the feedback and identify areas for improvement for the next simulation.

[0740] Step 14:

[0741] The server stores the generated evaluation scores and feedback history in a database for users to refer to later. Users can use this stored data to prepare for marriage proposals or confessions.

[0742] Step 15:

[0743] The user then runs the simulation again based on the feedback and implements the suggested improvements. By repeating this process, the user can improve their skills.

[0744] Example 2

[0745] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0746] Conventional VR simulation systems lack the ability to capture and analyze users' voices and movements in real time, making it difficult to accurately analyze users' emotions and subtle nuances. Furthermore, the quality of the feedback is low, making it difficult for users to effectively learn and improve. Therefore, there is a demand for a system that can provide users with high-quality feedback by capturing users' voices and movements in real time and analyzing them using an emotion engine.

[0747] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for setting a VR environment, means for capturing the user's voice and movements in real time, means including an artificial intelligence model for analyzing the captured voice and movement data and generating feedback, means including an emotion engine for analyzing emotions, means for displaying feedback and an evaluation score to the user, and means for saving the generated feedback and evaluation score. This allows the user to analyze their own performance in real time and receive high-quality feedback, thereby enabling them to effectively learn and improve their skills.

[0748] A "VR environment" is a computer-generated simulation space that allows users to experience visual and auditory sensations in a virtual reality space.

[0749] "Means for capturing user voice and movements in real time" refers to devices or software that instantly record a user's voice and body movements as digital data and transmit that information to the system.

[0750] "Means including an artificial intelligence model for analyzing captured voice and movement data and generating feedback" refers to a machine learning algorithm or AI engine for processing and analyzing collected voice and movement data and providing improvement advice to the user based on the results.

[0751] "Emotion engine for analyzing emotions" refers to software and algorithms that analyze a user's tone of voice, facial expressions, body movements, etc. to infer the user's emotional state.

[0752] The "means for displaying feedback and evaluation scores to the user" refers to a device or display screen that allows the user to visually confirm the generated advice and evaluation results.

[0753] "Means for storing generated feedback and evaluation scores" refers to hardware or software that records the analysis results and feedback in a database or storage system for future reference.

[0754] "Means for configuring specific situations and locations in a VR environment" refers to the interface and menu system that allows a user to select and configure the scenarios and locations they experience in virtual reality.

[0755] "Means for creating a VR environment based on a specific situation and location and displaying it to the user" refers to software that generates a virtual reality space based on a scenario or location set by the user and outputs it to a display device so that the user can experience that environment.

[0756] "Means for converting text in real time" refers to speech recognition technology and its implementation for instantly converting a user's voice into text data.

[0757] "Means for transmitting data to a server" refers to a communications protocol and its implementation for transmitting the captured digital data over a network to a server.

[0758] "Means for analyzing and generating feedback and evaluation scores" refers to software and algorithms that analyze the received data using AI models and emotion engines to generate specific advice and evaluation results to provide to users.

[0759] The present invention is a system in which a user simulates confessing or proposing in a virtual reality (VR) environment, analyzes and evaluates the process using artificial intelligence (AI), and provides feedback by combining it with an emotion engine. A detailed description of specific embodiments of the present invention is provided below.

[0760] User Registration and Login

[0761] A user starts an application and accesses the new registration screen. The user enters their email address, username, and password, and the device sends the entered information to the server. The server creates a new user account based on the received information and saves it in the database. An existing user enters their email address and password on the login screen, and the device sends the entered authentication information to the server. The server checks the information in the database and authenticates the user. After successful authentication, the server issues a login session and sends it to the device.

[0762] Setting the VR situation

[0763] After logging in, the user selects a simulation situation such as "confession" or "proposal" from the menu. The user sets a specific location, such as a park, restaurant, or home. The device sends the selected situation information to the server. The server then sends the corresponding VR data (video, audio, 3D environment data, etc.) to the device, which receives it and displays the VR environment that the user has set.

[0764] Start the conversation simulation

[0765] The user puts on VR goggles, enters the virtual reality environment, and controls an avatar. The user begins to interact with the AI ​​character. The device captures the user's voice and actions (body movements and gestures) in real time and transmits them to the server.

[0766] Analysis and feedback by AI and emotion engine

[0767] The server converts the received voice data into text. Specifically, it uses voice recognition software (e.g., Google Cloud Speech-to-Text). The server then passes the converted text and behavioral data to an AI model for analysis. The AI ​​models used here include natural language processing (NLP) models and computer vision models. Based on the analysis results, the emotion engine analyzes the user's tone of voice, facial expressions, body movements, etc. to infer their emotions. The server combines the results of the emotion engine and the AI ​​model to generate feedback and calculate an evaluation score. Examples of generated feedback include "It would be more effective if you smile more" and "Speaking with confidence helps you reach the other person."

[0768] View and save feedback

[0769] The server sends the generated feedback and evaluation score to the terminal and displays it to the user. The user can check the feedback and use it to improve their performance. The feedback history and evaluation score are saved on the server, so the user can refer to them later to check their progress and use them for the next simulation.

[0770] Specific examples

[0771] For example, if a user wants to simulate a marriage proposal at a restaurant, they launch the application and log in. By selecting "Proposal" and "Restaurant" from the menu, the device will display the corresponding VR environment. The user puts on VR goggles and controls an avatar in the VR environment to begin a conversation with an AI character. When the user utters the proposal phrase, "Will you marry me?", the device captures the audio and sends it to the server. The server converts the audio into text and analyzes it using an emotion engine. Based on the analysis results, feedback such as "Speak with more confidence" or "Try to look closely at the other person's face" is generated and presented to the user along with an evaluation score.

[0772] Prompt Sentence Examples

[0773] "Start a simulation of proposing at a restaurant"

[0774] "I want to practice the confession scene in the park."

[0775] The present invention thus provides an effective and practical system, combined with an emotion engine, that allows users to practice confession and proposal situations with a high success rate.

[0776] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0777] Step 1:

[0778] The user launches the application and accesses the new registration screen.

[0779] Input: Enter your email address, username, and password.

[0780] The terminal transmits this input information to the server.

[0781] Output: The user information sent to the server.

[0782] Step 2:

[0783] The server creates a new user account based on the received information and stores it in the database.

[0784] Input: User information sent from the device.

[0785] Output: A confirmation message for creating a new account.

[0786] Step 3:

[0787] An existing user enters their email address and password on the login screen.

[0788] Enter your email address and password.

[0789] The terminal transmits the entered authentication information to the server.

[0790] Output: The authentication information sent to the server.

[0791] Step 4:

[0792] The server authenticates the user by checking the credentials against a database.

[0793] Input: Received credentials.

[0794] Output: Issue of login session.

[0795] Step 5:

[0796] After successful authentication, the server issues a login session and sends it to the terminal.

[0797] Input: The authentication result by the server.

[0798] Output: Login session information.

[0799] Step 6:

[0800] After the user logs in, they select a simulation situation from the menu.

[0801] Input: Simulation selection (confession, proposal, etc.) and location setting (park, restaurant, home, etc.).

[0802] The terminal transmits the selected situation information to the server.

[0803] Output: The situation information sent to the server.

[0804] Step 7:

[0805] The server sends the corresponding VR data (video, audio, 3D environment data, etc.) to the terminal.

[0806] Input: Simulation situation information.

[0807] Output: VR data sent to the device.

[0808] Step 8:

[0809] The device displays the VR environment set for the user based on the VR data received.

[0810] Input: VR data sent from the server.

[0811] Output: The VR environment displayed to the user.

[0812] Step 9:

[0813] The user puts on VR goggles and enters the virtual reality environment to control an avatar.

[0814] Input: User interaction within the VR environment.

[0815] Output: User actions with avatar.

[0816] Step 10:

[0817] The user begins a dialogue with the AI ​​character.

[0818] Input: User speech and movements.

[0819] The device captures the user's voice and movements in real time and transmits them to the server.

[0820] Output: Audio and movement data sent to the server.

[0821] Step 11:

[0822] The server converts the received voice data into text.

[0823] Input: Received audio data.

[0824] Output: Text-converted audio data.

[0825] Step 12:

[0826] The server passes the converted data and behavioral data to the AI ​​model for analysis.

[0827] Input: Textual data, behavioral data.

[0828] Output: Analysis results.

[0829] Step 13:

[0830] Based on the analysis results, the emotion engine analyzes the user's tone of voice, facial expressions, body movements, etc. to estimate their emotions.

[0831] Input: Analysis results, voice tone, facial expressions, body movements.

[0832] Output: Emotion analysis results.

[0833] Step 14:

[0834] The server combines the results of the emotion engine and AI model to generate feedback and calculate an evaluation score.

[0835] Input: Sentiment analysis results, AI model results.

[0836] Output: Feedback and rating score.

[0837] Step 15:

[0838] The server generates feedback and sends the rating score to the device.

[0839] Input: Feedback and rating score.

[0840] Output: Feedback and rating score sent to the device.

[0841] Step 16:

[0842] The device displays the received feedback and rating score to the user.

[0843] Input: Feedback and rating score sent by the server.

[0844] Output: The feedback and rating score displayed to the user.

[0845] Step 17:

[0846] Users can view feedback and use it to improve their performance.

[0847] Input: The displayed feedback and rating score.

[0848] Output: User improvement actions.

[0849] Step 18:

[0850] Feedback history and rating scores are stored on the server.

[0851] Input: Generated feedback and rating scores.

[0852] Output: Feedback history and rating scores stored on the server.

[0853] This allows users to analyze their performance in real time and receive high-quality feedback, enabling them to effectively learn and improve their skills.

[0854] (Application example 2)

[0855] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0856] Improving customer satisfaction in modern brick-and-mortar stores requires employees to have high customer service skills. However, new and inexperienced employees have limited opportunities to practice in real-life customer service situations, and nervousness and mistakes made with real customers often hinder their learning. Furthermore, traditional training methods make it difficult to provide individual feedback quickly, and there is a lack of efficient ways to promote employee skill improvement. To solve these issues, the development of a customer service training system using VR is desired.

[0857] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for setting up a VR environment, means for capturing the user's voice and movements in real time, means including an artificial intelligence model for analyzing the captured voice and movement data and generating feedback, means for displaying the feedback to the user, means for saving the generated feedback and evaluation score, and means for creating a virtual training scenario for employees of a physical store to practice their customer service skills. This allows employees to effectively practice in a virtual reality customer service scenario and improve their skills while receiving specific feedback.

[0858] A "means for setting up a VR environment" is a device or program that creates a virtual reality space and provides an environment for users to experience a specific situation or location.

[0859] "Means for capturing user voice and movements in real time" refers to a device or program that instantly records the user's voice and body movements and provides the data for subsequent analysis.

[0860] "Means including an artificial intelligence model for analyzing captured voice and movement data and generating feedback" refers to a system that uses artificial intelligence to analyze recorded voice and movement data and provides the user with guidelines for improvement based on the results.

[0861] The "means for displaying feedback to the user" refers to a device or program that presents the analysis results to the user in an appropriate form and provides information that allows the user to check and improve their own performance.

[0862] The "means for storing the generated feedback and evaluation score" refers to a device or program that records the feedback provided to the user and the evaluation results, and stores them in a form that can be referenced later.

[0863] "A means for creating virtual training scenarios for brick-and-mortar store employees to practice customer service skills" is a system that recreates customer service situations in brick-and-mortar stores in virtual reality, and designs and provides scenarios to train employees to respond appropriately in the moment.

[0864] "Means for selecting situations and locations for practicing conversations and actions in customer service situations in a physical store" refers to a device or program that allows the user to select a specific customer service situation or environment and set up an appropriate practice scenario based on that selection.

[0865] The "means for generating specific advice as feedback to improve the user's customer service skills" is a system that provides specific guidelines on how the user should improve their customer service based on analyzed data.

[0866] This invention relates to a system that uses virtual reality (VR) to train employees in customer service skills in brick-and-mortar stores. Hereinafter, specific embodiments of the invention will be described in detail.

[0867] System configuration

[0868] The system mainly consists of the following elements:

[0869] 1. How to set up a VR environment

[0870] The server uses Unity to create a virtual reality space, providing an environment in which users can practice customer service in specific situations and locations.

[0871] 2. A means of capturing the user's voice and actions in real time

[0872] It uses the camera and microphone built into the user's smartphone or VR goggles to instantly record voice and movement.

[0873] 3. Means including an artificial intelligence model for analyzing the captured audio and motion data and generating feedback.

[0874] The server uses artificial intelligence models trained in TensorFlow and OpenCV to analyze the recorded voice and movement data and generate feedback for the user.

[0875] 4. How to display feedback to the user

[0876] Feedback is displayed on the smartphone or VR goggles screen using React Native, allowing users to see the analysis results in real time.

[0877] 5. Means of storing generated feedback and rating scores

[0878] The feedback and evaluation scores are stored on a server and made available to users for future reference, allowing them to track their progress and understand areas for improvement.

[0879] 6. A way to create virtual training scenarios for brick-and-mortar store associates to practice customer service skills

[0880] The server provides scenarios that recreate specific customer service situations in a real store (e.g., greeting customers at a restaurant), allowing users to practice in situations that are close to the real situation.

[0881] System Operation

[0882] 1. Setting up the VR environment

[0883] The server uses Unity to create a virtual environment based on the situation and location selected by the user, allowing the user to begin a customer service simulation in virtual reality.

[0884] 2. Audio and motion capture

[0885] The camera and microphone built into the user's smartphone or VR goggles capture the user's voice and movements in real time and transmit them to a server.

[0886] 3. Analyzing data and generating feedback

[0887] The server analyzes the captured voice and motion data using TensorFlow and OpenCV to evaluate the user's emotions and performance, and generates specific feedback for the user based on the analysis results.

[0888] 4. Viewing Feedback

[0889] The generated feedback and rating scores are displayed on the user's smartphone or VR goggles using React Native, allowing the user to improve their performance.

[0890] 5. Data storage

[0891] Feedback and evaluation scores are stored on a server for future reference, allowing users to track their progress and continually improve their skills.

[0892] Examples of concrete examples and prompts

[0893] For example, if a user wants to simulate "greeting customers at a restaurant," the following prompt sentences are exemplified:

[0894] "Start a simulation of greeting a customer in a VR environment. The scenario is 'restaurant'."

[0895] "Hello, welcome! My name is [Name]. I'll show you to your seat."

[0896] This allows users to practice in situations that are similar to actual customer service situations and improve their skills while receiving specific feedback.

[0897] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0898] Step 1:

[0899] A user launches an application and enters their email address and password on the login screen. The input data is sent to the server, which then performs the authentication process. If authentication is successful, the user is redirected to the main menu. The input is an email address and password, and the output is the authentication result (success or failure).

[0900] Step 2:

[0901] The user selects "Customer Service Training" from the main menu. Then, the training situation and location (e.g., greeting customers at a restaurant) are set. The set situation information is sent to the server. The set situation information is the input, and the setting information is sent to the server as the output.

[0902] Step 3:

[0903] The server uses Unity to create a virtual reality (VR) environment based on the situation and location selected by the user. The created VR environment data is sent to the user's device. The input is configuration information, and the output is VR environment data that is generated and sent to the user's device.

[0904] Step 4:

[0905] The user puts on the VR goggles and starts the simulation in the VR environment. The user's movements and voice are captured in real time and sent from the device to the server. The input is the user's movements and voice, and the captured data is sent to the server as the output.

[0906] Step 5:

[0907] The server uses TensorFlow to analyze the captured voice and motion data, which generates emotion and performance data for the user. The input is the captured data, and the output is the emotion and performance data.

[0908] Step 6:

[0909] The server generates specific feedback based on the analysis results. The generated feedback and evaluation score are sent to the device. The input is emotion and performance data, and the output is feedback and evaluation score.

[0910] Step 7:

[0911] The device uses React Native to display feedback and a rating score to the user, allowing the user to see their performance and understand areas for improvement. The input is the feedback and rating score, and the output is what is displayed to the user.

[0912] Step 8:

[0913] The generated feedback and rating score are stored on the server for future reference. The input is the feedback and rating score, and the output is stored in a database.

[0914] Step 9:

[0915] Users can then replay the simulation and improve their customer service skills based on the feedback. Users can practice repeatedly at their own pace and monitor their progress while honing their skills. The input is the previous feedback and evaluation score, and the output is improved performance.

[0916] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0917] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0918] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0919] [Third embodiment]

[0920] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0921] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0922] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0923] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0924] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0925] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0926] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0927] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0928] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0929] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0930] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0931] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0932] The present invention provides a system that allows a user to simulate a confession or proposal in a virtual reality (VR) environment, analyzes and evaluates the process using artificial intelligence (AI), and provides feedback. Hereinafter, specific embodiments of the present invention will be described in detail.

[0933] System Overview

[0934] User Registration and Login

[0935] A user starts an application and registers a new account. The user enters the required information (e.g., email address, username, password) and sends it to the server. The server creates a user account based on the received information and saves it in the database. A user with an existing account enters their email address and password on the login screen, is authenticated by the server, and logs into the system.

[0936] Setting the VR situation

[0937] After logging in, users can select a "confession" or "proposal" situation from the menu and then specify a specific location (e.g., a park, a restaurant, or their own home). Based on this, the server sends the relevant VR data to the device, which then displays the VR environment to the user.

[0938] Start the conversation simulation

[0939] The user operates an avatar in the VR space and begins to interact with the AI ​​character. The user's voice and actions are captured in real time by the device and sent to the server. The server converts the voice data into text and uses it for analysis along with the movement data.

[0940] AI analysis and feedback

[0941] The server analyzes the received voice and movement data using an AI model. This analysis evaluates the user's language, facial expressions, and behavior. Based on this, the AI ​​generates specific feedback and calculates an evaluation score. Feedback may include advice such as "Speak a little louder" or "Look at the other person's face more."

[0942] View and save feedback

[0943] The generated feedback and evaluation score are sent from the server to the device and displayed to the user. Based on this, the user can check their own performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later. This allows the user to check their progress and use the information to improve their next simulation.

[0944] Specific examples

[0945] For example, if a user wants to simulate a marriage proposal at a restaurant, they first launch the application and log in. After selecting "Proposal" and "Restaurant" from the menu, the device displays the corresponding VR environment. The user puts on VR goggles and controls an avatar in the VR environment to begin a conversation with the AI ​​character. When the user utters the proposal phrase, "Will you marry me?", the device captures the audio and sends it to the server. The server converts the audio into text and analyzes it using an AI model. Based on the results, the AI ​​generates feedback such as "Speak with more confidence," calculates an evaluation score, and presents it to the user. The user can then repeat the simulation based on this feedback to improve their skills.

[0946] Thus, the present invention provides an effective and practical system that allows users to practice confession and proposal situations with a high success rate.

[0947] The processing flow will be explained below.

[0948] Step 1:

[0949] The user launches the application and is presented with a new registration or login screen.

[0950] Step 2:

[0951] The user selects new registration and enters the required information (email address, username, password).

[0952] The terminal transmits the input information to the server.

[0953] Step 3:

[0954] The server stores the received information in a database and notifies the terminal that the account creation is complete.

[0955] Step 4:

[0956] The user enters their email address and password on the login screen and clicks "Login."

[0957] The terminal transmits the input information to the server.

[0958] Step 5:

[0959] The server checks the login information, and if authentication is successful, starts a session and notifies the terminal.

[0960] The device displays a menu screen.

[0961] Step 6:

[0962] The user selects a "confession" or "proposal" situation from the menu, and then selects a specific location (e.g., park, restaurant, home).

[0963] Step 7:

[0964] The terminal transmits the selected situation and location data to the server.

[0965] The server sends the relevant VR data back to the device.

[0966] Step 8:

[0967] The device creates a VR environment based on the returned data and displays it to the user, who then puts on the VR goggles and begins the simulation.

[0968] Step 9:

[0969] Users control an avatar in a VR environment and begin interacting with AI characters, with their voices and actions captured in real time.

[0970] Step 10:

[0971] The device converts the captured audio into text in real time and sends it to the server along with the motion data.

[0972] Step 11:

[0973] The server then uses an AI model to analyze the received voice and movement data, including the user's speech, facial expressions, and body movements.

[0974] Step 12:

[0975] The server generates specific feedback based on the analysis results, calculates an evaluation score, and sends the generated feedback and evaluation score back to the device.

[0976] Step 13:

[0977] The device displays the feedback and evaluation score to the user, who can then review the feedback and identify areas for improvement for the next simulation.

[0978] Step 14:

[0979] The server stores the generated evaluation scores and feedback history in a database so that users can refer to them later.

[0980] Step 15:

[0981] Users can then re-run the simulation based on the feedback to improve their skills.

[0982] Example 1

[0983] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0984] In modern society, improving communication skills in situations such as confessing your feelings or proposing is important, and there is a growing demand for practice systems for this purpose. However, existing systems lack real-time feedback and have difficulty for users checking their own progress. Furthermore, there are limited means to provide a virtual reality (VR) environment that is close to the real situation and generate detailed feedback, which makes it difficult to practice effectively.

[0985] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0986] In this invention, the server includes: a means for a user to register a new account and log in; a means for setting up a VR environment; a means for constructing and displaying a VR environment based on a situation and location selected by the user; a means for capturing the user's voice and movements in real time; a means including an artificial intelligence model for converting the captured voice data into text and analyzing it together with the movement data; a means for generating feedback and an evaluation score based on the analysis results; a means for displaying the generated feedback and evaluation score to the user; and a means for saving the generated feedback and evaluation score. This allows the user to receive real-time feedback in an environment that resembles a real situation, enabling effective practice while checking their progress.

[0987] "User" refers to an individual who uses the system to simulate confessing or proposing in a VR environment.

[0988] "Account" refers to a collection of information (e.g., email address, username, password) that a user registers to access the system.

[0989] "Login" refers to the operation of enabling a user to access the system using registered account information.

[0990] "VR environment" refers to a simulation space created using virtual reality technology.

[0991] "Situation" refers to a specific scene such as a confession or proposal of marriage that the user selects.

[0992] "Location" refers to a specific place where a situation is set (e.g., a park, a restaurant, your home, etc.).

[0993] "Server" refers to the central computing unit that receives user operations and input data, constructs the VR environment, analyzes the data, generates feedback, and so on.

[0994] "Terminal" refers to the device operated by the user (e.g., smartphone, tablet, VR goggles).

[0995] "Voice data" refers to data that captures what the user has said as a voice signal.

[0996] "Motion data" refers to data captured by sensors or other devices that captures the physical movements a user makes within a VR environment.

[0997] "Text conversion" refers to the process of converting audio data into text data.

[0998] "Artificial Intelligence Model" refers to the AI ​​software and algorithms used to analyze received data and generate feedback and / or evaluation scores.

[0999] "Analysis" refers to the process by which an artificial intelligence model evaluates a user's performance based on speech and motion data.

[1000] "Feedback" refers to specific advice or evaluation provided to the user based on the analysis results.

[1001] "Evaluation score" refers to the quantitative representation of a user's performance obtained by quantifying the analysis results.

[1002] "Storage" refers to the act of recording the generated feedback and rating scores in a database for future reference.

[1003] This invention is a system that allows users to simulate confessing or proposing in a virtual reality (VR) environment, and uses artificial intelligence (AI) to analyze and evaluate the process and provide feedback. Hereinafter, specific embodiments of the present invention will be described in detail.

[1004] User Registration and Login

[1005] User operations

[1006] A user starts the application and clicks the "New Registration" button to create an account. The user enters the required information (email address, username, password). Users with existing accounts can access the system by entering their email address and password on the login screen.

[1007] Device behavior

[1008] The terminal receives the user's input information and sends it to the server. In the case of login, the terminal sends the received authentication information to the server.

[1009] Server Operation

[1010] The server receives the new registration information and stores it in a database. It then checks the login information against the database and performs appropriate authentication. It then sends a response to the terminal and directs the user to the menu screen.

[1011] Setting the VR situation

[1012] User operations

[1013] The user selects a "confession" or "proposal" situation from the menu and sets a specific location (e.g., a park, restaurant, home, etc.).

[1014] Device behavior

[1015] The terminal receives the user's selection information and transmits it to the server.

[1016] Server Operation

[1017] Based on the received information, the server generates relevant VR data and sends it to the terminal.

[1018] Terminal display

[1019] The device receives the VR data and displays the virtual environment to the user through the VR goggles.

[1020] Start the conversation simulation

[1021] User operations

[1022] The user controls an avatar in the VR space and begins a conversation with the AI ​​character, for example, proposing marriage by saying, "Will you marry me?"

[1023] Terminal capture

[1024] The device captures the user's voice with a microphone and tracks their movements with sensors, which are then sent to a server in real time.

[1025] AI analysis and feedback

[1026] Server Processing

[1027] The server converts the received voice data into text using voice recognition software (e.g., Google Cloud Speech-to-Text). It also inputs the data along with movement data into an artificial intelligence model (e.g., IBM Watson, OpenAI GPT-4) for analysis. The analysis evaluates the user's language, facial expressions, and behavior.

[1028] Generate analysis results

[1029] The server generates feedback based on the analysis results and calculates an evaluation score that includes specific advice (e.g., "Speak more confidently").

[1030] View and save feedback

[1031] Server Send

[1032] The server sends the generated feedback and evaluation score to the terminal.

[1033] Terminal display

[1034] The device displays the received feedback and evaluation score to the user, who can review it and use it to improve their performance.

[1035] Save Server

[1036] The server stores the generated feedback and rating scores in a database for future reference by the user.

[1037] Specific examples

[1038] For example, if a user simulates a "proposal at a restaurant":

[1039] 1. The user launches the application and logs in.

[1040] 2. Select "Propose" from the menu and set "Restaurant."

[1041] 3. The server sends the VR data to the device, and the device displays the VR environment.

[1042] 4. The user puts on the VR goggles and proposes, "Will you marry me?"

[1043] 5. The device captures the audio and sends it to the server, which converts it into text and analyzes it using an AI model.

[1044] 6. The AI ​​model generates feedback such as "Speak more confidently" and calculates an evaluation score.

[1045] 7. Feedback and evaluation scores are displayed on the device, and the user can use them to re-run the simulation and improve their skills.

[1046] Thus, the present invention provides an effective and practical system that allows users to practice confession and proposal situations with a high success rate.

[1047] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1048] Processing Steps

[1049] Step 1:

[1050] The user launches the application and clicks the "New Registration" button, providing their email address, username, and password as input.

[1051] The terminal receives this input information and transmits it to the server.

[1052] The server creates a new account based on the received information and stores it in the database. It then sends a confirmation of success to the terminal, which displays a success message to the user.

[1053] Step 2:

[1054] The user enters an email address and password on the login screen.

[1055] The terminal receives the entered login information and sends it to the server.

[1056] The server verifies the received information against the account information stored in the database and, if authentication is successful, sends a response to the terminal to display the menu screen.

[1057] The device displays a menu screen.

[1058] Step 3:

[1059] The user selects "confession" or "proposal" from the menu screen and sets the situation and a specific location (e.g., park, restaurant, home, etc.).

[1060] The terminal receives the selection information and transmits it to the server.

[1061] Step 4:

[1062] The server uses the received information to prepare the corresponding VR data, for example, using 3D modeling software.

[1063] The generated VR data is sent to the terminal.

[1064] The device uses the received VR data to display the VR environment, and the user wears VR goggles to visually and physically experience the virtual environment.

[1065] Step 5:

[1066] The user controls an avatar in the VR space and begins a conversation with the AI ​​character, for example proposing marriage by saying, "Will you marry me?"

[1067] The device captures the user's voice with a microphone and tracks the avatar's movements with sensors, and transmits this data to a server in real time.

[1068] Step 6:

[1069] The server converts the received voice data into text using voice recognition software (e.g., Google Cloud Speech-to-Text), and also prepares the movement data for analysis.

[1070] This data is input into an artificial intelligence model (e.g., IBM Watson, OpenAI GPT-4) and analyzed, focusing on the user's language, facial expressions, and behavior.

[1071] Step 7:

[1072] The server generates specific feedback based on the analysis results of the AI ​​model, such as advice like "Speak with more confidence" and an evaluation score.

[1073] The generated feedback and evaluation score are transmitted to the terminal.

[1074] Step 8:

[1075] The device displays the received feedback and rating score to the user, allowing them to review their performance and understand areas for improvement.

[1076] The server stores the feedback and rating scores in a database for future reference by the user.

[1077] (Application example 1)

[1078] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1079] Improving service quality in customer service is a major challenge in today's industry. Training and skill development for new employees is particularly important, but on-site training has its limitations, making it difficult to find effective methods. Furthermore, there is a lack of systems for providing effective and instant feedback. To address these challenges, there is a demand for new training systems using virtual reality (VR) technology.

[1080] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1081] In this invention, the server includes a means for setting up a VR environment, a means for capturing a user's voice and movements in real time, a means including a generative AI model for analyzing the captured voice and movement data and generating feedback, a means for displaying the feedback to the user, a means for saving the generated feedback and evaluation score, and a means for the user to practice customer service in the VR environment, generate prompt sentences, and analyze the results, thereby enabling practical and effective customer service training in a virtual environment.

[1082] A "VR environment" is an environment that uses virtual reality technology to create and experience virtual spaces and situations that users can immerse themselves in.

[1083] "Capturing user voice and movements" refers to recording the voices uttered by the user and the body movements and gestures made by the user in real time.

[1084] A "generative AI model" is a model that uses artificial intelligence technology to analyze captured data and provide feedback and evaluation.

[1085] "Displaying feedback" refers to providing the generated feedback or evaluation results to the user visually or audibly.

[1086] "Storing evaluation scores" means recording the generated evaluation results in a database or storage device so that they can be referenced later.

[1087] A "prompt" is a textual question or instruction input to an AI model that guides the analysis and generation process.

[1088] "Customer service training" is a practice process for improving the skills and manners required for customer service work, in which the user simulates customer service.

[1089] "Real-time text transcription" is the process of instantly converting voice data into text.

[1090] "Setting the situation and location" means that the user selects the specific scene and location of the simulation and then constructs the VR environment based on that setting.

[1091] User Registration and Login

[1092] First, a user launches an application on a VR-compatible smartphone or head-mounted display (HMD) and creates a new account. The required information (e.g., email address, username, password) is entered and sent to the server. The server uses the received information to create a user account and stores it in a database. Users who already have an account can enter their email address and password on the login screen, receive authentication from the server, and access the system.

[1093] Setting the VR situation

[1094] After logging in, the user selects a simulation such as "Customer Service Training" from the menu. A specific scene (e.g., a store interior or counter) is set, and the server sends the relevant VR data to the device. The device displays the set VR environment to the user, and the user controls an avatar in the VR space.

[1095] Start the conversation simulation

[1096] The user controls an avatar in the VR space and starts a simulated conversation with an AI character. The user's voice and movements are captured in real time by the device and transmitted to the server.

[1097] AI analysis and feedback

[1098] The server converts the received voice data into text and analyzes it along with the behavioral data using a generative AI model. Based on the analysis results, the generative AI model generates feedback and an evaluation score. The feedback includes specific advice (e.g., "Speak with more confidence").

[1099] View and save feedback

[1100] The generated feedback and evaluation score are sent from the server to the device and displayed to the user, allowing the user to check their performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later, allowing the user to continuously improve their skills.

[1101] Program processing

[1102] The system is built using Python programs and open source libraries (SpeechRecognition, Transformers, etc.). Speech data is converted to text using the SpeechRecognition library and analyzed by a generative AI model (e.g., OpenAI's GPT series). The generated feedback and evaluation scores are displayed on the user interface. The results of the user's simulations are stored in a database.

[1103] Specific example explanation

[1104] For example, if a user wants to simulate "in-store customer service training," they launch the application and log in. By selecting "Customer Service Training" and "In-Store" from the menu, the device will display the corresponding VR environment. The user puts on the HMD and controls an avatar in the VR environment to begin a conversation with an AI character. Voice data is captured in the form of "Please tell me more about this product" and sent to the server. The server converts the voice into text and analyzes it with a generative AI model. Based on the results, feedback such as "Please explain more calmly" is generated, and an evaluation score is calculated and presented to the user. The user can then run the simulation again based on this feedback.

[1105] Prompt Sentence Examples

[1106] "Username: testuser

[1107] Scene: Customer service in store

[1108] Dialogue: Tell me more about this product."

[1109] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1110] Step 1:

[1111] User Registration and Login

[1112] A user starts the application on their device and enters their email address, username, and password on the account registration screen. The server receives this input data, creates a user account, and stores it in the database. A user with an existing account enters their email address and password on the login screen, receives authentication from the server, and accesses the system.

[1113] Input: User information (email address, username, password)

[1114] Output: Account creation or system login privileges

[1115] Step 2:

[1116] Setting the VR situation

[1117] After logging in, the user selects a simulation scene, such as "Customer Service Training," from the device's menu. The user sets a specific scene and location (e.g., inside a store or at a counter). The server sends relevant VR data based on the selected scene and location to the device, and the device uses that data to display the VR environment.

[1118] Input: Simulation scene and location selection

[1119] Output: Sending and displaying selected VR data

[1120] Step 3:

[1121] Start the conversation simulation

[1122] The user controls an avatar in the VR environment and begins to interact with the AI ​​character. The device captures the user's voice and actions in real time and sends the data to the server.

[1123] Input: User voice and movement data

[1124] Output: Sending captured data in real time

[1125] Step 4:

[1126] AI-based analysis and feedback generation

[1127] The server converts the received voice data into text and inputs it into a generative AI model along with the motion data. The generative AI model analyzes this data and generates feedback and an evaluation score. This feedback includes specific advice, such as "Speak more confidently."

[1128] Input: Voice data, motion data

[1129] Output: Feedback and rating score

[1130] Step 5:

[1131] View and save feedback

[1132] The generated feedback and evaluation score are sent from the server to the device and displayed to the user. Based on this feedback, the user can check their own performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later.

[1133] Input: Feedback and rating scores

[1134] Output: View and save feedback

[1135] Example prompt sentences:

[1136] "Username: testuser

[1137] Scene: Customer service in store

[1138] Dialogue: Tell me more about this product."

[1139] By following the steps above, a customer service training system can be implemented, which helps users improve their customer service skills in a practical environment.

[1140] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1141] The present invention is a system in which a user simulates confessing or proposing in a virtual reality (VR) environment, analyzes and evaluates the process using artificial intelligence (AI), and provides feedback by combining it with an emotion engine. A detailed description of specific embodiments of the present invention is provided below.

[1142] System Overview

[1143] User Registration and Login

[1144] A user starts an application and registers a new account. The user enters the required information (e.g., email address, username, password) and sends it to the server. The server creates a user account based on the received information and saves it in the database. A user with an existing account enters their email address and password on the login screen, is authenticated by the server, and logs into the system.

[1145] Setting the VR situation

[1146] After logging in, users can select a "confession" or "proposal" situation from the menu and then specify a specific location (e.g., a park, a restaurant, or their own home). Based on this, the server sends the relevant VR data to the device, which then displays the VR environment to the user.

[1147] Start the conversation simulation

[1148] The user operates an avatar in the VR space and begins to interact with the AI ​​character. The user's voice and actions are captured in real time by the device and sent to the server. The server converts this voice data into text and uses it for analysis along with the movement data.

[1149] Analysis and feedback by AI and emotion engine

[1150] The server analyzes the received voice and behavior data using an AI model and emotion engine. The emotion engine analyzes the user's tone of voice, facial expressions, and body movements to estimate their emotions. Based on this, the AI ​​generates feedback and calculates an evaluation score. For example, specific advice such as "It would be more effective if you smiled more" or "Speaking with confidence will help the other person understand you" is generated.

[1151] View and save feedback

[1152] The generated feedback and evaluation score are sent from the server to the device and displayed to the user. Based on this, the user can check their own performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later. This allows the user to check their progress and use the information to improve their next simulation.

[1153] Specific examples

[1154] For example, if a user wants to simulate a marriage proposal at a restaurant, they first launch the application and log in. After selecting "Proposal" and "Restaurant" from the menu, the device displays the corresponding VR environment. The user puts on VR goggles and controls an avatar in the VR environment to begin a conversation with an AI character. When the user utters the proposal phrase, "Will you marry me?", the device captures the audio and sends it to the server. The server converts the audio into text and analyzes it using an emotion engine. Based on the analysis results, feedback such as "Speak with more confidence" or "Try to look closely at the other person's face" is generated and presented to the user along with an evaluation score. The user can then repeat the simulation based on this feedback to improve their skills.

[1155] Thus, the present invention provides an effective and practical system combined with an emotion engine that allows users to practice confession and proposal situations with a high success rate.

[1156] The processing flow will be explained below.

[1157] Step 1:

[1158] The user launches the application and is presented with a new registration or login screen.

[1159] Step 2:

[1160] The user selects new registration and enters the required information (email address, username, password).

[1161] The terminal transmits the input information to the server.

[1162] Step 3:

[1163] The server stores the received information in a database and notifies the terminal that the account creation is complete.

[1164] Step 4:

[1165] The user enters their email address and password on the login screen and clicks "Login."

[1166] The terminal transmits the input information to the server.

[1167] Step 5:

[1168] The server checks the login information, and if authentication is successful, starts a session and notifies the terminal.

[1169] The device displays a menu screen.

[1170] Step 6:

[1171] The user selects a "confession" or "proposal" situation from the menu, and then selects a specific location (e.g., park, restaurant, home).

[1172] Step 7:

[1173] The terminal transmits the selected situation and location data to the server.

[1174] The server sends the relevant VR data back to the device.

[1175] Step 8:

[1176] The device creates a VR environment based on the returned data and displays it to the user, who then puts on the VR goggles and begins the simulation.

[1177] Step 9:

[1178] Users control an avatar in a VR environment and begin interacting with AI characters, with their voices and actions captured in real time.

[1179] Step 10:

[1180] The device converts the captured audio into text in real time and sends it to the server along with the motion data.

[1181] Step 11:

[1182] The server analyzes the received voice and movement data using an AI model and emotion engine, which analyzes the user's tone of voice, facial expressions, and body movements to estimate the user's emotions.

[1183] Step 12:

[1184] The server generates specific feedback based on the analysis results, calculates an evaluation score, and sends the generated feedback and evaluation score back to the device.

[1185] Step 13:

[1186] The device displays the feedback and evaluation score to the user, who can then review the feedback and identify areas for improvement for the next simulation.

[1187] Step 14:

[1188] The server stores the generated evaluation scores and feedback history in a database for users to refer to later. Users can use this stored data to prepare for marriage proposals or confessions.

[1189] Step 15:

[1190] The user then runs the simulation again based on the feedback and implements the suggested improvements. By repeating this process, the user can improve their skills.

[1191] Example 2

[1192] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1193] Conventional VR simulation systems lack the ability to capture and analyze users' voices and movements in real time, making it difficult to accurately analyze users' emotions and subtle nuances. Furthermore, the quality of the feedback is low, making it difficult for users to effectively learn and improve. Therefore, there is a demand for a system that can provide users with high-quality feedback by capturing users' voices and movements in real time and analyzing them using an emotion engine.

[1194] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for setting a VR environment, means for capturing the user's voice and movements in real time, means including an artificial intelligence model for analyzing the captured voice and movement data and generating feedback, means including an emotion engine for analyzing emotions, means for displaying feedback and an evaluation score to the user, and means for saving the generated feedback and evaluation score. This allows the user to analyze their own performance in real time and receive high-quality feedback, thereby enabling them to effectively learn and improve their skills.

[1195] A "VR environment" is a computer-generated simulation space that allows users to experience visual and auditory sensations in a virtual reality space.

[1196] "Means for capturing user voice and movements in real time" refers to devices or software that instantly record a user's voice and body movements as digital data and transmit that information to the system.

[1197] "Means including an artificial intelligence model for analyzing captured voice and movement data and generating feedback" refers to a machine learning algorithm or AI engine for processing and analyzing collected voice and movement data and providing improvement advice to the user based on the results.

[1198] "Emotion engine for analyzing emotions" refers to software and algorithms that analyze a user's tone of voice, facial expressions, body movements, etc. to infer the user's emotional state.

[1199] The "means for displaying feedback and evaluation scores to the user" refers to a device or display screen that allows the user to visually confirm the generated advice and evaluation results.

[1200] "Means for storing generated feedback and evaluation scores" refers to hardware or software that records the analysis results and feedback in a database or storage system for future reference.

[1201] "Means for configuring specific situations and locations in a VR environment" refers to the interface and menu system that allows a user to select and configure the scenarios and locations they experience in virtual reality.

[1202] "Means for creating a VR environment based on a specific situation and location and displaying it to the user" refers to software that generates a virtual reality space based on a scenario or location set by the user and outputs it to a display device so that the user can experience that environment.

[1203] "Means for converting text in real time" refers to speech recognition technology and its implementation for instantly converting a user's voice into text data.

[1204] "Means for transmitting data to a server" refers to a communications protocol and its implementation for transmitting the captured digital data over a network to a server.

[1205] "Means for analyzing and generating feedback and evaluation scores" refers to software and algorithms that analyze the received data using AI models and emotion engines to generate specific advice and evaluation results to provide to users.

[1206] The present invention is a system in which a user simulates confessing or proposing in a virtual reality (VR) environment, analyzes and evaluates the process using artificial intelligence (AI), and provides feedback by combining it with an emotion engine. A detailed description of specific embodiments of the present invention is provided below.

[1207] User Registration and Login

[1208] A user starts an application and accesses the new registration screen. The user enters their email address, username, and password, and the device sends the entered information to the server. The server creates a new user account based on the received information and saves it in the database. An existing user enters their email address and password on the login screen, and the device sends the entered authentication information to the server. The server checks the information in the database and authenticates the user. After successful authentication, the server issues a login session and sends it to the device.

[1209] Setting the VR situation

[1210] After logging in, the user selects a simulation situation such as "confession" or "proposal" from the menu. The user sets a specific location, such as a park, restaurant, or home. The device sends the selected situation information to the server. The server then sends the corresponding VR data (video, audio, 3D environment data, etc.) to the device, which receives it and displays the VR environment that the user has set.

[1211] Start the conversation simulation

[1212] The user puts on VR goggles, enters the virtual reality environment, and controls an avatar. The user begins to interact with the AI ​​character. The device captures the user's voice and actions (body movements and gestures) in real time and transmits them to the server.

[1213] Analysis and feedback by AI and emotion engine

[1214] The server converts the received voice data into text. Specifically, it uses voice recognition software (e.g., Google Cloud Speech-to-Text). The server then passes the converted text and behavioral data to an AI model for analysis. The AI ​​models used here include natural language processing (NLP) models and computer vision models. Based on the analysis results, the emotion engine analyzes the user's tone of voice, facial expressions, body movements, etc. to infer their emotions. The server combines the results of the emotion engine and the AI ​​model to generate feedback and calculate an evaluation score. Examples of generated feedback include "It would be more effective if you smile more" and "Speaking with confidence helps you reach the other person."

[1215] View and save feedback

[1216] The server sends the generated feedback and evaluation score to the terminal and displays it to the user. The user can check the feedback and use it to improve their performance. The feedback history and evaluation score are saved on the server, so the user can refer to them later to check their progress and use them for the next simulation.

[1217] Specific examples

[1218] For example, if a user wants to simulate a marriage proposal at a restaurant, they launch the application and log in. By selecting "Proposal" and "Restaurant" from the menu, the device will display the corresponding VR environment. The user puts on VR goggles and controls an avatar in the VR environment to begin a conversation with an AI character. When the user utters the proposal phrase, "Will you marry me?", the device captures the audio and sends it to the server. The server converts the audio into text and analyzes it using an emotion engine. Based on the analysis results, feedback such as "Speak with more confidence" or "Try to look closely at the other person's face" is generated and presented to the user along with an evaluation score.

[1219] Prompt Sentence Examples

[1220] "Start a simulation of proposing at a restaurant"

[1221] "I want to practice the confession scene in the park."

[1222] The present invention thus provides an effective and practical system, combined with an emotion engine, that allows users to practice confession and proposal situations with a high success rate.

[1223] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1224] Step 1:

[1225] The user launches the application and accesses the new registration screen.

[1226] Input: Enter your email address, username, and password.

[1227] The terminal transmits this input information to the server.

[1228] Output: The user information sent to the server.

[1229] Step 2:

[1230] The server creates a new user account based on the received information and stores it in the database.

[1231] Input: User information sent from the device.

[1232] Output: A confirmation message for creating a new account.

[1233] Step 3:

[1234] An existing user enters their email address and password on the login screen.

[1235] Enter your email address and password.

[1236] The terminal transmits the entered authentication information to the server.

[1237] Output: The authentication information sent to the server.

[1238] Step 4:

[1239] The server authenticates the user by checking the credentials against a database.

[1240] Input: Received credentials.

[1241] Output: Issue of login session.

[1242] Step 5:

[1243] After successful authentication, the server issues a login session and sends it to the terminal.

[1244] Input: The authentication result by the server.

[1245] Output: Login session information.

[1246] Step 6:

[1247] After the user logs in, they select a simulation situation from the menu.

[1248] Input: Simulation selection (confession, proposal, etc.) and location setting (park, restaurant, home, etc.).

[1249] The terminal transmits the selected situation information to the server.

[1250] Output: The situation information sent to the server.

[1251] Step 7:

[1252] The server sends the corresponding VR data (video, audio, 3D environment data, etc.) to the terminal.

[1253] Input: Simulation situation information.

[1254] Output: VR data sent to the device.

[1255] Step 8:

[1256] The device displays the VR environment set for the user based on the VR data received.

[1257] Input: VR data sent from the server.

[1258] Output: The VR environment displayed to the user.

[1259] Step 9:

[1260] The user puts on VR goggles and enters the virtual reality environment to control an avatar.

[1261] Input: User interaction within the VR environment.

[1262] Output: User actions with avatar.

[1263] Step 10:

[1264] The user begins a dialogue with the AI ​​character.

[1265] Input: User speech and movements.

[1266] The device captures the user's voice and movements in real time and transmits them to the server.

[1267] Output: Audio and movement data sent to the server.

[1268] Step 11:

[1269] The server converts the received voice data into text.

[1270] Input: Received audio data.

[1271] Output: Text-converted audio data.

[1272] Step 12:

[1273] The server passes the converted data and behavioral data to the AI ​​model for analysis.

[1274] Input: Textual data, behavioral data.

[1275] Output: Analysis results.

[1276] Step 13:

[1277] Based on the analysis results, the emotion engine analyzes the user's tone of voice, facial expressions, body movements, etc. to estimate their emotions.

[1278] Input: Analysis results, voice tone, facial expressions, body movements.

[1279] Output: Emotion analysis results.

[1280] Step 14:

[1281] The server combines the results of the emotion engine and AI model to generate feedback and calculate an evaluation score.

[1282] Input: Sentiment analysis results, AI model results.

[1283] Output: Feedback and rating score.

[1284] Step 15:

[1285] The server generates feedback and sends the rating score to the device.

[1286] Input: Feedback and rating score.

[1287] Output: Feedback and rating score sent to the device.

[1288] Step 16:

[1289] The device displays the received feedback and rating score to the user.

[1290] Input: Feedback and rating score sent by the server.

[1291] Output: The feedback and rating score displayed to the user.

[1292] Step 17:

[1293] Users can view feedback and use it to improve their performance.

[1294] Input: The displayed feedback and rating score.

[1295] Output: User improvement actions.

[1296] Step 18:

[1297] Feedback history and rating scores are stored on the server.

[1298] Input: Generated feedback and rating scores.

[1299] Output: Feedback history and rating scores stored on the server.

[1300] This allows users to analyze their performance in real time and receive high-quality feedback, enabling them to effectively learn and improve their skills.

[1301] (Application example 2)

[1302] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1303] Improving customer satisfaction in modern brick-and-mortar stores requires employees to have high customer service skills. However, new and inexperienced employees have limited opportunities to practice in real-life customer service situations, and nervousness and mistakes made with real customers often hinder their learning. Furthermore, traditional training methods make it difficult to provide individual feedback quickly, and there is a lack of efficient ways to promote employee skill improvement. To solve these issues, the development of a customer service training system using VR is desired.

[1304] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for setting up a VR environment, means for capturing the user's voice and movements in real time, means including an artificial intelligence model for analyzing the captured voice and movement data and generating feedback, means for displaying the feedback to the user, means for saving the generated feedback and evaluation score, and means for creating a virtual training scenario for employees of a physical store to practice their customer service skills. This allows employees to effectively practice in a virtual reality customer service scenario and improve their skills while receiving specific feedback.

[1305] A "means for setting up a VR environment" is a device or program that creates a virtual reality space and provides an environment for users to experience a specific situation or location.

[1306] "Means for capturing user voice and movements in real time" refers to a device or program that instantly records the user's voice and body movements and provides the data for subsequent analysis.

[1307] "Means including an artificial intelligence model for analyzing captured voice and movement data and generating feedback" refers to a system that uses artificial intelligence to analyze recorded voice and movement data and provides the user with guidelines for improvement based on the results.

[1308] The "means for displaying feedback to the user" refers to a device or program that presents the analysis results to the user in an appropriate form and provides information that allows the user to check and improve their own performance.

[1309] The "means for storing the generated feedback and evaluation score" refers to a device or program that records the feedback provided to the user and the evaluation results, and stores them in a form that can be referenced later.

[1310] "A means for creating virtual training scenarios for brick-and-mortar store employees to practice customer service skills" is a system that recreates customer service situations in brick-and-mortar stores in virtual reality, and designs and provides scenarios to train employees to respond appropriately in the moment.

[1311] "Means for selecting situations and locations for practicing conversations and actions in customer service situations in a physical store" refers to a device or program that allows the user to select a specific customer service situation or environment and set up an appropriate practice scenario based on that selection.

[1312] The "means for generating specific advice as feedback to improve the user's customer service skills" is a system that provides specific guidelines on how the user should improve their customer service based on analyzed data.

[1313] This invention relates to a system that uses virtual reality (VR) to train employees in customer service skills in brick-and-mortar stores. Hereinafter, specific embodiments of the invention will be described in detail.

[1314] System configuration

[1315] The system mainly consists of the following elements:

[1316] 1. How to set up a VR environment

[1317] The server uses Unity to create a virtual reality space, providing an environment in which users can practice customer service in specific situations and locations.

[1318] 2. A means of capturing the user's voice and actions in real time

[1319] It uses the camera and microphone built into the user's smartphone or VR goggles to instantly record voice and movement.

[1320] 3. Means including an artificial intelligence model for analyzing the captured audio and motion data and generating feedback.

[1321] The server uses artificial intelligence models trained in TensorFlow and OpenCV to analyze the recorded voice and movement data and generate feedback for the user.

[1322] 4. How to display feedback to the user

[1323] Feedback is displayed on the smartphone or VR goggles screen using React Native, allowing users to see the analysis results in real time.

[1324] 5. Means of storing generated feedback and rating scores

[1325] The feedback and evaluation scores are stored on a server and made available to users for future reference, allowing them to track their progress and understand areas for improvement.

[1326] 6. A way to create virtual training scenarios for brick-and-mortar store associates to practice customer service skills

[1327] The server provides scenarios that recreate specific customer service situations in a real store (e.g., greeting customers at a restaurant), allowing users to practice in situations that are close to the real situation.

[1328] System Operation

[1329] 1. Setting up the VR environment

[1330] The server uses Unity to create a virtual environment based on the situation and location selected by the user, allowing the user to begin a customer service simulation in virtual reality.

[1331] 2. Audio and motion capture

[1332] The camera and microphone built into the user's smartphone or VR goggles capture the user's voice and movements in real time and transmit them to a server.

[1333] 3. Analyzing data and generating feedback

[1334] The server analyzes the captured voice and motion data using TensorFlow and OpenCV to evaluate the user's emotions and performance, and generates specific feedback for the user based on the analysis results.

[1335] 4. Viewing Feedback

[1336] The generated feedback and rating scores are displayed on the user's smartphone or VR goggles using React Native, allowing the user to improve their performance.

[1337] 5. Data storage

[1338] Feedback and evaluation scores are stored on a server for future reference, allowing users to track their progress and continually improve their skills.

[1339] Examples of concrete examples and prompts

[1340] For example, if a user wants to simulate "greeting customers at a restaurant," the following prompt sentences are exemplified:

[1341] "Start a simulation of greeting a customer in a VR environment. The scenario is 'restaurant'."

[1342] "Hello, welcome! My name is [Name]. I'll show you to your seat."

[1343] This allows users to practice in situations that are similar to actual customer service situations and improve their skills while receiving specific feedback.

[1344] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1345] Step 1:

[1346] A user launches an application and enters their email address and password on the login screen. The input data is sent to the server, which then performs the authentication process. If authentication is successful, the user is redirected to the main menu. The input is an email address and password, and the output is the authentication result (success or failure).

[1347] Step 2:

[1348] The user selects "Customer Service Training" from the main menu. Then, the training situation and location (e.g., greeting customers at a restaurant) are set. The set situation information is sent to the server. The set situation information is the input, and the setting information is sent to the server as the output.

[1349] Step 3:

[1350] The server uses Unity to create a virtual reality (VR) environment based on the situation and location selected by the user. The created VR environment data is sent to the user's device. The input is configuration information, and the output is VR environment data that is generated and sent to the user's device.

[1351] Step 4:

[1352] The user puts on the VR goggles and starts the simulation in the VR environment. The user's movements and voice are captured in real time and sent from the device to the server. The input is the user's movements and voice, and the captured data is sent to the server as the output.

[1353] Step 5:

[1354] The server uses TensorFlow to analyze the captured voice and motion data, which generates emotion and performance data for the user. The input is the captured data, and the output is the emotion and performance data.

[1355] Step 6:

[1356] The server generates specific feedback based on the analysis results. The generated feedback and evaluation score are sent to the device. The input is emotion and performance data, and the output is feedback and evaluation score.

[1357] Step 7:

[1358] The device uses React Native to display feedback and a rating score to the user, allowing the user to see their performance and understand areas for improvement. The input is the feedback and rating score, and the output is what is displayed to the user.

[1359] Step 8:

[1360] The generated feedback and rating score are stored on the server for future reference. The input is the feedback and rating score, and the output is stored in a database.

[1361] Step 9:

[1362] Users can then replay the simulation and improve their customer service skills based on the feedback. Users can practice repeatedly at their own pace and monitor their progress while honing their skills. The input is the previous feedback and evaluation score, and the output is improved performance.

[1363] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1364] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1365] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1366] [Fourth embodiment]

[1367] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1368] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1369] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1370] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1371] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1372] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1373] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1374] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1375] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1376] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1377] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1378] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1379] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1380] The present invention provides a system that allows a user to simulate a confession or proposal in a virtual reality (VR) environment, analyzes and evaluates the process using artificial intelligence (AI), and provides feedback. Hereinafter, specific embodiments of the present invention will be described in detail.

[1381] System Overview

[1382] User Registration and Login

[1383] A user starts an application and registers a new account. The user enters the required information (e.g., email address, username, password) and sends it to the server. The server creates a user account based on the received information and saves it in the database. A user with an existing account enters their email address and password on the login screen, is authenticated by the server, and logs into the system.

[1384] Setting the VR situation

[1385] After logging in, users can select a "confession" or "proposal" situation from the menu and then specify a specific location (e.g., a park, a restaurant, or their own home). Based on this, the server sends the relevant VR data to the device, which then displays the VR environment to the user.

[1386] Start the conversation simulation

[1387] The user operates an avatar in the VR space and begins to interact with the AI ​​character. The user's voice and actions are captured in real time by the device and sent to the server. The server converts the voice data into text and uses it for analysis along with the movement data.

[1388] AI analysis and feedback

[1389] The server analyzes the received voice and movement data using an AI model. This analysis evaluates the user's language, facial expressions, and behavior. Based on this, the AI ​​generates specific feedback and calculates an evaluation score. Feedback may include advice such as "Speak a little louder" or "Look at the other person's face more."

[1390] View and save feedback

[1391] The generated feedback and evaluation score are sent from the server to the device and displayed to the user. Based on this, the user can check their own performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later. This allows the user to check their progress and use the information to improve their next simulation.

[1392] Specific examples

[1393] For example, if a user wants to simulate a marriage proposal at a restaurant, they first launch the application and log in. After selecting "Proposal" and "Restaurant" from the menu, the device displays the corresponding VR environment. The user puts on VR goggles and controls an avatar in the VR environment to begin a conversation with the AI ​​character. When the user utters the proposal phrase, "Will you marry me?", the device captures the audio and sends it to the server. The server converts the audio into text and analyzes it using an AI model. Based on the results, the AI ​​generates feedback such as "Speak with more confidence," calculates an evaluation score, and presents it to the user. The user can then repeat the simulation based on this feedback to improve their skills.

[1394] Thus, the present invention provides an effective and practical system that allows users to practice confession and proposal situations with a high success rate.

[1395] The processing flow will be explained below.

[1396] Step 1:

[1397] The user launches the application and is presented with a new registration or login screen.

[1398] Step 2:

[1399] The user selects new registration and enters the required information (email address, username, password).

[1400] The terminal transmits the input information to the server.

[1401] Step 3:

[1402] The server stores the received information in a database and notifies the terminal that the account creation is complete.

[1403] Step 4:

[1404] The user enters their email address and password on the login screen and clicks "Login."

[1405] The terminal transmits the input information to the server.

[1406] Step 5:

[1407] The server checks the login information, and if authentication is successful, starts a session and notifies the terminal.

[1408] The device displays a menu screen.

[1409] Step 6:

[1410] The user selects a "confession" or "proposal" situation from the menu, and then selects a specific location (e.g., park, restaurant, home).

[1411] Step 7:

[1412] The terminal transmits the selected situation and location data to the server.

[1413] The server sends the relevant VR data back to the device.

[1414] Step 8:

[1415] The device creates a VR environment based on the returned data and displays it to the user, who then puts on the VR goggles and begins the simulation.

[1416] Step 9:

[1417] Users control an avatar in a VR environment and begin interacting with AI characters, with their voices and actions captured in real time.

[1418] Step 10:

[1419] The device converts the captured audio into text in real time and sends it to the server along with the motion data.

[1420] Step 11:

[1421] The server then uses an AI model to analyze the received voice and movement data, including the user's speech, facial expressions, and body movements.

[1422] Step 12:

[1423] The server generates specific feedback based on the analysis results, calculates an evaluation score, and sends the generated feedback and evaluation score back to the device.

[1424] Step 13:

[1425] The device displays the feedback and evaluation score to the user, who can then review the feedback and identify areas for improvement for the next simulation.

[1426] Step 14:

[1427] The server stores the generated evaluation scores and feedback history in a database so that users can refer to them later.

[1428] Step 15:

[1429] Users can then re-run the simulation based on the feedback to improve their skills.

[1430] Example 1

[1431] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1432] In modern society, improving communication skills in situations such as confessing your feelings or proposing is important, and there is a growing demand for practice systems for this purpose. However, existing systems lack real-time feedback and have difficulty for users checking their own progress. Furthermore, there are limited means to provide a virtual reality (VR) environment that is close to the real situation and generate detailed feedback, which makes it difficult to practice effectively.

[1433] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1434] In this invention, the server includes: a means for a user to register a new account and log in; a means for setting up a VR environment; a means for constructing and displaying a VR environment based on a situation and location selected by the user; a means for capturing the user's voice and movements in real time; a means including an artificial intelligence model for converting the captured voice data into text and analyzing it together with the movement data; a means for generating feedback and an evaluation score based on the analysis results; a means for displaying the generated feedback and evaluation score to the user; and a means for saving the generated feedback and evaluation score. This allows the user to receive real-time feedback in an environment that resembles a real situation, enabling effective practice while checking their progress.

[1435] "User" refers to an individual who uses the system to simulate confessing or proposing in a VR environment.

[1436] "Account" refers to a collection of information (e.g., email address, username, password) that a user registers to access the system.

[1437] "Login" refers to the operation of enabling a user to access the system using registered account information.

[1438] "VR environment" refers to a simulation space created using virtual reality technology.

[1439] "Situation" refers to a specific scene such as a confession or proposal of marriage that the user selects.

[1440] "Location" refers to a specific place where a situation is set (e.g., a park, a restaurant, your home, etc.).

[1441] "Server" refers to the central computing unit that receives user operations and input data, constructs the VR environment, analyzes the data, generates feedback, and so on.

[1442] "Terminal" refers to the device operated by the user (e.g., smartphone, tablet, VR goggles).

[1443] "Voice data" refers to data that captures what the user has said as a voice signal.

[1444] "Motion data" refers to data captured by sensors or other devices that captures the physical movements a user makes within a VR environment.

[1445] "Text conversion" refers to the process of converting audio data into text data.

[1446] "Artificial Intelligence Model" refers to the AI ​​software and algorithms used to analyze received data and generate feedback and / or evaluation scores.

[1447] "Analysis" refers to the process by which an artificial intelligence model evaluates a user's performance based on speech and motion data.

[1448] "Feedback" refers to specific advice or evaluation provided to the user based on the analysis results.

[1449] "Evaluation score" refers to the quantitative representation of a user's performance obtained by quantifying the analysis results.

[1450] "Storage" refers to the act of recording the generated feedback and rating scores in a database for future reference.

[1451] This invention is a system that allows users to simulate confessing or proposing in a virtual reality (VR) environment, and uses artificial intelligence (AI) to analyze and evaluate the process and provide feedback. Hereinafter, specific embodiments of the present invention will be described in detail.

[1452] User Registration and Login

[1453] User operations

[1454] A user starts the application and clicks the "New Registration" button to create an account. The user enters the required information (email address, username, password). Users with existing accounts can access the system by entering their email address and password on the login screen.

[1455] Device behavior

[1456] The terminal receives the user's input information and sends it to the server. In the case of login, the terminal sends the received authentication information to the server.

[1457] Server Operation

[1458] The server receives the new registration information and stores it in a database. It then checks the login information against the database and performs appropriate authentication. It then sends a response to the terminal and directs the user to the menu screen.

[1459] Setting the VR situation

[1460] User operations

[1461] The user selects a "confession" or "proposal" situation from the menu and sets a specific location (e.g., a park, restaurant, home, etc.).

[1462] Device behavior

[1463] The terminal receives the user's selection information and transmits it to the server.

[1464] Server Operation

[1465] Based on the received information, the server generates relevant VR data and sends it to the terminal.

[1466] Terminal display

[1467] The device receives the VR data and displays the virtual environment to the user through the VR goggles.

[1468] Start the conversation simulation

[1469] User operations

[1470] The user controls an avatar in the VR space and begins a conversation with the AI ​​character, for example, proposing marriage by saying, "Will you marry me?"

[1471] Terminal capture

[1472] The device captures the user's voice with a microphone and tracks their movements with sensors, which are then sent to a server in real time.

[1473] AI analysis and feedback

[1474] Server Processing

[1475] The server converts the received voice data into text using voice recognition software (e.g., Google Cloud Speech-to-Text). It also inputs the data along with movement data into an artificial intelligence model (e.g., IBM Watson, OpenAI GPT-4) for analysis. The analysis evaluates the user's language, facial expressions, and behavior.

[1476] Generate analysis results

[1477] The server generates feedback based on the analysis results and calculates an evaluation score that includes specific advice (e.g., "Speak more confidently").

[1478] View and save feedback

[1479] Server Send

[1480] The server sends the generated feedback and evaluation score to the terminal.

[1481] Terminal display

[1482] The device displays the received feedback and evaluation score to the user, who can review it and use it to improve their performance.

[1483] Save Server

[1484] The server stores the generated feedback and rating scores in a database for future reference by the user.

[1485] Specific examples

[1486] For example, if a user simulates a "proposal at a restaurant":

[1487] 1. The user launches the application and logs in.

[1488] 2. Select "Propose" from the menu and set "Restaurant."

[1489] 3. The server sends the VR data to the device, and the device displays the VR environment.

[1490] 4. The user puts on the VR goggles and proposes, "Will you marry me?"

[1491] 5. The device captures the audio and sends it to the server, which converts it into text and analyzes it using an AI model.

[1492] 6. The AI ​​model generates feedback such as "Speak more confidently" and calculates an evaluation score.

[1493] 7. Feedback and evaluation scores are displayed on the device, and the user can use them to re-run the simulation and improve their skills.

[1494] Thus, the present invention provides an effective and practical system that allows users to practice confession and proposal situations with a high success rate.

[1495] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1496] Processing Steps

[1497] Step 1:

[1498] The user launches the application and clicks the "New Registration" button, providing their email address, username, and password as input.

[1499] The terminal receives this input information and transmits it to the server.

[1500] The server creates a new account based on the received information and stores it in the database. It then sends a confirmation of success to the terminal, which displays a success message to the user.

[1501] Step 2:

[1502] The user enters an email address and password on the login screen.

[1503] The terminal receives the entered login information and sends it to the server.

[1504] The server verifies the received information against the account information stored in the database and, if authentication is successful, sends a response to the terminal to display the menu screen.

[1505] The device displays a menu screen.

[1506] Step 3:

[1507] The user selects "confession" or "proposal" from the menu screen and sets the situation and a specific location (e.g., park, restaurant, home, etc.).

[1508] The terminal receives the selection information and transmits it to the server.

[1509] Step 4:

[1510] The server uses the received information to prepare the corresponding VR data, for example, using 3D modeling software.

[1511] The generated VR data is sent to the terminal.

[1512] The device uses the received VR data to display the VR environment, and the user wears VR goggles to visually and physically experience the virtual environment.

[1513] Step 5:

[1514] The user controls an avatar in the VR space and begins a conversation with the AI ​​character, for example proposing marriage by saying, "Will you marry me?"

[1515] The device captures the user's voice with a microphone and tracks the avatar's movements with sensors, and transmits this data to a server in real time.

[1516] Step 6:

[1517] The server converts the received voice data into text using voice recognition software (e.g., Google Cloud Speech-to-Text), and also prepares the movement data for analysis.

[1518] This data is input into an artificial intelligence model (e.g., IBM Watson, OpenAI GPT-4) and analyzed, focusing on the user's language, facial expressions, and behavior.

[1519] Step 7:

[1520] The server generates specific feedback based on the analysis results of the AI ​​model, such as advice like "Speak with more confidence" and an evaluation score.

[1521] The generated feedback and evaluation score are transmitted to the terminal.

[1522] Step 8:

[1523] The device displays the received feedback and rating score to the user, allowing them to review their performance and understand areas for improvement.

[1524] The server stores the feedback and rating scores in a database for future reference by the user.

[1525] (Application example 1)

[1526] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1527] Improving service quality in customer service is a major challenge in today's industry. Training and skill development for new employees is particularly important, but on-site training has its limitations, making it difficult to find effective methods. Furthermore, there is a lack of systems for providing effective and instant feedback. To address these challenges, there is a demand for new training systems using virtual reality (VR) technology.

[1528] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1529] In this invention, the server includes a means for setting up a VR environment, a means for capturing a user's voice and movements in real time, a means including a generative AI model for analyzing the captured voice and movement data and generating feedback, a means for displaying the feedback to the user, a means for saving the generated feedback and evaluation score, and a means for the user to practice customer service in the VR environment, generate prompt sentences, and analyze the results, thereby enabling practical and effective customer service training in a virtual environment.

[1530] A "VR environment" is an environment that uses virtual reality technology to create and experience virtual spaces and situations that users can immerse themselves in.

[1531] "Capturing user voice and movements" refers to recording the voices uttered by the user and the body movements and gestures made by the user in real time.

[1532] A "generative AI model" is a model that uses artificial intelligence technology to analyze captured data and provide feedback and evaluation.

[1533] "Displaying feedback" refers to providing the generated feedback or evaluation results to the user visually or audibly.

[1534] "Storing evaluation scores" means recording the generated evaluation results in a database or storage device so that they can be referenced later.

[1535] A "prompt" is a textual question or instruction input to an AI model that guides the analysis and generation process.

[1536] "Customer service training" is a practice process for improving the skills and manners required for customer service work, in which the user simulates customer service.

[1537] "Real-time text transcription" is the process of instantly converting voice data into text.

[1538] "Setting the situation and location" means that the user selects the specific scene and location of the simulation and then constructs the VR environment based on that setting.

[1539] User Registration and Login

[1540] First, a user launches an application on a VR-compatible smartphone or head-mounted display (HMD) and creates a new account. The required information (e.g., email address, username, password) is entered and sent to the server. The server uses the received information to create a user account and stores it in a database. Users who already have an account can enter their email address and password on the login screen, receive authentication from the server, and access the system.

[1541] Setting the VR situation

[1542] After logging in, the user selects a simulation such as "Customer Service Training" from the menu. A specific scene (e.g., a store interior or counter) is set, and the server sends the relevant VR data to the device. The device displays the set VR environment to the user, and the user controls an avatar in the VR space.

[1543] Start the conversation simulation

[1544] The user controls an avatar in the VR space and starts a simulated conversation with an AI character. The user's voice and movements are captured in real time by the device and transmitted to the server.

[1545] AI analysis and feedback

[1546] The server converts the received voice data into text and analyzes it along with the behavioral data using a generative AI model. Based on the analysis results, the generative AI model generates feedback and an evaluation score. The feedback includes specific advice (e.g., "Speak with more confidence").

[1547] View and save feedback

[1548] The generated feedback and evaluation score are sent from the server to the device and displayed to the user, allowing the user to check their performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later, allowing the user to continuously improve their skills.

[1549] Program processing

[1550] The system is built using Python programs and open source libraries (SpeechRecognition, Transformers, etc.). Speech data is converted to text using the SpeechRecognition library and analyzed by a generative AI model (e.g., OpenAI's GPT series). The generated feedback and evaluation scores are displayed on the user interface. The results of the user's simulations are stored in a database.

[1551] Specific example explanation

[1552] For example, if a user wants to simulate "in-store customer service training," they launch the application and log in. By selecting "Customer Service Training" and "In-Store" from the menu, the device will display the corresponding VR environment. The user puts on the HMD and controls an avatar in the VR environment to begin a conversation with an AI character. Voice data is captured in the form of "Please tell me more about this product" and sent to the server. The server converts the voice into text and analyzes it with a generative AI model. Based on the results, feedback such as "Please explain more calmly" is generated, and an evaluation score is calculated and presented to the user. The user can then run the simulation again based on this feedback.

[1553] Prompt Sentence Examples

[1554] "Username: testuser

[1555] Scene: Customer service in store

[1556] Dialogue: Tell me more about this product."

[1557] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1558] Step 1:

[1559] User Registration and Login

[1560] A user starts the application on their device and enters their email address, username, and password on the account registration screen. The server receives this input data, creates a user account, and stores it in the database. A user with an existing account enters their email address and password on the login screen, receives authentication from the server, and accesses the system.

[1561] Input: User information (email address, username, password)

[1562] Output: Account creation or system login privileges

[1563] Step 2:

[1564] Setting the VR situation

[1565] After logging in, the user selects a simulation scene, such as "Customer Service Training," from the device's menu. The user sets a specific scene and location (e.g., inside a store or at a counter). The server sends relevant VR data based on the selected scene and location to the device, and the device uses that data to display the VR environment.

[1566] Input: Simulation scene and location selection

[1567] Output: Sending and displaying selected VR data

[1568] Step 3:

[1569] Start the conversation simulation

[1570] The user controls an avatar in the VR environment and begins to interact with the AI ​​character. The device captures the user's voice and actions in real time and sends the data to the server.

[1571] Input: User voice and movement data

[1572] Output: Sending captured data in real time

[1573] Step 4:

[1574] AI-based analysis and feedback generation

[1575] The server converts the received voice data into text and inputs it into a generative AI model along with the motion data. The generative AI model analyzes this data and generates feedback and an evaluation score. This feedback includes specific advice, such as "Speak more confidently."

[1576] Input: Voice data, motion data

[1577] Output: Feedback and rating score

[1578] Step 5:

[1579] View and save feedback

[1580] The generated feedback and evaluation score are sent from the server to the device and displayed to the user. Based on this feedback, the user can check their own performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later.

[1581] Input: Feedback and rating scores

[1582] Output: View and save feedback

[1583] Example prompt sentences:

[1584] "Username: testuser

[1585] Scene: Customer service in store

[1586] Dialogue: Tell me more about this product."

[1587] By following the steps above, a customer service training system can be implemented, which helps users improve their customer service skills in a practical environment.

[1588] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1589] The present invention is a system in which a user simulates confessing or proposing in a virtual reality (VR) environment, analyzes and evaluates the process using artificial intelligence (AI), and provides feedback by combining it with an emotion engine. A detailed description of specific embodiments of the present invention is provided below.

[1590] System Overview

[1591] User Registration and Login

[1592] A user starts an application and registers a new account. The user enters the required information (e.g., email address, username, password) and sends it to the server. The server creates a user account based on the received information and saves it in the database. A user with an existing account enters their email address and password on the login screen, is authenticated by the server, and logs into the system.

[1593] Setting the VR situation

[1594] After logging in, users can select a "confession" or "proposal" situation from the menu and then specify a specific location (e.g., a park, a restaurant, or their own home). Based on this, the server sends the relevant VR data to the device, which then displays the VR environment to the user.

[1595] Start the conversation simulation

[1596] The user operates an avatar in the VR space and begins to interact with the AI ​​character. The user's voice and actions are captured in real time by the device and sent to the server. The server converts this voice data into text and uses it for analysis along with the movement data.

[1597] Analysis and feedback by AI and emotion engine

[1598] The server analyzes the received voice and behavior data using an AI model and emotion engine. The emotion engine analyzes the user's tone of voice, facial expressions, and body movements to estimate their emotions. Based on this, the AI ​​generates feedback and calculates an evaluation score. For example, specific advice such as "It would be more effective if you smiled more" or "Speaking with confidence will help the other person understand you" is generated.

[1599] View and save feedback

[1600] The generated feedback and evaluation score are sent from the server to the device and displayed to the user. Based on this, the user can check their own performance and understand areas for improvement. Furthermore, the evaluation score and feedback history are saved on the server and can be referenced later. This allows the user to check their progress and use the information to improve their next simulation.

[1601] Specific examples

[1602] For example, if a user wants to simulate a marriage proposal at a restaurant, they first launch the application and log in. After selecting "Proposal" and "Restaurant" from the menu, the device displays the corresponding VR environment. The user puts on VR goggles and controls an avatar in the VR environment to begin a conversation with an AI character. When the user utters the proposal phrase, "Will you marry me?", the device captures the audio and sends it to the server. The server converts the audio into text and analyzes it using an emotion engine. Based on the analysis results, feedback such as "Speak with more confidence" or "Try to look closely at the other person's face" is generated and presented to the user along with an evaluation score. The user can then repeat the simulation based on this feedback to improve their skills.

[1603] Thus, the present invention provides an effective and practical system combined with an emotion engine that allows users to practice confession and proposal situations with a high success rate.

[1604] The processing flow will be explained below.

[1605] Step 1:

[1606] The user launches the application and is presented with a new registration or login screen.

[1607] Step 2:

[1608] The user selects new registration and enters the required information (email address, username, password).

[1609] The terminal transmits the input information to the server.

[1610] Step 3:

[1611] The server stores the received information in a database and notifies the terminal that the account creation is complete.

[1612] Step 4:

[1613] The user enters their email address and password on the login screen and clicks "Login."

[1614] The terminal transmits the input information to the server.

[1615] Step 5:

[1616] The server checks the login information, and if authentication is successful, starts a session and notifies the terminal.

[1617] The device displays a menu screen.

[1618] Step 6:

[1619] The user selects a "confession" or "proposal" situation from the menu, and then selects a specific location (e.g., park, restaurant, home).

[1620] Step 7:

[1621] The terminal transmits the selected situation and location data to the server.

[1622] The server sends the relevant VR data back to the device.

[1623] Step 8:

[1624] The device creates a VR environment based on the returned data and displays it to the user, who then puts on the VR goggles and begins the simulation.

[1625] Step 9:

[1626] Users control an avatar in a VR environment and begin interacting with AI characters, with their voices and actions captured in real time.

[1627] Step 10:

[1628] The device converts the captured audio into text in real time and sends it to the server along with the motion data.

[1629] Step 11:

[1630] The server analyzes the received voice and movement data using an AI model and emotion engine, which analyzes the user's tone of voice, facial expressions, and body movements to estimate the user's emotions.

[1631] Step 12:

[1632] The server generates specific feedback based on the analysis results, calculates an evaluation score, and sends the generated feedback and evaluation score back to the device.

[1633] Step 13:

[1634] The device displays the feedback and evaluation score to the user, who can then review the feedback and identify areas for improvement for the next simulation.

[1635] Step 14:

[1636] The server stores the generated evaluation scores and feedback history in a database for users to refer to later. Users can use this stored data to prepare for marriage proposals or confessions.

[1637] Step 15:

[1638] The user then runs the simulation again based on the feedback and implements the suggested improvements. By repeating this process, the user can improve their skills.

[1639] Example 2

[1640] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1641] Conventional VR simulation systems lack the ability to capture and analyze users' voices and movements in real time, making it difficult to accurately analyze users' emotions and subtle nuances. Furthermore, the quality of the feedback is low, making it difficult for users to effectively learn and improve. Therefore, there is a demand for a system that can provide users with high-quality feedback by capturing users' voices and movements in real time and analyzing them using an emotion engine.

[1642] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for setting a VR environment, means for capturing the user's voice and movements in real time, means including an artificial intelligence model for analyzing the captured voice and movement data and generating feedback, means including an emotion engine for analyzing emotions, means for displaying feedback and an evaluation score to the user, and means for saving the generated feedback and evaluation score. This allows the user to analyze their own performance in real time and receive high-quality feedback, thereby enabling them to effectively learn and improve their skills.

[1643] A "VR environment" is a computer-generated simulation space that allows users to experience visual and auditory sensations in a virtual reality space.

[1644] "Means for capturing user voice and movements in real time" refers to devices or software that instantly record a user's voice and body movements as digital data and transmit that information to the system.

[1645] "Means including an artificial intelligence model for analyzing captured voice and movement data and generating feedback" refers to a machine learning algorithm or AI engine for processing and analyzing collected voice and movement data and providing improvement advice to the user based on the results.

[1646] "Emotion engine for analyzing emotions" refers to software and algorithms that analyze a user's tone of voice, facial expressions, body movements, etc. to infer the user's emotional state.

[1647] The "means for displaying feedback and evaluation scores to the user" refers to a device or display screen that allows the user to visually confirm the generated advice and evaluation results.

[1648] "Means for storing generated feedback and evaluation scores" refers to hardware or software that records the analysis results and feedback in a database or storage system for future reference.

[1649] "Means for configuring specific situations and locations in a VR environment" refers to the interface and menu system that allows a user to select and configure the scenarios and locations they experience in virtual reality.

[1650] "Means for creating a VR environment based on a specific situation and location and displaying it to the user" refers to software that generates a virtual reality space based on a scenario or location set by the user and outputs it to a display device so that the user can experience that environment.

[1651] "Means for converting text in real time" refers to speech recognition technology and its implementation for instantly converting a user's voice into text data.

[1652] "Means for transmitting data to a server" refers to a communications protocol and its implementation for transmitting the captured digital data over a network to a server.

[1653] "Means for analyzing and generating feedback and evaluation scores" refers to software and algorithms that analyze the received data using AI models and emotion engines to generate specific advice and evaluation results to provide to users.

[1654] The present invention is a system in which a user simulates confessing or proposing in a virtual reality (VR) environment, analyzes and evaluates the process using artificial intelligence (AI), and provides feedback by combining it with an emotion engine. A detailed description of specific embodiments of the present invention is provided below.

[1655] User Registration and Login

[1656] A user starts an application and accesses the new registration screen. The user enters their email address, username, and password, and the device sends the entered information to the server. The server creates a new user account based on the received information and saves it in the database. An existing user enters their email address and password on the login screen, and the device sends the entered authentication information to the server. The server checks the information in the database and authenticates the user. After successful authentication, the server issues a login session and sends it to the device.

[1657] Setting the VR situation

[1658] After logging in, the user selects a simulation situation such as "confession" or "proposal" from the menu. The user sets a specific location, such as a park, restaurant, or home. The device sends the selected situation information to the server. The server then sends the corresponding VR data (video, audio, 3D environment data, etc.) to the device, which receives it and displays the VR environment that the user has set.

[1659] Start the conversation simulation

[1660] The user puts on VR goggles, enters the virtual reality environment, and controls an avatar. The user begins to interact with the AI ​​character. The device captures the user's voice and actions (body movements and gestures) in real time and transmits them to the server.

[1661] Analysis and feedback by AI and emotion engine

[1662] The server converts the received voice data into text. Specifically, it uses voice recognition software (e.g., Google Cloud Speech-to-Text). The server then passes the converted text and behavioral data to an AI model for analysis. The AI ​​models used here include natural language processing (NLP) models and computer vision models. Based on the analysis results, the emotion engine analyzes the user's tone of voice, facial expressions, body movements, etc. to infer their emotions. The server combines the results of the emotion engine and the AI ​​model to generate feedback and calculate an evaluation score. Examples of generated feedback include "It would be more effective if you smile more" and "Speaking with confidence helps you reach the other person."

[1663] View and save feedback

[1664] The server sends the generated feedback and evaluation score to the terminal and displays it to the user. The user can check the feedback and use it to improve their performance. The feedback history and evaluation score are saved on the server, so the user can refer to them later to check their progress and use them for the next simulation.

[1665] Specific examples

[1666] For example, if a user wants to simulate a marriage proposal at a restaurant, they launch the application and log in. By selecting "Proposal" and "Restaurant" from the menu, the device will display the corresponding VR environment. The user puts on VR goggles and controls an avatar in the VR environment to begin a conversation with an AI character. When the user utters the proposal phrase, "Will you marry me?", the device captures the audio and sends it to the server. The server converts the audio into text and analyzes it using an emotion engine. Based on the analysis results, feedback such as "Speak with more confidence" or "Try to look closely at the other person's face" is generated and presented to the user along with an evaluation score.

[1667] Prompt Sentence Examples

[1668] "Start a simulation of proposing at a restaurant"

[1669] "I want to practice the confession scene in the park."

[1670] The present invention thus provides an effective and practical system, combined with an emotion engine, that allows users to practice confession and proposal situations with a high success rate.

[1671] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1672] Step 1:

[1673] The user launches the application and accesses the new registration screen.

[1674] Input: Enter your email address, username, and password.

[1675] The terminal transmits this input information to the server.

[1676] Output: The user information sent to the server.

[1677] Step 2:

[1678] The server creates a new user account based on the received information and stores it in the database.

[1679] Input: User information sent from the device.

[1680] Output: A confirmation message for creating a new account.

[1681] Step 3:

[1682] An existing user enters their email address and password on the login screen.

[1683] Enter your email address and password.

[1684] The terminal transmits the entered authentication information to the server.

[1685] Output: The authentication information sent to the server.

[1686] Step 4:

[1687] The server authenticates the user by checking the credentials against a database.

[1688] Input: Received credentials.

[1689] Output: Issue of login session.

[1690] Step 5:

[1691] After successful authentication, the server issues a login session and sends it to the terminal.

[1692] Input: The authentication result by the server.

[1693] Output: Login session information.

[1694] Step 6:

[1695] After the user logs in, they select a simulation situation from the menu.

[1696] Input: Simulation selection (confession, proposal, etc.) and location setting (park, restaurant, home, etc.).

[1697] The terminal transmits the selected situation information to the server.

[1698] Output: The situation information sent to the server.

[1699] Step 7:

[1700] The server sends the corresponding VR data (video, audio, 3D environment data, etc.) to the terminal.

[1701] Input: Simulation situation information.

[1702] Output: VR data sent to the device.

[1703] Step 8:

[1704] The device displays the VR environment set for the user based on the VR data received.

[1705] Input: VR data sent from the server.

[1706] Output: The VR environment displayed to the user.

[1707] Step 9:

[1708] The user puts on VR goggles and enters the virtual reality environment to control an avatar.

[1709] Input: User interaction within the VR environment.

[1710] Output: User actions with avatar.

[1711] Step 10:

[1712] The user begins a dialogue with the AI ​​character.

[1713] Input: User speech and movements.

[1714] The device captures the user's voice and movements in real time and transmits them to the server.

[1715] Output: Audio and movement data sent to the server.

[1716] Step 11:

[1717] The server converts the received voice data into text.

[1718] Input: Received audio data.

[1719] Output: Text-converted audio data.

[1720] Step 12:

[1721] The server passes the converted data and behavioral data to the AI ​​model for analysis.

[1722] Input: Textual data, behavioral data.

[1723] Output: Analysis results.

[1724] Step 13:

[1725] Based on the analysis results, the emotion engine analyzes the user's tone of voice, facial expressions, body movements, etc. to estimate their emotions.

[1726] Input: Analysis results, voice tone, facial expressions, body movements.

[1727] Output: Emotion analysis results.

[1728] Step 14:

[1729] The server combines the results of the emotion engine and AI model to generate feedback and calculate an evaluation score.

[1730] Input: Sentiment analysis results, AI model results.

[1731] Output: Feedback and rating score.

[1732] Step 15:

[1733] The server generates feedback and sends the rating score to the device.

[1734] Input: Feedback and rating score.

[1735] Output: Feedback and rating score sent to the device.

[1736] Step 16:

[1737] The device displays the received feedback and rating score to the user.

[1738] Input: Feedback and rating score sent by the server.

[1739] Output: The feedback and rating score displayed to the user.

[1740] Step 17:

[1741] Users can view feedback and use it to improve their performance.

[1742] Input: The displayed feedback and rating score.

[1743] Output: User improvement actions.

[1744] Step 18:

[1745] Feedback history and rating scores are stored on the server.

[1746] Input: Generated feedback and rating scores.

[1747] Output: Feedback history and rating scores stored on the server.

[1748] This allows users to analyze their performance in real time and receive high-quality feedback, enabling them to effectively learn and improve their skills.

[1749] (Application example 2)

[1750] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1751] Improving customer satisfaction in modern brick-and-mortar stores requires employees to have high customer service skills. However, new and inexperienced employees have limited opportunities to practice in real-life customer service situations, and nervousness and mistakes made with real customers often hinder their learning. Furthermore, traditional training methods make it difficult to provide individual feedback quickly, and there is a lack of efficient ways to promote employee skill improvement. To solve these issues, the development of a customer service training system using VR is desired.

[1752] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for setting up a VR environment, means for capturing the user's voice and movements in real time, means including an artificial intelligence model for analyzing the captured voice and movement data and generating feedback, means for displaying the feedback to the user, means for saving the generated feedback and evaluation score, and means for creating a virtual training scenario for employees of a physical store to practice their customer service skills. This allows employees to effectively practice in a virtual reality customer service scenario and improve their skills while receiving specific feedback.

[1753] A "means for setting up a VR environment" is a device or program that creates a virtual reality space and provides an environment for users to experience a specific situation or location.

[1754] "Means for capturing user voice and movements in real time" refers to a device or program that instantly records the user's voice and body movements and provides the data for subsequent analysis.

[1755] "Means including an artificial intelligence model for analyzing captured voice and movement data and generating feedback" refers to a system that uses artificial intelligence to analyze recorded voice and movement data and provides the user with guidelines for improvement based on the results.

[1756] The "means for displaying feedback to the user" refers to a device or program that presents the analysis results to the user in an appropriate form and provides information that allows the user to check and improve their own performance.

[1757] The "means for storing the generated feedback and evaluation score" refers to a device or program that records the feedback provided to the user and the evaluation results, and stores them in a form that can be referenced later.

[1758] "A means for creating virtual training scenarios for brick-and-mortar store employees to practice customer service skills" is a system that recreates customer service situations in brick-and-mortar stores in virtual reality, and designs and provides scenarios to train employees to respond appropriately in the moment.

[1759] "Means for selecting situations and locations for practicing conversations and actions in customer service situations in a physical store" refers to a device or program that allows the user to select a specific customer service situation or environment and set up an appropriate practice scenario based on that selection.

[1760] The "means for generating specific advice as feedback to improve the user's customer service skills" is a system that provides specific guidelines on how the user should improve their customer service based on analyzed data.

[1761] This invention relates to a system that uses virtual reality (VR) to train employees in customer service skills in brick-and-mortar stores. Hereinafter, specific embodiments of the invention will be described in detail.

[1762] System configuration

[1763] The system mainly consists of the following elements:

[1764] 1. How to set up a VR environment

[1765] The server uses Unity to create a virtual reality space, providing an environment in which users can practice customer service in specific situations and locations.

[1766] 2. A means of capturing the user's voice and actions in real time

[1767] It uses the camera and microphone built into the user's smartphone or VR goggles to instantly record voice and movement.

[1768] 3. Means including an artificial intelligence model for analyzing the captured audio and motion data and generating feedback.

[1769] The server uses artificial intelligence models trained in TensorFlow and OpenCV to analyze the recorded voice and movement data and generate feedback for the user.

[1770] 4. How to display feedback to the user

[1771] Feedback is displayed on the smartphone or VR goggles screen using React Native, allowing users to see the analysis results in real time.

[1772] 5. Means of storing generated feedback and rating scores

[1773] The feedback and evaluation scores are stored on a server and made available to users for future reference, allowing them to track their progress and understand areas for improvement.

[1774] 6. A way to create virtual training scenarios for brick-and-mortar store associates to practice customer service skills

[1775] The server provides scenarios that recreate specific customer service situations in a real store (e.g., greeting customers at a restaurant), allowing users to practice in situations that are close to the real situation.

[1776] System Operation

[1777] 1. Setting up the VR environment

[1778] The server uses Unity to create a virtual environment based on the situation and location selected by the user, allowing the user to begin a customer service simulation in virtual reality.

[1779] 2. Audio and motion capture

[1780] The camera and microphone built into the user's smartphone or VR goggles capture the user's voice and movements in real time and transmit them to a server.

[1781] 3. Analyzing data and generating feedback

[1782] The server analyzes the captured voice and motion data using TensorFlow and OpenCV to evaluate the user's emotions and performance, and generates specific feedback for the user based on the analysis results.

[1783] 4. Viewing Feedback

[1784] The generated feedback and rating scores are displayed on the user's smartphone or VR goggles using React Native, allowing the user to improve their performance.

[1785] 5. Data storage

[1786] Feedback and evaluation scores are stored on a server for future reference, allowing users to track their progress and continually improve their skills.

[1787] Examples of concrete examples and prompts

[1788] For example, if a user wants to simulate "greeting customers at a restaurant," the following prompt sentences are exemplified:

[1789] "Start a simulation of greeting a customer in a VR environment. The scenario is 'restaurant'."

[1790] "Hello, welcome! My name is [Name]. I'll show you to your seat."

[1791] This allows users to practice in situations that are similar to actual customer service situations and improve their skills while receiving specific feedback.

[1792] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1793] Step 1:

[1794] A user launches an application and enters their email address and password on the login screen. The input data is sent to the server, which then performs the authentication process. If authentication is successful, the user is redirected to the main menu. The input is an email address and password, and the output is the authentication result (success or failure).

[1795] Step 2:

[1796] The user selects "Customer Service Training" from the main menu. Then, the training situation and location (e.g., greeting customers at a restaurant) are set. The set situation information is sent to the server. The set situation information is the input, and the setting information is sent to the server as the output.

[1797] Step 3:

[1798] The server uses Unity to create a virtual reality (VR) environment based on the situation and location selected by the user. The created VR environment data is sent to the user's device. The input is configuration information, and the output is VR environment data that is generated and sent to the user's device.

[1799] Step 4:

[1800] The user puts on the VR goggles and starts the simulation in the VR environment. The user's movements and voice are captured in real time and sent from the device to the server. The input is the user's movements and voice, and the captured data is sent to the server as the output.

[1801] Step 5:

[1802] The server uses TensorFlow to analyze the captured voice and motion data, which generates emotion and performance data for the user. The input is the captured data, and the output is the emotion and performance data.

[1803] Step 6:

[1804] The server generates specific feedback based on the analysis results. The generated feedback and evaluation score are sent to the device. The input is emotion and performance data, and the output is feedback and evaluation score.

[1805] Step 7:

[1806] The device uses React Native to display feedback and a rating score to the user, allowing the user to see their performance and understand areas for improvement. The input is the feedback and rating score, and the output is what is displayed to the user.

[1807] Step 8:

[1808] The generated feedback and rating score are stored on the server for future reference. The input is the feedback and rating score, and the output is stored in a database.

[1809] Step 9:

[1810] Users can then replay the simulation and improve their customer service skills based on the feedback. Users can practice repeatedly at their own pace and monitor their progress while honing their skills. The input is the previous feedback and evaluation score, and the output is improved performance.

[1811] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1812] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1813] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1814] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1815] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1816] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1817] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1818] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1819] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1820] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1821] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1822] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1823] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1824] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1825] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1826] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1827] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1828] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1829] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1830] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1831] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1832] The following is further disclosed regarding the above embodiment.

[1833] (Claim 1)

[1834] A means of setting up the VR environment,

[1835] means for capturing the user's voice and movements in real time;

[1836] means including an artificial intelligence model for analyzing the captured audio and motion data and generating feedback;

[1837] a means for displaying feedback to the user;

[1838] a means for storing the generated feedback and rating scores;

[1839] A system including:

[1840] (Claim 2)

[1841] A means for users to select a confession or proposal simulation and set a specific situation and location in the VR environment;

[1842] A means of constructing and displaying a VR environment based on a specific situation and location to the user;

[1843] The system of claim 1 further comprising:

[1844] (Claim 3)

[1845] A means for converting data captured from the user's voice and actions into text in real time and transmitting the text to a server;

[1846] means for analyzing the received data and generating feedback and a rating score;

[1847] The system of claim 1 further comprising:

[1848] "Example 1"

[1849] (Claim 1)

[1850] a means for users to register new accounts and log in;

[1851] A means of setting up the VR environment,

[1852] a means for constructing and displaying a VR environment based on a user-selected situation and location;

[1853] means for capturing the user's voice and movements in real time;

[1854] means for converting the captured voice data into text and analyzing it together with the motion data, the means including an artificial intelligence model;

[1855] means for generating a feedback and evaluation score based on the analysis results;

[1856] means for displaying the generated feedback and rating score to the user;

[1857] a means for storing the generated feedback and rating scores;

[1858] A system including:

[1859] (Claim 2)

[1860] A means for users to select a confession or proposal simulation and set a specific situation and location in the VR environment;

[1861] A means of constructing and displaying a VR environment based on a specific situation and location to the user;

[1862] The system of claim 1 further comprising:

[1863] (Claim 3)

[1864] A means for converting data captured from the user's voice and actions into text in real time and transmitting the text to a server;

[1865] means for analyzing the received data and generating feedback and a rating score;

[1866] The system of claim 1 further comprising:

[1867] "Application Example 1"

[1868] (Claim 1)

[1869] A means of setting up the VR environment,

[1870] means for capturing the user's voice and movements in real time;

[1871] means for analyzing the captured audio and motion data and generating feedback, the means including a generative AI model;

[1872] a means for displaying feedback to the user;

[1873] a means for storing the generated feedback and rating scores;

[1874] A means for users to practice customer service in a VR environment, generate prompt sentences, and analyze the results.

[1875] A system including:

[1876] (Claim 2)

[1877] A means for users to select confession, proposal, and customer service training simulations and set specific situations and locations in the VR environment;

[1878] A means of constructing and displaying a VR environment based on a specific situation and location to the user;

[1879] The system of claim 1 further comprising:

[1880] (Claim 3)

[1881] A means for converting data captured from the user's voice and actions into text in real time and transmitting the text to a server;

[1882] means for analyzing the received data and generating feedback and a rating score;

[1883] A means for generating prompt sentences and analyzing them using an AI model;

[1884] The system of claim 1 further comprising:

[1885] "Example 2: Combining Emotion Engines"

[1886] (Claim 1)

[1887] A means of setting up the VR environment,

[1888] means for capturing the user's voice and movements in real time;

[1889] means including an artificial intelligence model for analyzing the captured audio and motion data and generating feedback;

[1890] means for analyzing emotions, the means including an emotion engine;

[1891] means for displaying the feedback and rating score to the user;

[1892] a means for storing the generated feedback and rating scores;

[1893] A system including:

[1894] (Claim 2)

[1895] A means for users to select a confession or proposal simulation and set a specific situation and location in the VR environment;

[1896] A means of constructing and displaying a VR environment based on a specific situation and location to the user;

[1897] The system of claim 1 further comprising:

[1898] (Claim 3)

[1899] A means for converting data captured from the user's voice and actions into text in real time and transmitting the text to a server;

[1900] means for analyzing the received data and generating feedback and a rating score;

[1901] The system of claim 1 further comprising:

[1902] "Application example 2 when combining emotion engines"

[1903] (Claim 1)

[1904] A means of setting up the VR environment,

[1905] means for capturing the user's voice and movements in real time;

[1906] means including an artificial intelligence model for analyzing the captured audio and motion data and generating feedback;

[1907] a means for displaying feedback to the user;

[1908] a means for storing the generated feedback and rating scores;

[1909] A means to create virtual training scenarios for brick-and-mortar store associates to practice customer service skills; and

[1910] A system including:

[1911] (Claim 2)

[1912] A means for users to select a confession or proposal simulation and set a specific situation and location in the VR environment;

[1913] A means of constructing and displaying a VR environment based on a specific situation and location to the user;

[1914] A means of selecting situations and locations for practicing conversations and actions in customer service situations in a brick-and-mortar store;

[1915] A means of constructing and displaying a VR environment based on a specific situation and location to the user;

[1916] The system of claim 1 further comprising:

[1917] (Claim 3)

[1918] A means for converting data captured from the user's voice and actions into text in real time and transmitting the text to a server;

[1919] means for analyzing the received data and generating feedback and a rating score;

[1920] A means for generating specific advice as feedback for improving the user's customer service skills;

[1921] The system of claim 1 further comprising: [Explanation of symbols]

[1922] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of setting up the VR environment, means for capturing the user's voice and movements in real time; means including an artificial intelligence model for analyzing the captured audio and motion data and generating feedback; a means for displaying feedback to the user; a means for storing the generated feedback and rating scores; A system including:

2. A means for users to select a confession or proposal simulation and set a specific situation and location in the VR environment; A means of constructing and displaying a VR environment based on a specific situation and location to the user; The system of claim 1 further comprising:

3. A means for converting data captured from the user's voice and actions into text in real time and transmitting the text to a server; means for analyzing the received data and generating feedback and a rating score; The system of claim 1 further comprising:

Images