system

Abstract

We provide the system. [Solution] A means of receiving authentication information from the user terminal and performing the authentication procedure, A means of sending virtual environment data to the user terminal after authentication and generating a virtual space, A means of incorporating artificial intelligence that communicates with a user terminal using text, voice, or gestures, A means for analyzing a user's transaction history and interaction history to generate individually optimized financial product information, A means for transmitting generated financial product information to the user terminal, A means for receiving procedural requests from users, generating related documents, and sending them to the user's terminal, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a need to overcome the constraint that consumers have to physically visit a bank branch, especially the inconvenience caused by geographical factors and time constraints. Also, due to the increase in digital channels, there is a lack of support for the elderly and users who are not accustomed to digital devices, along with the personalized experience in face-to-face customer service. Furthermore, the bank side needs a method to deepen the relationship with customers while reducing the operating costs of branches.

Means for Solving the Problems

[0005] This invention provides a system characterized by performing user authentication using authentication information from a user terminal and generating a virtual space based on the authentication result. This system is equipped with artificial intelligence functions that communicate with the user using text, voice, or gestures. Furthermore, it analyzes the user's transaction history, generates personalized financial product information, and transmits it to the user terminal. This makes it possible to provide a near-realistic experience without requiring a direct visit to a branch, and also enhances support for elderly users and users unfamiliar with digital technology.

[0006] A "user terminal" is an electronic device used by a user, and serves as an access and interface device for this system.

[0007] "Authentication information" refers to information used to verify a user's identity, and includes IDs, passwords, or biometric authentication data.

[0008] "Authentication process" refers to the process of verifying a user's identity based on their authentication information and granting them access.

[0009] "Virtual environment data" refers to information used to generate a virtual space experienced by a user, and includes visual and sensory elements.

[0010] A "virtual space" is a computer-generated, three-dimensional space that users can experience.

[0011] "Artificial intelligence" refers to algorithms and software designed to mimic or assist human intellectual tasks.

[0012] "Transaction history" refers to a record of a user's past financial transactions and is data used to generate future suggestions and information.

[0013] "Financial product information" refers to information about financial products provided to customers, including detailed data such as conditions and benefits. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The virtual bank branch system according to the present invention is designed to meet the diverse needs and environments of various users. Users install a dedicated application on their user terminal and communicate with the server via the internet.

[0036] First, the user launches the app and enters their authentication information, which the device then sends to the server. The server verifies the received authentication information against its database, and only if authentication is successful does it send virtual environment data to the user. Using this virtual environment data, the device utilizes AR / VR technology to generate a three-dimensional virtual bank branch for the user.

[0037] In the generated virtual space, an AI bank teller is on standby, and users can communicate with the AI ​​via text, voice, or gestures, as if they were in a real branch. The user's voice and text input are processed by the terminal and sent to the server. The AI ​​on the server identifies the user's requests using advanced analytical capabilities and sends the optimal response back to the terminal in real time.

[0038] Furthermore, this system can analyze a user's past transaction history and provide individually optimized financial product information. When a user wishes to proceed with specific procedures such as opening a new account or consulting about a loan, the server automatically generates the necessary information and provides appropriate forms and documents through the user's terminal. The user completes the procedure online by filling out these documents and submitting them to the server.

[0039] For example, if a user wishes to open a new account, they can inform the AI ​​banker of this, and the server will send the necessary procedural information to the user's terminal. The user can then complete the process by entering the required information into a form.

[0040] Thus, the present invention provides convenient and efficient banking services that transcend geographical and temporal constraints, and is designed to be easily used, especially by elderly people who are unfamiliar with digital technology, through an intuitive interface.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The user launches the virtual bank branch application on their terminal. The user enters their authentication information (e.g., user ID and password, or biometric authentication data) on the authentication screen.

[0044] Step 2:

[0045] The terminal receives the entered authentication information, encrypts it, and then sends it to the server. The server compares the received authentication information with its database to determine whether the authentication was successful or not.

[0046] Step 3:

[0047] If the server confirms successful authentication, it generates virtual environment data and sends it to the terminal. The terminal receives the virtual environment data and uses AR / VR technology to generate a virtual space that is reflected in the user's vision.

[0048] Step 4:

[0049] The user visually recognizes an AI bank teller in a virtual space and initiates a request via text or voice. The terminal transcribes the user's statements and input into text and sends that data to the server.

[0050] Step 5:

[0051] The server uses generative AI to analyze the user's intent and generate appropriate responses and suggestions. The generated responses are then sent to the terminal and provided to the user in real time.

[0052] Step 6:

[0053] If a user requests details or procedural information for a specific financial service, the server will refer to the user's transaction history and prepare individually optimized product information and procedural steps. It will then send the necessary digital documents to the user's device.

[0054] Step 7:

[0055] The terminal displays generated suggestions and required documents to the user, allowing the user to ask additional questions about unclear points and enter information into the documents. If the user proceeds with the procedure, the process is completed by sending the entered information back to the server.

[0056] This process allows users to receive necessary banking services without physically visiting a bank branch.

[0057] (Example 1)

[0058] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0059] In modern society, there is a demand for efficient and intuitive financial services that transcend geographical and temporal constraints. However, current systems have challenges such as the inconvenience users have to visit physical financial institutions and a lack of adequate support for users unfamiliar with digital technology. In particular, for users with little digital experience, such as the elderly, the complexity of online operations prevents them from fully realizing the benefits of these services.

[0060] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0061] In this invention, the server includes means for receiving identification information from a user information processing device and performing an authentication process, means for transmitting virtual reality data to the user information processing device after authentication and constructing a virtual environment, and means for an information processing device equipped with enhanced intelligence that interacts with the user information processing device using text information, voice information, or motion recognition. This enables users to enjoy intuitive and efficient financial services online without being restricted by their location or time.

[0062] A "user information processing device" is a device operated by a user that has functions to enable the input and display of identification information and the experience of a virtual environment.

[0063] "Identification information" refers to information necessary for user authentication, and typically includes user IDs, passwords, and biometric authentication data.

[0064] The "authentication process" refers to a series of procedures that verify the legitimacy of a user based on identification information received from a user information processing device.

[0065] "Virtual reality data" refers to data transmitted to a user information processing device, and includes information for building a virtual environment and providing an experience to the user.

[0066] A "virtual environment" is a space that provides three-dimensional visual and auditory information generated through information processing, enabling users to have an experience that closely resembles reality.

[0067] An "intelligence-enhanced information processing device" is a device that utilizes artificial intelligence technology to generate the optimal response based on the input information.

[0068] "Textual information, audio information, and motion recognition" refer to various forms of data that users input through information processing devices, and are means of communication.

[0069] "Financial services information" refers to information about financial products and services provided to users, including content optimized to meet user needs.

[0070] "Business history and interaction history" refers to records of transactions and communications when a user uses the system, and is data used to improve the provision of services to users.

[0071] A "storage device" is a device that stores information and makes it easy to retrieve when needed.

[0072] "Response accuracy" is an indicator that shows the ability of artificial intelligence to provide appropriate and accurate responses to user input.

[0073] This invention provides a system in which a user information processing device, a server, and an intelligent information processing device work in cooperation to improve user convenience. This system begins with the user installing a dedicated application on the information processing device. The information processing device includes devices such as smartphones and tablets, which communicate with the server via the internet.

[0074] The server receives identification information from the user information processing device and performs an authentication process. Database matching technology is used in this process to verify the user's identity. Upon successful authentication, the server uses a graphics engine such as OpenGL or Unity to send virtual reality data to the user information processing device. This allows the information processing device to generate a virtual environment and provide the user with a three-dimensional visual experience.

[0075] Within this virtual environment, users can interact with an intelligent information processing device through text, voice, or motion recognition. For example, a user wishing to open a new account might use a prompt such as "I want to open a new account." This input is parsed, and the server generates the most appropriate response to the user's needs and sends it back to the information processing device.

[0076] The enhanced intelligence-based information processing device uses generative AI models and machine learning techniques to analyze the user's business history and dialogue history. This generates and provides the user with individually optimized financial service information. An intuitive interface is provided to support user operation, making it easy to use, especially for users unfamiliar with digital technology.

[0077] Thus, the present invention realizes an environment in which users can receive efficient and intuitive financial services without being restricted by location or time.

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] The user installs a dedicated application on the information processing device. Upon launching the application, the user enters their user ID and password or biometric authentication data as authentication information. The device then sends this information to the server. The entered authentication information is transmitted securely using encryption technology.

[0081] Step 2:

[0082] The server receives authentication information from the terminal and compares it with the registered information in the database. The server uses a database search algorithm to verify that the user's authentication information matches. If authentication is successful, the server generates virtual reality data and sends it to the terminal. At this time, customization is applied based on the user's history.

[0083] Step 3:

[0084] The device generates a virtual environment using a graphics engine (such as OpenGL or Unity) based on the received virtual reality data. Through this virtual environment, the user experiences the appearance of a three-dimensional virtual bank branch right before their eyes. The device utilizes AR / VR technology to provide the user with visual information.

[0085] Step 4:

[0086] Within the virtual environment, users provide input through text, voice, or actions. For example, a user might say, "I want to open a new account." The terminal analyzes this input, converts it into text data using speech recognition technology, and sends it to the server.

[0087] Step 5:

[0088] The server uses a generated AI model based on the analysis of user input to recognize user requests. The AI ​​model performs data calculations (natural language processing) to generate the optimal response to the input prompt. The server generates the response data and sends it back to the terminal.

[0089] Step 6:

[0090] The terminal receives response data from the server and provides feedback to the user. This includes responses using speech synthesis technology and text messages on the screen. The user then decides on their next action based on this feedback. This process enables an environment where financial services can be accessed smoothly from any location.

[0091] (Application Example 1)

[0092] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0093] In modern society, geographical and temporal constraints on accessing financial services remain significant barriers for users. Furthermore, traditional financial procedures often lack sufficient optimization and individualized support for the user experience, creating a demand for efficient and personalized services. In particular, there is a growing need for intuitive interfaces for users unfamiliar with digital environments.

[0094] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0095] In this invention, the server includes means for receiving authentication information from a user device and performing authentication processing; means for transmitting virtual environment data to the user device after authentication and generating a virtual space; means for incorporating a machine learning model that communicates with the user device using text, voice, or actions; means for analyzing the user's operation history and dialogue history and generating individually optimized financial service information; means for transmitting the generated financial service information to the user device; means for receiving procedure requests from the user, generating related documents and transmitting them to the user device; means for generating a virtual financial facility using augmented reality technology; and means for generating voice or screen changes and providing the user with a flow of procedures. This enables users to receive intuitive and personalized financial services without being bound by location or time.

[0096] "User device" refers to a terminal device used by a user, and includes mobile information terminals such as smartphones and tablets.

[0097] "Authentication information" refers to data necessary to verify a user's identity, and includes user IDs, passwords, biometric information, and other similar information.

[0098] "Virtual environment data" refers to a set of information used to generate a digital three-dimensional space provided to the user, and is intended to realize virtual reality and augmented reality experiences.

[0099] A "machine learning model" is a part of a computer program that includes algorithms to optimize user interaction based on data and generate appropriate responses.

[0100] "Financial service information" refers to information about individually optimized financial products and services provided to users, and is generated based on the user's past transaction history and current needs.

[0101] Augmented reality technology is a technique that overlays digital information onto images of the real world, enabling users to interact with virtual objects.

[0102] A "virtual financial facility" is a digitally simulated facility that users can visit within a virtual environment, and it functions as a place where users can receive financial services.

[0103] To implement this invention, the user must first install a dedicated application on their user device, such as a smartphone. This application is designed to enhance the user experience using augmented reality and machine learning technologies. The user device communicates with a server via the internet to send and receive necessary data.

[0104] The server processes the authentication information received from the user terminal and verifies it against the database. Upon successful authentication, the server sends virtual environment data to the user terminal, where a virtual space is created. In this environment, machine learning models are utilized, enabling natural communication with the user.

[0105] The user terminal displays a generated virtual financial facility, allowing the user to visually immerse themselves in that space. AI-powered financial service information is generated on the server based on the user's actions and transaction history, and personalized content is provided to the user. Furthermore, if necessary, the server generates relevant digital documents and provides them through the user terminal.

[0106] The hardware will consist of mobile devices such as smartphones and tablets, while the software will utilize augmented reality development frameworks such as ARKit and ARCore, as well as machine learning engines such as TENSORFLOW®. This system will leverage cloud infrastructure such as AWS® and Google® Cloud for data processing and storage.

[0107] One concrete example is a scenario where a user interacts with an AI banker in a virtual space and opens a new savings account. In this scenario, the AI ​​banker proposes the most suitable plan to the user via voice input, and the necessary procedures are carried out based on the user's selection.

[0108] An example of a prompt to input into the generating AI model might be: "Please describe in detail a scenario in which a user interacts with an AI in a virtual space and opens a new savings account. Pay particular attention to the use of voice input and a visual interface."

[0109] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0110] Step 1:

[0111] The user launches a dedicated application and enters their authentication information. The entered authentication information is sent from the terminal to the server. The server compares this information with existing data in the database and performs authentication. If authentication is successful, the server returns permission to proceed to the next step to the terminal.

[0112] Step 2:

[0113] The server generates virtual environment data and sends it to the user's terminal. Upon receiving this data, the terminal uses augmented reality technology to display a virtual financial facility to the user. The user enters this virtual space to begin a real-time experience.

[0114] Step 3:

[0115] The user interacts with an AI banker in a virtual space. When the user provides voice or text input, the terminal sends it to the server. The server analyzes the input using a generative AI model and generates an appropriate response. The generated response is then returned to the user, and the interaction continues.

[0116] Step 4:

[0117] Based on the user's past operation and interaction history, the server generates individually optimized financial service information. This information is generated by analyzing the user's transaction patterns and needs and processing the data. The resulting information is sent to the terminal and presented to the user.

[0118] Step 5:

[0119] When a user decides to use a specific financial service, the server generates the relevant digital documents. Taking the required data as input, the server automatically fills in the document format and sends the completed version to the user's terminal. The user then completes the requested procedures or applications through these digital documents.

[0120] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0121] The system according to the present invention is built to further personalize the user experience and enable natural communication by incorporating an emotion engine.

[0122] The user first launches a dedicated application installed on their device and enters authentication information to access the system. The device sends the entered authentication information to the server, which performs authentication. If authentication is successful, the server sends virtual environment data to the user's device. Based on the received data, the device generates a realistic virtual space using AR / VR technology. In this virtual space, the user can communicate with an AI banker.

[0123] The AI ​​bank teller can analyze the user's voice and facial expressions through an emotion engine to infer their emotions. Based on this emotion analysis, the server generates an optimal response tailored to the user's emotions and sends it to the terminal. For example, if the AI ​​bank teller determines that the user is nervous, it can respond in a gentle and calm tone to alleviate their anxiety.

[0124] Furthermore, the server integrates the acquired emotional data with the user's past interaction data for deeper analysis. Using the results of this analysis, it becomes possible to individually customize and provide the user with the most suitable financial product information. For example, if a user is interested in a new investment product but also feels anxious during the conversation, the server will select responses that gently explain the details and build trust.

[0125] In this way, by incorporating an emotion engine, this system achieves a more human-like interaction that goes beyond simple question-and-answering, greatly improving user satisfaction.

[0126] The following describes the processing flow.

[0127] Step 1:

[0128] The user launches a dedicated application on their device and enters their authentication information using their user ID and password or biometric authentication on the login screen.

[0129] Step 2:

[0130] The terminal sends the entered authentication information to the server. The server compares the received information with its internal database to determine whether authentication was successful.

[0131] Step 3:

[0132] Upon successful authentication, the server sends virtual environment data to the user's terminal. Based on the received data, the terminal uses AR / VR technology to generate the virtual space that the user will experience.

[0133] Step 4:

[0134] Users interact with an AI bank teller in a virtual space and make requests via voice or text. For example, when requesting a balance check, the terminal converts the voice into text and sends it to the server.

[0135] Step 5:

[0136] The server analyzes the received user voice data using an emotion engine, estimating emotions based on voice tone and user facial expression data. Based on the analysis results, it generates an appropriate response that matches the user's emotions.

[0137] Step 6:

[0138] The server sends the generated response data to the terminal. The terminal then receives a response from the AI ​​bank teller in voice or text format, in a tone that matches the user's emotions.

[0139] Step 7:

[0140] If a user wants to know more about a proposed financial product or procedure, the server takes the user's request and emotional state into consideration, generates personalized information and relevant documents, and sends them to the terminal.

[0141] Step 8:

[0142] The terminal displays documents to the user and allows the user to proceed through the interface if input is required. Through this entire process, the system provides the user with a natural and personalized experience.

[0143] (Example 2)

[0144] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0145] Modern users demand more personalized and natural interactions when acquiring information and processing information through digital systems. However, conventional systems fail to adequately generate appropriate responses based on user emotions or provide realistic experiences within virtual spaces. As a result, user satisfaction declines, and it becomes difficult to provide efficient and reliable services.

[0146] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0147] In this invention, the server includes an information processing device that receives authentication information and performs authentication procedures, an information processing device that transmits virtual environment data to an output device after authentication and generates visual information, and an artificial intelligence device equipped with an information processing device that analyzes the user's voice or facial expressions using emotion analysis technology and infers their emotional state. This makes it possible to generate personalized responses based on the user's emotions and to provide a realistic experience in a virtual space.

[0148] "Authentication information" refers to information used to identify a user and verify their access rights to the system.

[0149] An "information processing device" is a device that has the functions of receiving, analyzing, and transmitting data, and is used to perform various tasks.

[0150] An "output device" is a device that provides digital data to users through senses such as sight and hearing.

[0151] "Emotion analysis technology" is a technology that uses data such as voice and facial expressions to infer a user's emotional state.

[0152] An "artificial intelligence device" is a device that possesses automated learning and analytical capabilities and performs tasks by mimicking human intelligence.

[0153] A "generative AI model" is an algorithm and computational model used for artificial intelligence responses and data generation.

[0154] "Transaction information" refers to information that records the details of transactions performed by a user.

[0155] "Interaction information" refers to information that records the history of interactions between the user and the system.

[0156] "Service information" refers to information about products and services provided to users.

[0157] "Visual information" refers to visual content presented to users through displays or projectors.

[0158] This invention provides a system that generates personalized responses using emotion analysis technology to improve the user experience. Specific embodiments for carrying out the invention are described below.

[0159] The server uses an information processing device to process authentication information received from the user terminal. This device receives authentication data via a secure communication protocol and authenticates the user by comparing it with information in a database. Once authentication is confirmed, the server encrypts the virtual environment data and sends it to the user terminal.

[0160] The device receives this virtual environment data and uses AR / VR technology to enable the user to experience an immersive virtual space. Specifically, the device uses a high-performance graphics processing unit (GPU) to generate visual information and renders high-quality 3D images in real time.

[0161] The user interacts with an artificial intelligence device within the generated virtual space. This device utilizes emotion analysis technology to analyze the user's voice and facial expression data acquired through a voice input device and camera, and estimates the user's emotional state in real time. The analysis results are transmitted from the terminal to the server.

[0162] The server uses a generation AI model based on the analysis results to generate natural conversational text that aligns with the user's emotions in real time. This response is personalized based on past transaction and interaction information. The generated response is sent to the terminal, further enhancing the user's virtual reality experience.

[0163] As a concrete example, a scenario could be imagined where a user prompts an AI banker to "tell me about a new investment product," and the AI ​​banker, based on this information and considering the user's interests and concerns, provides a detailed and reassuring explanation.

[0164] In this way, the entire system can achieve interactions that are tailored to the individual emotions of each user, thereby increasing trust in and satisfaction with the service.

[0165] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0166] Step 1:

[0167] The user launches a dedicated application on their terminal and enters authentication information. This input data includes the user ID and password. The terminal initiates an authentication request by sending this authentication information to the server. The output is the transmission of authentication information to the server.

[0168] Step 2:

[0169] The server receives the authentication information and compares it with existing user data stored in its internal database. This data processing securely handles the entered authentication information and verifies the user's authentication. If authentication is successful, an authentication token is generated and sent to the terminal as output.

[0170] Step 3:

[0171] The server creates virtual environment data for authenticated users. This involves selecting and preparing customized 3D models and simulation data based on the service the user is trying to access. The output is the transmission of the prepared virtual environment data to the terminal.

[0172] Step 4:

[0173] The device uses received virtual environment data to generate a virtual space utilizing AR / VR technology. In this process, a graphics processing unit (GPU) is used to render 3D images in real time, providing the user with a high-quality visual experience. The output is the virtual space provided to the user.

[0174] Step 5:

[0175] The user begins an interaction with an AI bank teller in a virtual space. The user inputs prompts through their voice, and this voice data is captured by the terminal's voice input device. The output is the collected voice data.

[0176] Step 6:

[0177] The device analyzes voice data in real time using emotion analysis technology to infer the user's emotional state. This analysis includes voice tone and facial expression analysis. The output is the inferred emotional state.

[0178] Step 7:

[0179] The server uses a generative AI model based on the results of sentiment analysis to generate the most appropriate response for the user. This response generation process considers the user's emotional state and past transaction information to create personalized conversation content. The output is the generated response message.

[0180] Step 8:

[0181] The terminal receives response messages from the server and transmits them to the user by playing them back as audio or text within the virtual space. The output is the audio or text response delivered to the user.

[0182] This series of steps allows the system to provide users with personalized interactions and natural dialogue, thereby improving the user experience.

[0183] (Application Example 2)

[0184] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0185] In modern financial services, while user interaction is efficient, it lacks emotional depth and personalized experiences. This can lead to users proceeding with transactions while harboring anxieties and doubts, resulting in a lower user experience and satisfaction. Online financial transactions, in particular, struggle to replicate the sense of security that comes from face-to-face interactions. Therefore, there is a need for more natural and trustworthy interactions while minimizing user intervention.

[0186] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0187] In this invention, the server includes means for receiving authentication information from a user terminal and performing an authentication procedure, means for transmitting virtual environment data to the user terminal after authentication and generating a virtual space, and means for analyzing the user's voice and facial expressions to infer emotions and optimizing the response of the intelligence in the virtual space based on the analysis results. This makes it possible for users to receive personalized and emotionally sensitive financial services without anxiety or doubt.

[0188] A "user terminal" is a device used by a user and functions as an interface for executing virtual spaces and authentication processes.

[0189] "Authentication information" refers to data necessary to verify the user's identity and guarantees the security of system access.

[0190] "Virtual environment data" refers to the information necessary to construct the digital space that users access through an interface.

[0191] A "virtual space" is a digital interface built using augmented reality or virtual reality technologies, a space where users interact in real time.

[0192] "Intelligence" refers to algorithms and systems that understand user statements and actions and provide appropriate responses and services.

[0193] "Transaction history" refers to a series of records of financial transactions conducted by a user in the past, and is used to analyze the user's financial behavior.

[0194] "Interaction history" refers to a record of data generated through user interactions with the system, and contributes to the optimization of the service.

[0195] "Financial product information" refers to data about specific financial services and products provided to users.

[0196] "Voice and facial expression analysis" is a technology that analyzes voice and visual data to infer the user's emotions and adjust responses accordingly.

[0197] "Response optimization" is the process of generating the most appropriate and effective response based on the user's specific state and requests.

[0198] The system implementing this invention consists of a user terminal, a server, and a generative AI model. The user terminal refers to a device such as a smartphone or a head-mounted display, through which the user accesses and interacts with a virtual space. First, the user launches a dedicated application installed on the terminal, enters authentication information, and sends it to the server. The server analyzes the voice and facial recognition data using Google Cloud Speech-to-Text and Facial Recognition libraries to infer the user's emotions. Based on this emotion analysis, IBM Watson® Tone Analyzer plays a role in generating the optimal response.

[0199] The server processes the captured user's voice and facial expression data in real time and sends a customized response to the user's terminal. This response allows the virtual intelligence to provide the user with an emotionally responsive service. For example, if a user is interested in a financial product but feels a little uneasy, the system generates a response based on the prompt message "User is displaying a puzzled expression while asking about credit card benefits. Provide a soothing and clear explanation emphasizing the advantages of the product, using simple and reassuring language." In this way, the user can proceed with financial transactions with greater confidence.

[0200] The server integrates sentiment inference with past interaction history to provide users with the most relevant financial product information and improve the accuracy of interactions. This system significantly enhances the user experience, as users not only receive information but also enjoy a conversational experience that takes emotions into account.

[0201] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0202] Step 1:

[0203] The user launches a dedicated application on their device. The device captures the authentication information entered by the user and sends it to the server.

[0204] Input: User authentication information (e.g., username, password)

[0205] Output: Authentication information sent to the server

[0206] Step 2:

[0207] The server uses the received authentication information to perform the authentication process. If authentication is successful, it prepares the virtual environment data for the user.

[0208] Input: Authentication information

[0209] Output: Virtual environment data

[0210] Step 3:

[0211] The terminal generates a virtual space using augmented reality or virtual reality technology based on the received virtual environment data. The user then begins interacting with the system through this space.

[0212] Input: Virtual environment data

[0213] Output: Generated virtual space

[0214] Step 4:

[0215] Users interact with intelligence in a virtual space through voice and actions. The terminal captures this voice and facial expression data and sends it to the server.

[0216] Input: User's voice and facial expression data

[0217] Output: Audio and facial expression data sent to the server

[0218] Step 5:

[0219] The server uses Google Cloud Speech-to-Text and Facial Recognition libraries to convert speech data into text and analyze facial expression data to infer emotions. Based on these analysis results, IBM Watson Tone Analyzer is used to generate the optimal response.

[0220] Input: Voice and facial expression data

[0221] Output: Predicted sentiment data and generated responses

[0222] Step 6:

[0223] The server sends the generated response to the terminal and displays it to the user through intelligence. The user receives this response and gets support in making decisions about financial products.

[0224] Input: Generated response

[0225] Output: Response provided to the user

[0226] Step 7:

[0227] The server integrates the user's past interaction history with newly inferred sentiment data to provide the user with personalized financial product information.

[0228] Input: Interaction history and sentiment data

[0229] Output: Personalized financial product information

[0230] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0231] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0232] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0233] [Second Embodiment]

[0234] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0235] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0236] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0237] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0238] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0239] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0240] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0241] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0242] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0243] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0244] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0245] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0246] The virtual bank branch system according to the present invention is designed to meet the diverse needs and environments of various users. Users install a dedicated application on their user terminal and communicate with the server via the internet.

[0247] First, the user launches the app and enters their authentication information, which the device then sends to the server. The server verifies the received authentication information against its database, and only if authentication is successful does it send virtual environment data to the user. Using this virtual environment data, the device utilizes AR / VR technology to generate a three-dimensional virtual bank branch for the user.

[0248] In the generated virtual space, an AI bank teller is on standby, and users can communicate with the AI ​​via text, voice, or gestures, as if they were in a real branch. The user's voice and text input are processed by the terminal and sent to the server. The AI ​​on the server identifies the user's requests using advanced analytical capabilities and sends the optimal response back to the terminal in real time.

[0249] Furthermore, this system can analyze a user's past transaction history and provide individually optimized financial product information. When a user wishes to proceed with specific procedures such as opening a new account or consulting about a loan, the server automatically generates the necessary information and provides appropriate forms and documents through the user's terminal. The user completes the procedure online by filling out these documents and submitting them to the server.

[0250] For example, if a user wishes to open a new account, they can inform the AI ​​banker of this, and the server will send the necessary procedural information to the user's terminal. The user can then complete the process by entering the required information into a form.

[0251] Thus, the present invention provides convenient and efficient banking services that transcend geographical and temporal constraints, and is designed to be easily used, especially by elderly people who are unfamiliar with digital technology, through an intuitive interface.

[0252] The following describes the processing flow.

[0253] Step 1:

[0254] The user launches the virtual bank branch application on their terminal. The user enters their authentication information (e.g., user ID and password, or biometric authentication data) on the authentication screen.

[0255] Step 2:

[0256] The terminal receives the entered authentication information, encrypts it, and then sends it to the server. The server compares the received authentication information with its database to determine whether the authentication was successful or not.

[0257] Step 3:

[0258] If the server confirms successful authentication, it generates virtual environment data and sends it to the terminal. The terminal receives the virtual environment data and uses AR / VR technology to generate a virtual space that is reflected in the user's vision.

[0259] Step 4:

[0260] The user visually recognizes an AI bank teller in a virtual space and initiates a request via text or voice. The terminal transcribes the user's statements and input into text and sends that data to the server.

[0261] Step 5:

[0262] The server uses generative AI to analyze the user's intent and generate appropriate responses and suggestions. The generated responses are then sent to the terminal and provided to the user in real time.

[0263] Step 6:

[0264] If a user requests details or procedural information for a specific financial service, the server will refer to the user's transaction history and prepare individually optimized product information and procedural steps. It will then send the necessary digital documents to the user's device.

[0265] Step 7:

[0266] The terminal displays generated suggestions and required documents to the user, allowing the user to ask additional questions about unclear points and enter information into the documents. If the user proceeds with the procedure, the process is completed by sending the entered information back to the server.

[0267] This process allows users to receive necessary banking services without physically visiting a bank branch.

[0268] (Example 1)

[0269] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0270] In modern society, there is a demand for efficient and intuitive financial services that transcend geographical and temporal constraints. However, current systems have challenges such as the inconvenience users have to visit physical financial institutions and a lack of adequate support for users unfamiliar with digital technology. In particular, for users with little digital experience, such as the elderly, the complexity of online operations prevents them from fully realizing the benefits of these services.

[0271] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0272] In this invention, the server includes means for receiving identification information from a user information processing device and performing an authentication process, means for transmitting virtual reality data to the user information processing device after authentication and constructing a virtual environment, and means for an information processing device equipped with enhanced intelligence that interacts with the user information processing device using text information, voice information, or motion recognition. This enables users to enjoy intuitive and efficient financial services online without being restricted by their location or time.

[0273] A "user information processing device" is a device operated by a user that has functions to enable the input and display of identification information and the experience of a virtual environment.

[0274] "Identification information" refers to information necessary for user authentication, and typically includes user IDs, passwords, and biometric authentication data.

[0275] The "authentication process" refers to a series of procedures that verify the legitimacy of a user based on identification information received from a user information processing device.

[0276] "Virtual reality data" refers to data transmitted to a user information processing device, and includes information for building a virtual environment and providing an experience to the user.

[0277] A "virtual environment" is a space that provides three-dimensional visual and auditory information generated through information processing, enabling users to have an experience that closely resembles reality.

[0278] An "intelligence-enhanced information processing device" is a device that utilizes artificial intelligence technology to generate the optimal response based on the input information.

[0279] "Textual information, audio information, and motion recognition" refer to various forms of data that users input through information processing devices, and are means of communication.

[0280] "Financial services information" refers to information about financial products and services provided to users, including content optimized to meet user needs.

[0281] "Business history and interaction history" refers to records of transactions and communications when a user uses the system, and is data used to improve the provision of services to users.

[0282] A "storage device" is a device that stores information and makes it easy to retrieve when needed.

[0283] "Response accuracy" is an indicator that shows the ability of artificial intelligence to provide appropriate and accurate responses to user input.

[0284] In this invention, in order to improve the convenience of users, a system is provided in which a user information processing device, a server, and an information processing device with enhanced intelligence cooperate to operate. This system starts with the user installing a dedicated application on the information processing device. The information processing device includes devices such as smartphones and tablets, which communicate with the server via the Internet.

[0285] The server receives identification information from the user information processing device and performs an authentication process. In this process, database collation technology is used to confirm the identity of the user. When the authentication is successful, the server uses a graphics engine such as OpenGL or Unity to transmit virtual reality data to the user information processing device. As a result, the information processing device generates a virtual environment and provides the user with a three-dimensional visual experience.

[0286] Within this virtual environment, the user can interact with the information processing device with enhanced intelligence through character information, voice information, or motion recognition. For example, a user who wishes to open a new account uses a prompt sentence such as "I want to open a new account". This input is analyzed, and the server generates an optimal response according to the user's needs and sends it back to the information processing device.

[0287] The information processing device with enhanced intelligence analyzes the user's business history and interaction history using a generative AI model and machine learning technology. As a result, individually optimized financial service information is generated and provided to the user. An intuitive interface that supports the user's operations is realized, and it is easily accessible especially to users who are not accustomed to digital operations.

[0288] Thus, the present invention realizes an environment in which users can receive efficient and intuitive financial services without being restricted by location or time.

[0289] The flow of specific processing in Example 1 will be described using FIG. 11.

[0290] Step 1:

[0291] The user installs a dedicated application on the information processing device. Upon launching the application, the user enters their user ID and password or biometric authentication data as authentication information. The device then sends this information to the server. The entered authentication information is transmitted securely using encryption technology.

[0292] Step 2:

[0293] The server receives authentication information from the terminal and compares it with the registered information in the database. The server uses a database search algorithm to verify that the user's authentication information matches. If authentication is successful, the server generates virtual reality data and sends it to the terminal. At this time, customization is applied based on the user's history.

[0294] Step 3:

[0295] The device generates a virtual environment using a graphics engine (such as OpenGL or Unity) based on the received virtual reality data. Through this virtual environment, the user experiences the appearance of a three-dimensional virtual bank branch right before their eyes. The device utilizes AR / VR technology to provide the user with visual information.

[0296] Step 4:

[0297] Within the virtual environment, users provide input through text, voice, or actions. For example, a user might say, "I want to open a new account." The terminal analyzes this input, converts it into text data using speech recognition technology, and sends it to the server.

[0298] Step 5:

[0299] The server uses a generated AI model based on the analysis of user input to recognize user requests. The AI ​​model performs data calculations (natural language processing) to generate the optimal response to the input prompt. The server generates the response data and sends it back to the terminal.

[0300] Step 6:

[0301] The terminal receives response data from the server and provides feedback to the user. This includes responses using speech synthesis technology and text messages on the screen. The user then decides on their next action based on this feedback. This process enables an environment where financial services can be accessed smoothly from any location.

[0302] (Application Example 1)

[0303] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0304] In modern society, geographical and temporal constraints on accessing financial services remain significant barriers for users. Furthermore, traditional financial procedures often lack sufficient optimization and individualized support for the user experience, creating a demand for efficient and personalized services. In particular, there is a growing need for intuitive interfaces for users unfamiliar with digital environments.

[0305] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0306] In this invention, the server includes means for receiving authentication information from a user device and performing authentication processing; means for transmitting virtual environment data to the user device after authentication to generate a virtual space; means for mounting a machine learning model for communicating with the user device using text, voice, or actions; means for analyzing the user's operation history and interaction history to generate individually optimized financial service information; means for transmitting the generated financial service information to the user device; means for receiving a procedure request from the user, generating relevant documents, and transmitting them to the user device; means for generating a virtual financial facility using augmented reality technology; and means for generating changes in voice or screen to provide the user with the flow of the procedure. As a result, users can receive intuitive and personalized financial services without being restricted by location or time.

[0307] The "user device" is a terminal device for the user to operate, including portable information terminals such as smartphones and tablets.

[0308] The "authentication information" is data necessary to confirm the user's identity, including user ID, password, biometric information, etc.

[0309] The "virtual environment data" is an information set used to generate a digital three-dimensional space provided to the user, for realizing virtual reality and augmented reality experiences.

[0310] The "machine learning model" is a part of a computer program including an algorithm for optimizing interaction with the user based on data and generating appropriate responses.

[0311] The "financial service information" is information about individually optimized financial products and services provided to the user, generated based on the user's past transaction history and current needs.

[0312] Augmented reality technology is a technique that overlays digital information onto images of the real world, enabling users to interact with virtual objects.

[0313] A "virtual financial facility" is a digitally simulated facility that users can visit within a virtual environment, and it functions as a place where users can receive financial services.

[0314] To implement this invention, the user must first install a dedicated application on their user device, such as a smartphone. This application is designed to enhance the user experience using augmented reality and machine learning technologies. The user device communicates with a server via the internet to send and receive necessary data.

[0315] The server processes the authentication information received from the user terminal and verifies it against the database. Upon successful authentication, the server sends virtual environment data to the user terminal, where a virtual space is created. In this environment, machine learning models are utilized, enabling natural communication with the user.

[0316] The user terminal displays a generated virtual financial facility, allowing the user to visually immerse themselves in that space. AI-powered financial service information is generated on the server based on the user's actions and transaction history, and personalized content is provided to the user. Furthermore, if necessary, the server generates relevant digital documents and provides them through the user terminal.

[0317] The hardware will consist of mobile devices such as smartphones and tablets, while the software will utilize augmented reality development frameworks such as ARKit and ARCore, as well as TensorFlow and other machine learning engines. This system will leverage cloud infrastructure such as AWS and Google Cloud for data processing and storage.

[0318] One concrete example is a scenario where a user interacts with an AI banker in a virtual space and opens a new savings account. In this scenario, the AI ​​banker proposes the most suitable plan to the user via voice input, and the necessary procedures are carried out based on the user's selection.

[0319] An example of a prompt to input into the generating AI model might be: "Please describe in detail a scenario in which a user interacts with an AI in a virtual space and opens a new savings account. Pay particular attention to the use of voice input and a visual interface."

[0320] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0321] Step 1:

[0322] The user launches a dedicated application and enters their authentication information. The entered authentication information is sent from the terminal to the server. The server compares this information with existing data in the database and performs authentication. If authentication is successful, the server returns permission to proceed to the next step to the terminal.

[0323] Step 2:

[0324] The server generates virtual environment data and sends it to the user's terminal. Upon receiving this data, the terminal uses augmented reality technology to display a virtual financial facility to the user. The user enters this virtual space to begin a real-time experience.

[0325] Step 3:

[0326] The user interacts with an AI banker in a virtual space. When the user provides voice or text input, the terminal sends it to the server. The server analyzes the input using a generative AI model and generates an appropriate response. The generated response is then returned to the user, and the interaction continues.

[0327] Step 4:

[0328] Based on the user's past operation and interaction history, the server generates individually optimized financial service information. This information is generated by analyzing the user's transaction patterns and needs and processing the data. The resulting information is sent to the terminal and presented to the user.

[0329] Step 5:

[0330] When a user decides to use a specific financial service, the server generates the relevant digital documents. Taking the required data as input, the server automatically fills in the document format and sends the completed version to the user's terminal. The user then completes the requested procedures or applications through these digital documents.

[0331] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0332] The system according to the present invention is built to further personalize the user experience and enable natural communication by incorporating an emotion engine.

[0333] The user first launches a dedicated application installed on their device and enters authentication information to access the system. The device sends the entered authentication information to the server, which performs authentication. If authentication is successful, the server sends virtual environment data to the user's device. Based on the received data, the device generates a realistic virtual space using AR / VR technology. In this virtual space, the user can communicate with an AI banker.

[0334] The AI ​​bank teller can analyze the user's voice and facial expressions through an emotion engine to infer their emotions. Based on this emotion analysis, the server generates an optimal response tailored to the user's emotions and sends it to the terminal. For example, if the AI ​​bank teller determines that the user is nervous, it can respond in a gentle and calm tone to alleviate their anxiety.

[0335] Furthermore, the server integrates the acquired emotional data with the user's past interaction data for deeper analysis. Using the results of this analysis, it becomes possible to individually customize and provide the user with the most suitable financial product information. For example, if a user is interested in a new investment product but also feels anxious during the conversation, the server will select responses that gently explain the details and build trust.

[0336] In this way, by incorporating an emotion engine, this system achieves a more human-like interaction that goes beyond simple question-and-answering, greatly improving user satisfaction.

[0337] The following describes the processing flow.

[0338] Step 1:

[0339] The user launches a dedicated application on their device and enters their authentication information using their user ID and password or biometric authentication on the login screen.

[0340] Step 2:

[0341] The terminal sends the entered authentication information to the server. The server compares the received information with its internal database to determine whether authentication was successful.

[0342] Step 3:

[0343] Upon successful authentication, the server sends virtual environment data to the user's terminal. Based on the received data, the terminal uses AR / VR technology to generate the virtual space that the user will experience.

[0344] Step 4:

[0345] Users interact with an AI bank teller in a virtual space and make requests via voice or text. For example, when requesting a balance check, the terminal converts the voice into text and sends it to the server.

[0346] Step 5:

[0347] The server analyzes the received user voice data using an emotion engine, estimating emotions based on voice tone and user facial expression data. Based on the analysis results, it generates an appropriate response that matches the user's emotions.

[0348] Step 6:

[0349] The server sends the generated response data to the terminal. The terminal then receives a response from the AI ​​bank teller in voice or text format, in a tone that matches the user's emotions.

[0350] Step 7:

[0351] If a user wants to know more about a proposed financial product or procedure, the server takes the user's request and emotional state into consideration, generates personalized information and relevant documents, and sends them to the terminal.

[0352] Step 8:

[0353] The terminal displays documents to the user and allows the user to proceed through the interface if input is required. Through this entire process, the system provides the user with a natural and personalized experience.

[0354] (Example 2)

[0355] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0356] Modern users demand more personalized and natural interactions when acquiring information and processing information through digital systems. However, conventional systems fail to adequately generate appropriate responses based on user emotions or provide realistic experiences within virtual spaces. As a result, user satisfaction declines, and it becomes difficult to provide efficient and reliable services.

[0357] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0358] In this invention, the server includes an information processing device that receives authentication information and performs authentication procedures, an information processing device that transmits virtual environment data to an output device after authentication and generates visual information, and an artificial intelligence device equipped with an information processing device that analyzes the user's voice or facial expressions using emotion analysis technology and infers their emotional state. This makes it possible to generate personalized responses based on the user's emotions and to provide a realistic experience in a virtual space.

[0359] "Authentication information" refers to information used to identify a user and verify their access rights to the system.

[0360] An "information processing device" is a device that has the functions of receiving, analyzing, and transmitting data, and is used to perform various tasks.

[0361] An "output device" is a device that provides digital data to users through senses such as sight and hearing.

[0362] "Emotion analysis technology" is a technology that uses data such as voice and facial expressions to infer a user's emotional state.

[0363] An "artificial intelligence device" is a device that possesses automated learning and analytical capabilities and performs tasks by mimicking human intelligence.

[0364] A "generative AI model" is an algorithm and computational model used for artificial intelligence responses and data generation.

[0365] "Transaction information" refers to information that records the details of transactions performed by a user.

[0366] "Interaction information" refers to information that records the history of interactions between the user and the system.

[0367] "Service information" refers to information about products and services provided to users.

[0368] "Visual information" refers to visual content presented to users through displays or projectors.

[0369] This invention provides a system that generates personalized responses using emotion analysis technology to improve the user experience. Specific embodiments for carrying out the invention are described below.

[0370] The server uses an information processing device to process authentication information received from the user terminal. This device receives authentication data via a secure communication protocol and authenticates the user by comparing it with information in a database. Once authentication is confirmed, the server encrypts the virtual environment data and sends it to the user terminal.

[0371] The device receives this virtual environment data and uses AR / VR technology to enable the user to experience an immersive virtual space. Specifically, the device uses a high-performance graphics processing unit (GPU) to generate visual information and renders high-quality 3D images in real time.

[0372] The user interacts with an artificial intelligence device within the generated virtual space. This device utilizes emotion analysis technology to analyze the user's voice and facial expression data acquired through a voice input device and camera, and estimates the user's emotional state in real time. The analysis results are transmitted from the terminal to the server.

[0373] The server uses a generation AI model based on the analysis results to generate natural conversational text that aligns with the user's emotions in real time. This response is personalized based on past transaction and interaction information. The generated response is sent to the terminal, further enhancing the user's virtual reality experience.

[0374] As a concrete example, a scenario could be imagined where a user prompts an AI banker to "tell me about a new investment product," and the AI ​​banker, based on this information and considering the user's interests and concerns, provides a detailed and reassuring explanation.

[0375] In this way, the entire system can achieve interactions that are tailored to the individual emotions of each user, thereby increasing trust in and satisfaction with the service.

[0376] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0377] Step 1:

[0378] The user launches a dedicated application on their terminal and enters authentication information. This input data includes the user ID and password. The terminal initiates an authentication request by sending this authentication information to the server. The output is the transmission of authentication information to the server.

[0379] Step 2:

[0380] The server receives the authentication information and compares it with existing user data stored in its internal database. This data processing securely handles the entered authentication information and verifies the user's authentication. If authentication is successful, an authentication token is generated and sent to the terminal as output.

[0381] Step 3:

[0382] The server creates virtual environment data for authenticated users. This involves selecting and preparing customized 3D models and simulation data based on the service the user is trying to access. The output is the transmission of the prepared virtual environment data to the terminal.

[0383] Step 4:

[0384] The device uses received virtual environment data to generate a virtual space utilizing AR / VR technology. In this process, a graphics processing unit (GPU) is used to render 3D images in real time, providing the user with a high-quality visual experience. The output is the virtual space provided to the user.

[0385] Step 5:

[0386] The user begins an interaction with an AI bank teller in a virtual space. The user inputs prompts through their voice, and this voice data is captured by the terminal's voice input device. The output is the collected voice data.

[0387] Step 6:

[0388] The device analyzes voice data in real time using emotion analysis technology to infer the user's emotional state. This analysis includes voice tone and facial expression analysis. The output is the inferred emotional state.

[0389] Step 7:

[0390] The server uses a generative AI model based on the results of sentiment analysis to generate the most appropriate response for the user. This response generation process considers the user's emotional state and past transaction information to create personalized conversation content. The output is the generated response message.

[0391] Step 8:

[0392] The terminal receives response messages from the server and transmits them to the user by playing them back as audio or text within the virtual space. The output is the audio or text response delivered to the user.

[0393] This series of steps allows the system to provide users with personalized interactions and natural dialogue, thereby improving the user experience.

[0394] (Application Example 2)

[0395] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0396] In modern financial services, while user interaction is efficient, it lacks emotional depth and personalized experiences. This can lead to users proceeding with transactions while harboring anxieties and doubts, resulting in a lower user experience and satisfaction. Online financial transactions, in particular, struggle to replicate the sense of security that comes from face-to-face interactions. Therefore, there is a need for more natural and trustworthy interactions while minimizing user intervention.

[0397] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0398] In this invention, the server includes means for receiving authentication information from a user terminal and performing an authentication procedure, means for transmitting virtual environment data to the user terminal after authentication and generating a virtual space, and means for analyzing the user's voice and facial expressions to infer emotions and optimizing the response of the intelligence in the virtual space based on the analysis results. This makes it possible for users to receive personalized and emotionally sensitive financial services without anxiety or doubt.

[0399] A "user terminal" is a device used by a user and functions as an interface for executing virtual spaces and authentication processes.

[0400] "Authentication information" refers to data necessary to verify the user's identity and guarantees the security of system access.

[0401] "Virtual environment data" refers to the information necessary to construct the digital space that users access through an interface.

[0402] A "virtual space" is a digital interface built using augmented reality or virtual reality technologies, a space where users interact in real time.

[0403] "Intelligence" refers to algorithms and systems that understand user statements and actions and provide appropriate responses and services.

[0404] "Transaction history" refers to a series of records of financial transactions conducted by a user in the past, and is used to analyze the user's financial behavior.

[0405] "Interaction history" refers to a record of data generated through user interactions with the system, and contributes to the optimization of the service.

[0406] "Financial product information" refers to data about specific financial services and products provided to users.

[0407] "Voice and facial expression analysis" is a technology that analyzes voice and visual data to infer the user's emotions and adjust responses accordingly.

[0408] "Response optimization" is the process of generating the most appropriate and effective response based on the user's specific state and requests.

[0409] The system implementing this invention consists of a user terminal, a server, and a generative AI model. The user terminal refers to a device such as a smartphone or a head-mounted display, through which the user accesses and interacts with a virtual space. First, the user launches a dedicated application installed on the terminal, enters authentication information, and sends it to the server. The server analyzes the voice and facial recognition data using Google Cloud Speech-to-Text and Facial Recognition libraries to infer the user's emotions. Based on this emotion analysis, IBM Watson Tone Analyzer plays a role in generating the optimal response.

[0410] The server processes the captured user's voice and facial expression data in real time and sends a customized response to the user's terminal. This response allows the virtual intelligence to provide the user with an emotionally responsive service. For example, if a user is interested in a financial product but feels a little uneasy, the system generates a response based on the prompt message "User is displaying a puzzled expression while asking about credit card benefits. Provide a soothing and clear explanation emphasizing the advantages of the product, using simple and reassuring language." In this way, the user can proceed with financial transactions with greater confidence.

[0411] The server integrates sentiment inference with past interaction history to provide users with the most relevant financial product information and improve the accuracy of interactions. This system significantly enhances the user experience, as users not only receive information but also enjoy a conversational experience that takes emotions into account.

[0412] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0413] Step 1:

[0414] The user launches a dedicated application on their device. The device captures the authentication information entered by the user and sends it to the server.

[0415] Input: User authentication information (e.g., username, password)

[0416] Output: Authentication information sent to the server

[0417] Step 2:

[0418] The server uses the received authentication information to perform the authentication process. If authentication is successful, it prepares the virtual environment data for the user.

[0419] Input: Authentication information

[0420] Output: Virtual environment data

[0421] Step 3:

[0422] The terminal generates a virtual space using augmented reality or virtual reality technology based on the received virtual environment data. The user then begins interacting with the system through this space.

[0423] Input: Virtual environment data

[0424] Output: Generated virtual space

[0425] Step 4:

[0426] Users interact with intelligence in a virtual space through voice and actions. The terminal captures this voice and facial expression data and sends it to the server.

[0427] Input: User's voice and facial expression data

[0428] Output: Audio and facial expression data sent to the server

[0429] Step 5:

[0430] The server uses Google Cloud Speech-to-Text and Facial Recognition libraries to convert speech data into text and analyze facial expression data to infer emotions. Based on these analysis results, IBM Watson Tone Analyzer is used to generate the optimal response.

[0431] Input: Voice and facial expression data

[0432] Output: Predicted sentiment data and generated responses

[0433] Step 6:

[0434] The server sends the generated response to the terminal and displays it to the user through intelligence. The user receives this response and gets support in making decisions about financial products.

[0435] Input: Generated response

[0436] Output: Response provided to the user

[0437] Step 7:

[0438] The server integrates the user's past interaction history with newly inferred sentiment data to provide the user with personalized financial product information.

[0439] Input: Interaction history and sentiment data

[0440] Output: Personalized financial product information

[0441] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0442] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0443] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0444] [Third Embodiment]

[0445] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0446] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0447] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0448] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0449] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0450] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0451] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0452] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0453] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0454] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0455] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0456] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0457] The virtual bank branch system according to the present invention is designed to meet the diverse needs and environments of various users. Users install a dedicated application on their user terminal and communicate with the server via the internet.

[0458] First, the user launches the app and enters their authentication information, which the device then sends to the server. The server verifies the received authentication information against its database, and only if authentication is successful does it send virtual environment data to the user. Using this virtual environment data, the device utilizes AR / VR technology to generate a three-dimensional virtual bank branch for the user.

[0459] In the generated virtual space, an AI bank teller is on standby, and users can communicate with the AI ​​via text, voice, or gestures, as if they were in a real branch. The user's voice and text input are processed by the terminal and sent to the server. The AI ​​on the server identifies the user's requests using advanced analytical capabilities and sends the optimal response back to the terminal in real time.

[0460] Furthermore, this system can analyze a user's past transaction history and provide individually optimized financial product information. When a user wishes to proceed with specific procedures such as opening a new account or consulting about a loan, the server automatically generates the necessary information and provides appropriate forms and documents through the user's terminal. The user completes the procedure online by filling out these documents and submitting them to the server.

[0461] For example, if a user wishes to open a new account, they can inform the AI ​​banker of this, and the server will send the necessary procedural information to the user's terminal. The user can then complete the process by entering the required information into a form.

[0462] Thus, the present invention provides convenient and efficient banking services that transcend geographical and temporal constraints, and is designed to be easily used, especially by elderly people who are unfamiliar with digital technology, through an intuitive interface.

[0463] The following describes the processing flow.

[0464] Step 1:

[0465] The user launches the virtual bank branch application on their terminal. The user enters their authentication information (e.g., user ID and password, or biometric authentication data) on the authentication screen.

[0466] Step 2:

[0467] The terminal receives the entered authentication information, encrypts it, and then sends it to the server. The server compares the received authentication information with its database to determine whether the authentication was successful or not.

[0468] Step 3:

[0469] If the server confirms successful authentication, it generates virtual environment data and sends it to the terminal. The terminal receives the virtual environment data and uses AR / VR technology to generate a virtual space that is reflected in the user's vision.

[0470] Step 4:

[0471] The user visually recognizes an AI bank teller in a virtual space and initiates a request via text or voice. The terminal transcribes the user's statements and input into text and sends that data to the server.

[0472] Step 5:

[0473] The server uses generative AI to analyze the user's intent and generate appropriate responses and suggestions. The generated responses are then sent to the terminal and provided to the user in real time.

[0474] Step 6:

[0475] If a user requests details or procedural information for a specific financial service, the server will refer to the user's transaction history and prepare individually optimized product information and procedural steps. It will then send the necessary digital documents to the user's device.

[0476] Step 7:

[0477] The terminal displays generated suggestions and required documents to the user, allowing the user to ask additional questions about unclear points and enter information into the documents. If the user proceeds with the procedure, the process is completed by sending the entered information back to the server.

[0478] This process allows users to receive necessary banking services without physically visiting a bank branch.

[0479] (Example 1)

[0480] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0481] In modern society, there is a demand for efficient and intuitive financial services that transcend geographical and temporal constraints. However, current systems have challenges such as the inconvenience users have to visit physical financial institutions and a lack of adequate support for users unfamiliar with digital technology. In particular, for users with little digital experience, such as the elderly, the complexity of online operations prevents them from fully realizing the benefits of these services.

[0482] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0483] In this invention, the server includes means for receiving identification information from a user information processing device and performing an authentication process, means for transmitting virtual reality data to the user information processing device after authentication and constructing a virtual environment, and means for an information processing device equipped with enhanced intelligence that interacts with the user information processing device using text information, voice information, or motion recognition. This enables users to enjoy intuitive and efficient financial services online without being restricted by their location or time.

[0484] A "user information processing device" is a device operated by a user that has functions to enable the input and display of identification information and the experience of a virtual environment.

[0485] "Identification information" refers to information necessary for user authentication, and typically includes user IDs, passwords, and biometric authentication data.

[0486] The "authentication process" refers to a series of procedures that verify the legitimacy of a user based on identification information received from a user information processing device.

[0487] "Virtual reality data" refers to data transmitted to a user information processing device, and includes information for building a virtual environment and providing an experience to the user.

[0488] A "virtual environment" is a space that provides three-dimensional visual and auditory information generated through information processing, enabling users to have an experience that closely resembles reality.

[0489] An "intelligence-enhanced information processing device" is a device that utilizes artificial intelligence technology to generate the optimal response based on the input information.

[0490] "Textual information, audio information, and motion recognition" refer to various forms of data that users input through information processing devices, and are means of communication.

[0491] "Financial services information" refers to information about financial products and services provided to users, including content optimized to meet user needs.

[0492] "Business history and interaction history" refers to records of transactions and communications when a user uses the system, and is data used to improve the provision of services to users.

[0493] A "storage device" is a device that stores information and makes it easy to retrieve when needed.

[0494] "Response accuracy" is an indicator that shows the ability of artificial intelligence to provide appropriate and accurate responses to user input.

[0495] This invention provides a system in which a user information processing device, a server, and an intelligent information processing device work in cooperation to improve user convenience. This system begins with the user installing a dedicated application on the information processing device. The information processing device includes devices such as smartphones and tablets, which communicate with the server via the internet.

[0496] The server receives identification information from the user information processing device and performs an authentication process. Database matching technology is used in this process to verify the user's identity. Upon successful authentication, the server uses a graphics engine such as OpenGL or Unity to send virtual reality data to the user information processing device. This allows the information processing device to generate a virtual environment and provide the user with a three-dimensional visual experience.

[0497] Within this virtual environment, users can interact with an intelligent information processing device through text, voice, or motion recognition. For example, a user wishing to open a new account might use a prompt such as "I want to open a new account." This input is parsed, and the server generates the most appropriate response to the user's needs and sends it back to the information processing device.

[0498] The enhanced intelligence-based information processing device uses generative AI models and machine learning techniques to analyze the user's business history and dialogue history. This generates and provides the user with individually optimized financial service information. An intuitive interface is provided to support user operation, making it easy to use, especially for users unfamiliar with digital technology.

[0499] Thus, the present invention realizes an environment in which users can receive efficient and intuitive financial services without being restricted by location or time.

[0500] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0501] Step 1:

[0502] The user installs a dedicated application on the information processing device. Upon launching the application, the user enters their user ID and password or biometric authentication data as authentication information. The device then sends this information to the server. The entered authentication information is transmitted securely using encryption technology.

[0503] Step 2:

[0504] The server receives authentication information from the terminal and compares it with the registered information in the database. The server uses a database search algorithm to verify that the user's authentication information matches. If authentication is successful, the server generates virtual reality data and sends it to the terminal. At this time, customization is applied based on the user's history.

[0505] Step 3:

[0506] The device generates a virtual environment using a graphics engine (such as OpenGL or Unity) based on the received virtual reality data. Through this virtual environment, the user experiences the appearance of a three-dimensional virtual bank branch right before their eyes. The device utilizes AR / VR technology to provide the user with visual information.

[0507] Step 4:

[0508] Within the virtual environment, users provide input through text, voice, or actions. For example, a user might say, "I want to open a new account." The terminal analyzes this input, converts it into text data using speech recognition technology, and sends it to the server.

[0509] Step 5:

[0510] The server uses a generated AI model based on the analysis of user input to recognize user requests. The AI ​​model performs data calculations (natural language processing) to generate the optimal response to the input prompt. The server generates the response data and sends it back to the terminal.

[0511] Step 6:

[0512] The terminal receives response data from the server and provides feedback to the user. This includes responses using speech synthesis technology and text messages on the screen. The user then decides on their next action based on this feedback. This process enables an environment where financial services can be accessed smoothly from any location.

[0513] (Application Example 1)

[0514] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0515] In modern society, geographical and temporal constraints on accessing financial services remain significant barriers for users. Furthermore, traditional financial procedures often lack sufficient optimization and individualized support for the user experience, creating a demand for efficient and personalized services. In particular, there is a growing need for intuitive interfaces for users unfamiliar with digital environments.

[0516] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0517] In this invention, the server includes means for receiving authentication information from a user device and performing authentication processing; means for transmitting virtual environment data to the user device after authentication and generating a virtual space; means for incorporating a machine learning model that communicates with the user device using text, voice, or actions; means for analyzing the user's operation history and dialogue history and generating individually optimized financial service information; means for transmitting the generated financial service information to the user device; means for receiving procedure requests from the user, generating related documents and transmitting them to the user device; means for generating a virtual financial facility using augmented reality technology; and means for generating voice or screen changes and providing the user with a flow of procedures. This enables users to receive intuitive and personalized financial services without being bound by location or time.

[0518] "User device" refers to a terminal device used by a user, and includes mobile information terminals such as smartphones and tablets.

[0519] "Authentication information" refers to data necessary to verify a user's identity, and includes user IDs, passwords, biometric information, and other similar information.

[0520] "Virtual environment data" refers to a set of information used to generate a digital three-dimensional space provided to the user, and is intended to realize virtual reality and augmented reality experiences.

[0521] A "machine learning model" is a part of a computer program that includes algorithms to optimize user interaction based on data and generate appropriate responses.

[0522] "Financial service information" refers to information about individually optimized financial products and services provided to users, and is generated based on the user's past transaction history and current needs.

[0523] Augmented reality technology is a technique that overlays digital information onto images of the real world, enabling users to interact with virtual objects.

[0524] A "virtual financial facility" is a digitally simulated facility that users can visit within a virtual environment, and it functions as a place where users can receive financial services.

[0525] To implement this invention, the user must first install a dedicated application on their user device, such as a smartphone. This application is designed to enhance the user experience using augmented reality and machine learning technologies. The user device communicates with a server via the internet to send and receive necessary data.

[0526] The server processes the authentication information received from the user terminal and verifies it against the database. Upon successful authentication, the server sends virtual environment data to the user terminal, where a virtual space is created. In this environment, machine learning models are utilized, enabling natural communication with the user.

[0527] The user terminal displays a generated virtual financial facility, allowing the user to visually immerse themselves in that space. AI-powered financial service information is generated on the server based on the user's actions and transaction history, and personalized content is provided to the user. Furthermore, if necessary, the server generates relevant digital documents and provides them through the user terminal.

[0528] The hardware will consist of mobile devices such as smartphones and tablets, while the software will utilize augmented reality development frameworks such as ARKit and ARCore, as well as TensorFlow and other machine learning engines. This system will leverage cloud infrastructure such as AWS and Google Cloud for data processing and storage.

[0529] One concrete example is a scenario where a user interacts with an AI banker in a virtual space and opens a new savings account. In this scenario, the AI ​​banker proposes the most suitable plan to the user via voice input, and the necessary procedures are carried out based on the user's selection.

[0530] An example of a prompt to input into the generating AI model might be: "Please describe in detail a scenario in which a user interacts with an AI in a virtual space and opens a new savings account. Pay particular attention to the use of voice input and a visual interface."

[0531] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0532] Step 1:

[0533] The user launches a dedicated application and enters their authentication information. The entered authentication information is sent from the terminal to the server. The server compares this information with existing data in the database and performs authentication. If authentication is successful, the server returns permission to proceed to the next step to the terminal.

[0534] Step 2:

[0535] The server generates virtual environment data and sends it to the user's terminal. Upon receiving this data, the terminal uses augmented reality technology to display a virtual financial facility to the user. The user enters this virtual space to begin a real-time experience.

[0536] Step 3:

[0537] The user interacts with an AI banker in a virtual space. When the user provides voice or text input, the terminal sends it to the server. The server analyzes the input using a generative AI model and generates an appropriate response. The generated response is then returned to the user, and the interaction continues.

[0538] Step 4:

[0539] Based on the user's past operation and interaction history, the server generates individually optimized financial service information. This information is generated by analyzing the user's transaction patterns and needs and processing the data. The resulting information is sent to the terminal and presented to the user.

[0540] Step 5:

[0541] When a user decides to use a specific financial service, the server generates the relevant digital documents. Taking the required data as input, the server automatically fills in the document format and sends the completed version to the user's terminal. The user then completes the requested procedures or applications through these digital documents.

[0542] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0543] The system according to the present invention is built to further personalize the user experience and enable natural communication by incorporating an emotion engine.

[0544] The user first launches a dedicated application installed on their device and enters authentication information to access the system. The device sends the entered authentication information to the server, which performs authentication. If authentication is successful, the server sends virtual environment data to the user's device. Based on the received data, the device generates a realistic virtual space using AR / VR technology. In this virtual space, the user can communicate with an AI banker.

[0545] The AI ​​bank teller can analyze the user's voice and facial expressions through an emotion engine to infer their emotions. Based on this emotion analysis, the server generates an optimal response tailored to the user's emotions and sends it to the terminal. For example, if the AI ​​bank teller determines that the user is nervous, it can respond in a gentle and calm tone to alleviate their anxiety.

[0546] Furthermore, the server integrates the acquired emotional data with the user's past interaction data for deeper analysis. Using the results of this analysis, it becomes possible to individually customize and provide the user with the most suitable financial product information. For example, if a user is interested in a new investment product but also feels anxious during the conversation, the server will select responses that gently explain the details and build trust.

[0547] In this way, by incorporating an emotion engine, this system achieves a more human-like interaction that goes beyond simple question-and-answering, greatly improving user satisfaction.

[0548] The following describes the processing flow.

[0549] Step 1:

[0550] The user launches a dedicated application on their device and enters their authentication information using their user ID and password or biometric authentication on the login screen.

[0551] Step 2:

[0552] The terminal sends the entered authentication information to the server. The server compares the received information with its internal database to determine whether authentication was successful.

[0553] Step 3:

[0554] Upon successful authentication, the server sends virtual environment data to the user's terminal. Based on the received data, the terminal uses AR / VR technology to generate the virtual space that the user will experience.

[0555] Step 4:

[0556] Users interact with an AI bank teller in a virtual space and make requests via voice or text. For example, when requesting a balance check, the terminal converts the voice into text and sends it to the server.

[0557] Step 5:

[0558] The server analyzes the received user voice data using an emotion engine, estimating emotions based on voice tone and user facial expression data. Based on the analysis results, it generates an appropriate response that matches the user's emotions.

[0559] Step 6:

[0560] The server sends the generated response data to the terminal. The terminal then receives a response from the AI ​​bank teller in voice or text format, in a tone that matches the user's emotions.

[0561] Step 7:

[0562] If a user wants to know more about a proposed financial product or procedure, the server takes the user's request and emotional state into consideration, generates personalized information and relevant documents, and sends them to the terminal.

[0563] Step 8:

[0564] The terminal displays documents to the user and allows the user to proceed through the interface if input is required. Through this entire process, the system provides the user with a natural and personalized experience.

[0565] (Example 2)

[0566] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0567] Modern users demand more personalized and natural interactions when acquiring information and processing information through digital systems. However, conventional systems fail to adequately generate appropriate responses based on user emotions or provide realistic experiences within virtual spaces. As a result, user satisfaction declines, and it becomes difficult to provide efficient and reliable services.

[0568] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0569] In this invention, the server includes an information processing device that receives authentication information and performs authentication procedures, an information processing device that transmits virtual environment data to an output device after authentication and generates visual information, and an artificial intelligence device equipped with an information processing device that analyzes the user's voice or facial expressions using emotion analysis technology and infers their emotional state. This makes it possible to generate personalized responses based on the user's emotions and to provide a realistic experience in a virtual space.

[0570] "Authentication information" refers to information used to identify a user and verify their access rights to the system.

[0571] An "information processing device" is a device that has the functions of receiving, analyzing, and transmitting data, and is used to perform various tasks.

[0572] An "output device" is a device that provides digital data to users through senses such as sight and hearing.

[0573] "Emotion analysis technology" is a technology that uses data such as voice and facial expressions to infer a user's emotional state.

[0574] An "artificial intelligence device" is a device that possesses automated learning and analytical capabilities and performs tasks by mimicking human intelligence.

[0575] A "generative AI model" is an algorithm and computational model used for artificial intelligence responses and data generation.

[0576] "Transaction information" refers to information that records the details of transactions performed by a user.

[0577] "Interaction information" refers to information that records the history of interactions between the user and the system.

[0578] "Service information" refers to information about products and services provided to users.

[0579] "Visual information" refers to visual content presented to users through displays or projectors.

[0580] This invention provides a system that generates personalized responses using emotion analysis technology to improve the user experience. Specific embodiments for carrying out the invention are described below.

[0581] The server uses an information processing device to process authentication information received from the user terminal. This device receives authentication data via a secure communication protocol and authenticates the user by comparing it with information in a database. Once authentication is confirmed, the server encrypts the virtual environment data and sends it to the user terminal.

[0582] The device receives this virtual environment data and uses AR / VR technology to enable the user to experience an immersive virtual space. Specifically, the device uses a high-performance graphics processing unit (GPU) to generate visual information and renders high-quality 3D images in real time.

[0583] The user interacts with an artificial intelligence device within the generated virtual space. This device utilizes emotion analysis technology to analyze the user's voice and facial expression data acquired through a voice input device and camera, and estimates the user's emotional state in real time. The analysis results are transmitted from the terminal to the server.

[0584] The server uses a generation AI model based on the analysis results to generate natural conversational text that aligns with the user's emotions in real time. This response is personalized based on past transaction and interaction information. The generated response is sent to the terminal, further enhancing the user's virtual reality experience.

[0585] As a concrete example, a scenario could be imagined where a user prompts an AI banker to "tell me about a new investment product," and the AI ​​banker, based on this information and considering the user's interests and concerns, provides a detailed and reassuring explanation.

[0586] In this way, the entire system can achieve interactions that are tailored to the individual emotions of each user, thereby increasing trust in and satisfaction with the service.

[0587] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0588] Step 1:

[0589] The user launches a dedicated application on their terminal and enters authentication information. This input data includes the user ID and password. The terminal initiates an authentication request by sending this authentication information to the server. The output is the transmission of authentication information to the server.

[0590] Step 2:

[0591] The server receives the authentication information and compares it with existing user data stored in its internal database. This data processing securely handles the entered authentication information and verifies the user's authentication. If authentication is successful, an authentication token is generated and sent to the terminal as output.

[0592] Step 3:

[0593] The server creates virtual environment data for authenticated users. This involves selecting and preparing customized 3D models and simulation data based on the service the user is trying to access. The output is the transmission of the prepared virtual environment data to the terminal.

[0594] Step 4:

[0595] The device uses received virtual environment data to generate a virtual space utilizing AR / VR technology. In this process, a graphics processing unit (GPU) is used to render 3D images in real time, providing the user with a high-quality visual experience. The output is the virtual space provided to the user.

[0596] Step 5:

[0597] The user begins an interaction with an AI bank teller in a virtual space. The user inputs prompts through their voice, and this voice data is captured by the terminal's voice input device. The output is the collected voice data.

[0598] Step 6:

[0599] The device analyzes voice data in real time using emotion analysis technology to infer the user's emotional state. This analysis includes voice tone and facial expression analysis. The output is the inferred emotional state.

[0600] Step 7:

[0601] The server uses a generative AI model based on the results of sentiment analysis to generate the most appropriate response for the user. This response generation process considers the user's emotional state and past transaction information to create personalized conversation content. The output is the generated response message.

[0602] Step 8:

[0603] The terminal receives response messages from the server and transmits them to the user by playing them back as audio or text within the virtual space. The output is the audio or text response delivered to the user.

[0604] This series of steps allows the system to provide users with personalized interactions and natural dialogue, thereby improving the user experience.

[0605] (Application Example 2)

[0606] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0607] In modern financial services, while user interaction is efficient, it lacks emotional depth and personalized experiences. This can lead to users proceeding with transactions while harboring anxieties and doubts, resulting in a lower user experience and satisfaction. Online financial transactions, in particular, struggle to replicate the sense of security that comes from face-to-face interactions. Therefore, there is a need for more natural and trustworthy interactions while minimizing user intervention.

[0608] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0609] In this invention, the server includes means for receiving authentication information from a user terminal and performing an authentication procedure, means for transmitting virtual environment data to the user terminal after authentication and generating a virtual space, and means for analyzing the user's voice and facial expressions to infer emotions and optimizing the response of the intelligence in the virtual space based on the analysis results. This makes it possible for users to receive personalized and emotionally sensitive financial services without anxiety or doubt.

[0610] A "user terminal" is a device used by a user and functions as an interface for executing virtual spaces and authentication processes.

[0611] "Authentication information" refers to data necessary to verify the user's identity and guarantees the security of system access.

[0612] "Virtual environment data" refers to the information necessary to construct the digital space that users access through an interface.

[0613] A "virtual space" is a digital interface built using augmented reality or virtual reality technologies, a space where users interact in real time.

[0614] "Intelligence" refers to algorithms and systems that understand user statements and actions and provide appropriate responses and services.

[0615] "Transaction history" refers to a series of records of financial transactions conducted by a user in the past, and is used to analyze the user's financial behavior.

[0616] "Interaction history" refers to a record of data generated through user interactions with the system, and contributes to the optimization of the service.

[0617] "Financial product information" refers to data about specific financial services and products provided to users.

[0618] "Voice and facial expression analysis" is a technology that analyzes voice and visual data to infer the user's emotions and adjust responses accordingly.

[0619] "Response optimization" is the process of generating the most appropriate and effective response based on the user's specific state and requests.

[0620] The system implementing this invention consists of a user terminal, a server, and a generative AI model. The user terminal refers to a device such as a smartphone or a head-mounted display, through which the user accesses and interacts with a virtual space. First, the user launches a dedicated application installed on the terminal, enters authentication information, and sends it to the server. The server analyzes the voice and facial recognition data using Google Cloud Speech-to-Text and Facial Recognition libraries to infer the user's emotions. Based on this emotion analysis, IBM Watson Tone Analyzer plays a role in generating the optimal response.

[0621] The server processes the captured user's voice and facial expression data in real time and sends a customized response to the user's terminal. This response allows the virtual intelligence to provide the user with an emotionally responsive service. For example, if a user is interested in a financial product but feels a little uneasy, the system generates a response based on the prompt message "User is displaying a puzzled expression while asking about credit card benefits. Provide a soothing and clear explanation emphasizing the advantages of the product, using simple and reassuring language." In this way, the user can proceed with financial transactions with greater confidence.

[0622] The server integrates sentiment inference with past interaction history to provide users with the most relevant financial product information and improve the accuracy of interactions. This system significantly enhances the user experience, as users not only receive information but also enjoy a conversational experience that takes emotions into account.

[0623] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0624] Step 1:

[0625] The user launches a dedicated application on their device. The device captures the authentication information entered by the user and sends it to the server.

[0626] Input: User authentication information (e.g., username, password)

[0627] Output: Authentication information sent to the server

[0628] Step 2:

[0629] The server uses the received authentication information to perform the authentication process. If authentication is successful, it prepares the virtual environment data for the user.

[0630] Input: Authentication information

[0631] Output: Virtual environment data

[0632] Step 3:

[0633] The terminal generates a virtual space using augmented reality or virtual reality technology based on the received virtual environment data. The user then begins interacting with the system through this space.

[0634] Input: Virtual environment data

[0635] Output: Generated virtual space

[0636] Step 4:

[0637] Users interact with intelligence in a virtual space through voice and actions. The terminal captures this voice and facial expression data and sends it to the server.

[0638] Input: User's voice and facial expression data

[0639] Output: Audio and facial expression data sent to the server

[0640] Step 5:

[0641] The server uses Google Cloud Speech-to-Text and Facial Recognition libraries to convert speech data into text and analyze facial expression data to infer emotions. Based on these analysis results, IBM Watson Tone Analyzer is used to generate the optimal response.

[0642] Input: Voice and facial expression data

[0643] Output: Predicted sentiment data and generated responses

[0644] Step 6:

[0645] The server sends the generated response to the terminal and displays it to the user through intelligence. The user receives this response and gets support in making decisions about financial products.

[0646] Input: Generated response

[0647] Output: Response provided to the user

[0648] Step 7:

[0649] The server integrates the user's past interaction history with newly inferred sentiment data to provide the user with personalized financial product information.

[0650] Input: Interaction history and sentiment data

[0651] Output: Personalized financial product information

[0652] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0653] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0654] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0655] [Fourth Embodiment]

[0656] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0657] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0658] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0659] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0660] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0661] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0662] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0663] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0664] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0665] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0666] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0667] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0668] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0669] The virtual bank branch system according to the present invention is designed to meet the diverse needs and environments of various users. Users install a dedicated application on their user terminal and communicate with the server via the internet.

[0670] First, the user launches the app and enters their authentication information, which the device then sends to the server. The server verifies the received authentication information against its database, and only if authentication is successful does it send virtual environment data to the user. Using this virtual environment data, the device utilizes AR / VR technology to generate a three-dimensional virtual bank branch for the user.

[0671] In the generated virtual space, an AI bank teller is on standby, and users can communicate with the AI ​​via text, voice, or gestures, as if they were in a real branch. The user's voice and text input are processed by the terminal and sent to the server. The AI ​​on the server identifies the user's requests using advanced analytical capabilities and sends the optimal response back to the terminal in real time.

[0672] Furthermore, this system can analyze a user's past transaction history and provide individually optimized financial product information. When a user wishes to proceed with specific procedures such as opening a new account or consulting about a loan, the server automatically generates the necessary information and provides appropriate forms and documents through the user's terminal. The user completes the procedure online by filling out these documents and submitting them to the server.

[0673] For example, if a user wishes to open a new account, they can inform the AI ​​banker of this, and the server will send the necessary procedural information to the user's terminal. The user can then complete the process by entering the required information into a form.

[0674] Thus, the present invention provides convenient and efficient banking services that transcend geographical and temporal constraints, and is designed to be easily used, especially by elderly people who are unfamiliar with digital technology, through an intuitive interface.

[0675] The following describes the processing flow.

[0676] Step 1:

[0677] The user launches the virtual bank branch application on their terminal. The user enters their authentication information (e.g., user ID and password, or biometric authentication data) on the authentication screen.

[0678] Step 2:

[0679] The terminal receives the entered authentication information, encrypts it, and then sends it to the server. The server compares the received authentication information with its database to determine whether the authentication was successful or not.

[0680] Step 3:

[0681] If the server confirms successful authentication, it generates virtual environment data and sends it to the terminal. The terminal receives the virtual environment data and uses AR / VR technology to generate a virtual space that is reflected in the user's vision.

[0682] Step 4:

[0683] The user visually recognizes an AI bank teller in a virtual space and initiates a request via text or voice. The terminal transcribes the user's statements and input into text and sends that data to the server.

[0684] Step 5:

[0685] The server uses generative AI to analyze the user's intent and generate appropriate responses and suggestions. The generated responses are then sent to the terminal and provided to the user in real time.

[0686] Step 6:

[0687] If a user requests details or procedural information for a specific financial service, the server will refer to the user's transaction history and prepare individually optimized product information and procedural steps. It will then send the necessary digital documents to the user's device.

[0688] Step 7:

[0689] The terminal displays generated suggestions and required documents to the user, allowing the user to ask additional questions about unclear points and enter information into the documents. If the user proceeds with the procedure, the process is completed by sending the entered information back to the server.

[0690] This process allows users to receive necessary banking services without physically visiting a bank branch.

[0691] (Example 1)

[0692] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0693] In modern society, there is a demand for efficient and intuitive financial services that transcend geographical and temporal constraints. However, current systems have challenges such as the inconvenience users have to visit physical financial institutions and a lack of adequate support for users unfamiliar with digital technology. In particular, for users with little digital experience, such as the elderly, the complexity of online operations prevents them from fully realizing the benefits of these services.

[0694] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0695] In this invention, the server includes means for receiving identification information from a user information processing device and performing an authentication process, means for transmitting virtual reality data to the user information processing device after authentication and constructing a virtual environment, and means for an information processing device equipped with enhanced intelligence that interacts with the user information processing device using text information, voice information, or motion recognition. This enables users to enjoy intuitive and efficient financial services online without being restricted by their location or time.

[0696] A "user information processing device" is a device operated by a user that has functions to enable the input and display of identification information and the experience of a virtual environment.

[0697] "Identification information" refers to information necessary for user authentication, and typically includes user IDs, passwords, and biometric authentication data.

[0698] The "authentication process" refers to a series of procedures that verify the legitimacy of a user based on identification information received from a user information processing device.

[0699] "Virtual reality data" refers to data transmitted to a user information processing device, and includes information for building a virtual environment and providing an experience to the user.

[0700] A "virtual environment" is a space that provides three-dimensional visual and auditory information generated through information processing, enabling users to have an experience that closely resembles reality.

[0701] An "intelligence-enhanced information processing device" is a device that utilizes artificial intelligence technology to generate the optimal response based on the input information.

[0702] "Textual information, audio information, and motion recognition" refer to various forms of data that users input through information processing devices, and are means of communication.

[0703] "Financial services information" refers to information about financial products and services provided to users, including content optimized to meet user needs.

[0704] "Business history and interaction history" refers to records of transactions and communications when a user uses the system, and is data used to improve the provision of services to users.

[0705] A "storage device" is a device that stores information and makes it easy to retrieve when needed.

[0706] "Response accuracy" is an indicator that shows the ability of artificial intelligence to provide appropriate and accurate responses to user input.

[0707] This invention provides a system in which a user information processing device, a server, and an intelligent information processing device work in cooperation to improve user convenience. This system begins with the user installing a dedicated application on the information processing device. The information processing device includes devices such as smartphones and tablets, which communicate with the server via the internet.

[0708] The server receives identification information from the user information processing device and performs an authentication process. Database matching technology is used in this process to verify the user's identity. Upon successful authentication, the server uses a graphics engine such as OpenGL or Unity to send virtual reality data to the user information processing device. This allows the information processing device to generate a virtual environment and provide the user with a three-dimensional visual experience.

[0709] Within this virtual environment, users can interact with an intelligent information processing device through text, voice, or motion recognition. For example, a user wishing to open a new account might use a prompt such as "I want to open a new account." This input is parsed, and the server generates the most appropriate response to the user's needs and sends it back to the information processing device.

[0710] The enhanced intelligence-based information processing device uses generative AI models and machine learning techniques to analyze the user's business history and dialogue history. This generates and provides the user with individually optimized financial service information. An intuitive interface is provided to support user operation, making it easy to use, especially for users unfamiliar with digital technology.

[0711] Thus, the present invention realizes an environment in which users can receive efficient and intuitive financial services without being restricted by location or time.

[0712] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0713] Step 1:

[0714] The user installs a dedicated application on the information processing device. Upon launching the application, the user enters their user ID and password or biometric authentication data as authentication information. The device then sends this information to the server. The entered authentication information is transmitted securely using encryption technology.

[0715] Step 2:

[0716] The server receives authentication information from the terminal and compares it with the registered information in the database. The server uses a database search algorithm to verify that the user's authentication information matches. If authentication is successful, the server generates virtual reality data and sends it to the terminal. At this time, customization is applied based on the user's history.

[0717] Step 3:

[0718] The device generates a virtual environment using a graphics engine (such as OpenGL or Unity) based on the received virtual reality data. Through this virtual environment, the user experiences the appearance of a three-dimensional virtual bank branch right before their eyes. The device utilizes AR / VR technology to provide the user with visual information.

[0719] Step 4:

[0720] Within the virtual environment, users provide input through text, voice, or actions. For example, a user might say, "I want to open a new account." The terminal analyzes this input, converts it into text data using speech recognition technology, and sends it to the server.

[0721] Step 5:

[0722] The server uses a generated AI model based on the analysis of user input to recognize user requests. The AI ​​model performs data calculations (natural language processing) to generate the optimal response to the input prompt. The server generates the response data and sends it back to the terminal.

[0723] Step 6:

[0724] The terminal receives response data from the server and provides feedback to the user. This includes responses using speech synthesis technology and text messages on the screen. The user then decides on their next action based on this feedback. This process enables an environment where financial services can be accessed smoothly from any location.

[0725] (Application Example 1)

[0726] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0727] In modern society, geographical and temporal constraints on accessing financial services remain significant barriers for users. Furthermore, traditional financial procedures often lack sufficient optimization and individualized support for the user experience, creating a demand for efficient and personalized services. In particular, there is a growing need for intuitive interfaces for users unfamiliar with digital environments.

[0728] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0729] In this invention, the server includes means for receiving authentication information from a user device and performing authentication processing; means for transmitting virtual environment data to the user device after authentication and generating a virtual space; means for incorporating a machine learning model that communicates with the user device using text, voice, or actions; means for analyzing the user's operation history and dialogue history and generating individually optimized financial service information; means for transmitting the generated financial service information to the user device; means for receiving procedure requests from the user, generating related documents and transmitting them to the user device; means for generating a virtual financial facility using augmented reality technology; and means for generating voice or screen changes and providing the user with a flow of procedures. This enables users to receive intuitive and personalized financial services without being bound by location or time.

[0730] "User device" refers to a terminal device used by a user, and includes mobile information terminals such as smartphones and tablets.

[0731] "Authentication information" refers to data necessary to verify a user's identity, and includes user IDs, passwords, biometric information, and other similar information.

[0732] "Virtual environment data" refers to a set of information used to generate a digital three-dimensional space provided to the user, and is intended to realize virtual reality and augmented reality experiences.

[0733] A "machine learning model" is a part of a computer program that includes algorithms to optimize user interaction based on data and generate appropriate responses.

[0734] "Financial service information" refers to information about individually optimized financial products and services provided to users, and is generated based on the user's past transaction history and current needs.

[0735] Augmented reality technology is a technique that overlays digital information onto images of the real world, enabling users to interact with virtual objects.

[0736] A "virtual financial facility" is a digitally simulated facility that users can visit within a virtual environment, and it functions as a place where users can receive financial services.

[0737] To implement this invention, the user must first install a dedicated application on their user device, such as a smartphone. This application is designed to enhance the user experience using augmented reality and machine learning technologies. The user device communicates with a server via the internet to send and receive necessary data.

[0738] The server processes the authentication information received from the user terminal and verifies it against the database. Upon successful authentication, the server sends virtual environment data to the user terminal, where a virtual space is created. In this environment, machine learning models are utilized, enabling natural communication with the user.

[0739] The user terminal displays a generated virtual financial facility, allowing the user to visually immerse themselves in that space. AI-powered financial service information is generated on the server based on the user's actions and transaction history, and personalized content is provided to the user. Furthermore, if necessary, the server generates relevant digital documents and provides them through the user terminal.

[0740] The hardware will consist of mobile devices such as smartphones and tablets, while the software will utilize augmented reality development frameworks such as ARKit and ARCore, as well as TensorFlow and other machine learning engines. This system will leverage cloud infrastructure such as AWS and Google Cloud for data processing and storage.

[0741] One concrete example is a scenario where a user interacts with an AI banker in a virtual space and opens a new savings account. In this scenario, the AI ​​banker proposes the most suitable plan to the user via voice input, and the necessary procedures are carried out based on the user's selection.

[0742] An example of a prompt to input into the generating AI model might be: "Please describe in detail a scenario in which a user interacts with an AI in a virtual space and opens a new savings account. Pay particular attention to the use of voice input and a visual interface."

[0743] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0744] Step 1:

[0745] The user launches a dedicated application and enters their authentication information. The entered authentication information is sent from the terminal to the server. The server compares this information with existing data in the database and performs authentication. If authentication is successful, the server returns permission to proceed to the next step to the terminal.

[0746] Step 2:

[0747] The server generates virtual environment data and sends it to the user's terminal. Upon receiving this data, the terminal uses augmented reality technology to display a virtual financial facility to the user. The user enters this virtual space to begin a real-time experience.

[0748] Step 3:

[0749] The user interacts with an AI banker in a virtual space. When the user provides voice or text input, the terminal sends it to the server. The server analyzes the input using a generative AI model and generates an appropriate response. The generated response is then returned to the user, and the interaction continues.

[0750] Step 4:

[0751] Based on the user's past operation and interaction history, the server generates individually optimized financial service information. This information is generated by analyzing the user's transaction patterns and needs and processing the data. The resulting information is sent to the terminal and presented to the user.

[0752] Step 5:

[0753] When a user decides to use a specific financial service, the server generates the relevant digital documents. Taking the required data as input, the server automatically fills in the document format and sends the completed version to the user's terminal. The user then completes the requested procedures or applications through these digital documents.

[0754] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0755] The system according to the present invention is built to further personalize the user experience and enable natural communication by incorporating an emotion engine.

[0756] The user first launches a dedicated application installed on their device and enters authentication information to access the system. The device sends the entered authentication information to the server, which performs authentication. If authentication is successful, the server sends virtual environment data to the user's device. Based on the received data, the device generates a realistic virtual space using AR / VR technology. In this virtual space, the user can communicate with an AI banker.

[0757] The AI ​​bank teller can analyze the user's voice and facial expressions through an emotion engine to infer their emotions. Based on this emotion analysis, the server generates an optimal response tailored to the user's emotions and sends it to the terminal. For example, if the AI ​​bank teller determines that the user is nervous, it can respond in a gentle and calm tone to alleviate their anxiety.

[0758] Furthermore, the server integrates the acquired emotional data with the user's past interaction data for deeper analysis. Using the results of this analysis, it becomes possible to individually customize and provide the user with the most suitable financial product information. For example, if a user is interested in a new investment product but also feels anxious during the conversation, the server will select responses that gently explain the details and build trust.

[0759] In this way, by incorporating an emotion engine, this system achieves a more human-like interaction that goes beyond simple question-and-answering, greatly improving user satisfaction.

[0760] The following describes the processing flow.

[0761] Step 1:

[0762] The user launches a dedicated application on their device and enters their authentication information using their user ID and password or biometric authentication on the login screen.

[0763] Step 2:

[0764] The terminal sends the entered authentication information to the server. The server compares the received information with its internal database to determine whether authentication was successful.

[0765] Step 3:

[0766] Upon successful authentication, the server sends virtual environment data to the user's terminal. Based on the received data, the terminal uses AR / VR technology to generate the virtual space that the user will experience.

[0767] Step 4:

[0768] Users interact with an AI bank teller in a virtual space and make requests via voice or text. For example, when requesting a balance check, the terminal converts the voice into text and sends it to the server.

[0769] Step 5:

[0770] The server analyzes the received user voice data using an emotion engine, estimating emotions based on voice tone and user facial expression data. Based on the analysis results, it generates an appropriate response that matches the user's emotions.

[0771] Step 6:

[0772] The server sends the generated response data to the terminal. The terminal then receives a response from the AI ​​bank teller in voice or text format, in a tone that matches the user's emotions.

[0773] Step 7:

[0774] If a user wants to know more about a proposed financial product or procedure, the server takes the user's request and emotional state into consideration, generates personalized information and relevant documents, and sends them to the terminal.

[0775] Step 8:

[0776] The terminal displays documents to the user and allows the user to proceed through the interface if input is required. Through this entire process, the system provides the user with a natural and personalized experience.

[0777] (Example 2)

[0778] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0779] Modern users demand more personalized and natural interactions when acquiring information and processing information through digital systems. However, conventional systems fail to adequately generate appropriate responses based on user emotions or provide realistic experiences within virtual spaces. As a result, user satisfaction declines, and it becomes difficult to provide efficient and reliable services.

[0780] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0781] In this invention, the server includes an information processing device that receives authentication information and performs authentication procedures, an information processing device that transmits virtual environment data to an output device after authentication and generates visual information, and an artificial intelligence device equipped with an information processing device that analyzes the user's voice or facial expressions using emotion analysis technology and infers their emotional state. This makes it possible to generate personalized responses based on the user's emotions and to provide a realistic experience in a virtual space.

[0782] "Authentication information" refers to information used to identify a user and verify their access rights to the system.

[0783] An "information processing device" is a device that has the functions of receiving, analyzing, and transmitting data, and is used to perform various tasks.

[0784] An "output device" is a device that provides digital data to users through senses such as sight and hearing.

[0785] "Emotion analysis technology" is a technology that uses data such as voice and facial expressions to infer a user's emotional state.

[0786] An "artificial intelligence device" is a device that possesses automated learning and analytical capabilities and performs tasks by mimicking human intelligence.

[0787] A "generative AI model" is an algorithm and computational model used for artificial intelligence responses and data generation.

[0788] "Transaction information" refers to information that records the details of transactions performed by a user.

[0789] "Interaction information" refers to information that records the history of interactions between the user and the system.

[0790] "Service information" refers to information about products and services provided to users.

[0791] "Visual information" refers to visual content presented to users through displays or projectors.

[0792] This invention provides a system that generates personalized responses using emotion analysis technology to improve the user experience. Specific embodiments for carrying out the invention are described below.

[0793] The server uses an information processing device to process authentication information received from the user terminal. This device receives authentication data via a secure communication protocol and authenticates the user by comparing it with information in a database. Once authentication is confirmed, the server encrypts the virtual environment data and sends it to the user terminal.

[0794] The device receives this virtual environment data and uses AR / VR technology to enable the user to experience an immersive virtual space. Specifically, the device uses a high-performance graphics processing unit (GPU) to generate visual information and renders high-quality 3D images in real time.

[0795] The user interacts with an artificial intelligence device within the generated virtual space. This device utilizes emotion analysis technology to analyze the user's voice and facial expression data acquired through a voice input device and camera, and estimates the user's emotional state in real time. The analysis results are transmitted from the terminal to the server.

[0796] The server uses a generation AI model based on the analysis results to generate natural conversational text that aligns with the user's emotions in real time. This response is personalized based on past transaction and interaction information. The generated response is sent to the terminal, further enhancing the user's virtual reality experience.

[0797] As a concrete example, a scenario could be imagined where a user prompts an AI banker to "tell me about a new investment product," and the AI ​​banker, based on this information and considering the user's interests and concerns, provides a detailed and reassuring explanation.

[0798] In this way, the entire system can achieve interactions that are tailored to the individual emotions of each user, thereby increasing trust in and satisfaction with the service.

[0799] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0800] Step 1:

[0801] The user launches a dedicated application on their terminal and enters authentication information. This input data includes the user ID and password. The terminal initiates an authentication request by sending this authentication information to the server. The output is the transmission of authentication information to the server.

[0802] Step 2:

[0803] The server receives the authentication information and compares it with existing user data stored in its internal database. This data processing securely handles the entered authentication information and verifies the user's authentication. If authentication is successful, an authentication token is generated and sent to the terminal as output.

[0804] Step 3:

[0805] The server creates virtual environment data for authenticated users. This involves selecting and preparing customized 3D models and simulation data based on the service the user is trying to access. The output is the transmission of the prepared virtual environment data to the terminal.

[0806] Step 4:

[0807] The device uses received virtual environment data to generate a virtual space utilizing AR / VR technology. In this process, a graphics processing unit (GPU) is used to render 3D images in real time, providing the user with a high-quality visual experience. The output is the virtual space provided to the user.

[0808] Step 5:

[0809] The user begins an interaction with an AI bank teller in a virtual space. The user inputs prompts through their voice, and this voice data is captured by the terminal's voice input device. The output is the collected voice data.

[0810] Step 6:

[0811] The device analyzes voice data in real time using emotion analysis technology to infer the user's emotional state. This analysis includes voice tone and facial expression analysis. The output is the inferred emotional state.

[0812] Step 7:

[0813] The server uses a generative AI model based on the results of sentiment analysis to generate the most appropriate response for the user. This response generation process considers the user's emotional state and past transaction information to create personalized conversation content. The output is the generated response message.

[0814] Step 8:

[0815] The terminal receives response messages from the server and transmits them to the user by playing them back as audio or text within the virtual space. The output is the audio or text response delivered to the user.

[0816] This series of steps allows the system to provide users with personalized interactions and natural dialogue, thereby improving the user experience.

[0817] (Application Example 2)

[0818] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0819] In modern financial services, while user interaction is efficient, it lacks emotional depth and personalized experiences. This can lead to users proceeding with transactions while harboring anxieties and doubts, resulting in a lower user experience and satisfaction. Online financial transactions, in particular, struggle to replicate the sense of security that comes from face-to-face interactions. Therefore, there is a need for more natural and trustworthy interactions while minimizing user intervention.

[0820] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0821] In this invention, the server includes means for receiving authentication information from a user terminal and performing an authentication procedure, means for transmitting virtual environment data to the user terminal after authentication and generating a virtual space, and means for analyzing the user's voice and facial expressions to infer emotions and optimizing the response of the intelligence in the virtual space based on the analysis results. This makes it possible for users to receive personalized and emotionally sensitive financial services without anxiety or doubt.

[0822] A "user terminal" is a device used by a user and functions as an interface for executing virtual spaces and authentication processes.

[0823] "Authentication information" refers to data necessary to verify the user's identity and guarantees the security of system access.

[0824] "Virtual environment data" refers to the information necessary to construct the digital space that users access through an interface.

[0825] A "virtual space" is a digital interface built using augmented reality or virtual reality technologies, a space where users interact in real time.

[0826] "Intelligence" refers to algorithms and systems that understand user statements and actions and provide appropriate responses and services.

[0827] "Transaction history" refers to a series of records of financial transactions conducted by a user in the past, and is used to analyze the user's financial behavior.

[0828] "Interaction history" refers to a record of data generated through user interactions with the system, and contributes to the optimization of the service.

[0829] "Financial product information" refers to data about specific financial services and products provided to users.

[0830] "Voice and facial expression analysis" is a technology that analyzes voice and visual data to infer the user's emotions and adjust responses accordingly.

[0831] "Response optimization" is the process of generating the most appropriate and effective response based on the user's specific state and requests.

[0832] The system implementing this invention consists of a user terminal, a server, and a generative AI model. The user terminal refers to a device such as a smartphone or a head-mounted display, through which the user accesses and interacts with a virtual space. First, the user launches a dedicated application installed on the terminal, enters authentication information, and sends it to the server. The server analyzes the voice and facial recognition data using Google Cloud Speech-to-Text and Facial Recognition libraries to infer the user's emotions. Based on this emotion analysis, IBM Watson Tone Analyzer plays a role in generating the optimal response.

[0833] The server processes the captured user's voice and facial expression data in real time and sends a customized response to the user's terminal. This response allows the virtual intelligence to provide the user with an emotionally responsive service. For example, if a user is interested in a financial product but feels a little uneasy, the system generates a response based on the prompt message "User is displaying a puzzled expression while asking about credit card benefits. Provide a soothing and clear explanation emphasizing the advantages of the product, using simple and reassuring language." In this way, the user can proceed with financial transactions with greater confidence.

[0834] The server integrates sentiment inference with past interaction history to provide users with the most relevant financial product information and improve the accuracy of interactions. This system significantly enhances the user experience, as users not only receive information but also enjoy a conversational experience that takes emotions into account.

[0835] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0836] Step 1:

[0837] The user launches a dedicated application on their device. The device captures the authentication information entered by the user and sends it to the server.

[0838] Input: User authentication information (e.g., username, password)

[0839] Output: Authentication information sent to the server

[0840] Step 2:

[0841] The server uses the received authentication information to perform the authentication process. If authentication is successful, it prepares the virtual environment data for the user.

[0842] Input: Authentication information

[0843] Output: Virtual environment data

[0844] Step 3:

[0845] The terminal generates a virtual space using augmented reality or virtual reality technology based on the received virtual environment data. The user then begins interacting with the system through this space.

[0846] Input: Virtual environment data

[0847] Output: Generated virtual space

[0848] Step 4:

[0849] Users interact with intelligence in a virtual space through voice and actions. The terminal captures this voice and facial expression data and sends it to the server.

[0850] Input: User's voice and facial expression data

[0851] Output: Audio and facial expression data sent to the server

[0852] Step 5:

[0853] The server uses Google Cloud Speech-to-Text and Facial Recognition libraries to convert speech data into text and analyze facial expression data to infer emotions. Based on these analysis results, IBM Watson Tone Analyzer is used to generate the optimal response.

[0854] Input: Voice and facial expression data

[0855] Output: Predicted sentiment data and generated responses

[0856] Step 6:

[0857] The server sends the generated response to the terminal and displays it to the user through intelligence. The user receives this response and gets support in making decisions about financial products.

[0858] Input: Generated response

[0859] Output: Response provided to the user

[0860] Step 7:

[0861] The server integrates the user's past interaction history with newly inferred sentiment data to provide the user with personalized financial product information.

[0862] Input: Interaction history and sentiment data

[0863] Output: Personalized financial product information

[0864] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0865] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0866] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0867] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0868] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0869] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0870] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0871] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0872] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0873] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0874] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0875] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0876] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0877] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0878] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0879] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0880] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0881] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0882] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0883] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0884] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0885] The following is further disclosed regarding the embodiments described above.

[0886] (Claim 1)

[0887] A means of receiving authentication information from the user terminal and performing the authentication procedure,

[0888] A means of sending virtual environment data to the user terminal after authentication and generating a virtual space,

[0889] A means of incorporating artificial intelligence that communicates with a user terminal using text, voice, or gestures,

[0890] A means for analyzing a user's transaction history and interaction history to generate individually optimized financial product information,

[0891] A means for transmitting generated financial product information to the user terminal,

[0892] A means for receiving procedural requests from users, generating related documents, and sending them to the user's terminal,

[0893] A system that includes this.

[0894] (Claim 2)

[0895] The system according to claim 1, characterized in that it includes means for improving the accuracy of responses by artificial intelligence by accumulating user interactions in a virtual space in a database and analyzing that data.

[0896] (Claim 3)

[0897] The system according to claim 1, characterized in that it includes means for generating visual information that allows the user to experience as if they were in the real world, using augmented reality or virtual reality technology installed in the user terminal.

[0898] "Example 1"

[0899] (Claim 1)

[0900] A means for receiving identification information from a user information processing device and performing an authentication process,

[0901] A means of transmitting virtual reality data to a user information processing device after authentication and constructing a virtual environment,

[0902] A means comprising an information processing device with enhanced intelligence that interacts with a user information processing device using text information, audio information, or motion recognition,

[0903] A means for analyzing a user's business history and dialogue history to generate individually optimized financial service information,

[0904] A means for transmitting generated financial service information to a user information processing device,

[0905] A means for receiving business requests from users, generating necessary documents, and transmitting them to a user information processing device,

[0906] A system that includes this.

[0907] (Claim 2)

[0908] The system according to claim 1, characterized in that it includes means for improving the response accuracy of an intelligent information processing device by storing user interactions within a virtual environment in a storage device and analyzing the information.

[0909] (Claim 3)

[0910] The system according to claim 1, characterized in that it includes means for generating visual information that allows the user to experience as if they were in the real world, using augmented reality or virtual reality technology provided by the user information processing device.

[0911] "Application Example 1"

[0912] (Claim 1)

[0913] A means for receiving authentication information from a user device and performing authentication processing,

[0914] A means of sending virtual environment data to the user device after authentication and generating a virtual space,

[0915] A means for incorporating a machine learning model that communicates with a user device using text, voice, or actions,

[0916] A means for analyzing a user's operation history and dialogue history to generate individually optimized financial service information,

[0917] Means for transmitting generated financial service information to a user device,

[0918] A means for receiving procedural requests from users, generating related documents, and sending them to the user's device,

[0919] A means of generating a virtual financial facility using augmented reality technology,

[0920] A means of generating audio or screen changes and providing the user with a procedural flow,

[0921] A system that includes this.

[0922] (Claim 2)

[0923] The system according to claim 1, characterized in that it includes means for improving the response accuracy of a machine learning model by accumulating user interactions in a virtual space in a database and analyzing that data.

[0924] (Claim 3)

[0925] The system according to claim 1, characterized in that it includes means for generating visual information that the user can perceive as real, using augmented reality technology installed in the user device.

[0926] "Example 2 of combining an emotion engine"

[0927] (Claim 1)

[0928] An information processing device that receives authentication information and performs authentication procedures,

[0929] An information processing device that, after authentication, transmits virtual environment data to an output device and generates visual information,

[0930] An artificial intelligence device equipped with an information processing device that analyzes the user's voice or facial expressions using emotion analysis technology and infers their emotional state,

[0931] An information processing device that uses a generative AI model to generate responses that are in line with the user's emotions,

[0932] An information processing device that analyzes transaction information and interaction information and generates individually optimized service information,

[0933] An information processing device that transmits the generated service information to an output device,

[0934] An information processing device that receives a request from a user, generates related information, and transmits it to an output device,

[0935] A system that includes this.

[0936] (Claim 2)

[0937] The system according to claim 1, characterized by comprising an information processing device that stores user interactions within a virtual space in a memory device and analyzes the information to improve the response accuracy of artificial intelligence.

[0938] (Claim 3)

[0939] The system according to claim 1, characterized in that it includes an information processing device that generates visual information that allows the user to experience as if they were in the real world, using augmented reality or virtual reality technology incorporated in the output device.

[0940] "Application example 2 when combining with an emotional engine"

[0941] (Claim 1)

[0942] A means of receiving authentication information from the user terminal and performing the authentication procedure,

[0943] A means of sending virtual environment data to the user terminal after authentication and generating a virtual space,

[0944] A means of incorporating intelligence to communicate with a user terminal using text, voice, or actions,

[0945] A means for analyzing a user's transaction history and interaction history to generate individually optimized financial product information,

[0946] A means for transmitting generated financial product information to the user terminal,

[0947] A means for receiving a procedure request from a user, generating related documents, and sending them to the user's terminal,

[0948] A means for analyzing the user's voice and facial expressions to infer emotions, and optimizing the response of an intelligence in a virtual space based on the analysis results,

[0949] A system that includes this.

[0950] (Claim 2)

[0951] The system according to claim 1, characterized in that it includes means for improving the accuracy of intelligent responses by accumulating interactions with users in a virtual space and analyzing that data.

[0952] (Claim 3)

[0953] The system according to claim 1, characterized in that it includes means for generating visual information that allows the user to experience as if they were in the real world, using augmented reality or virtual reality technology installed in the user terminal. [Explanation of symbols]

[0954] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of receiving authentication information from the user terminal and performing the authentication procedure, A means of sending virtual environment data to the user terminal after authentication and generating a virtual space, A means of incorporating artificial intelligence that communicates with a user terminal using text, voice, or gestures, A means for analyzing a user's transaction history and interaction history to generate individually optimized financial product information, A means for transmitting generated financial product information to the user terminal, A means for receiving procedural requests from users, generating related documents, and sending them to the user's terminal, A system that includes this.

2. The system according to claim 1, characterized in that it includes means for improving the accuracy of responses by artificial intelligence by accumulating user interactions in a virtual space in a database and analyzing that data.

3. The system according to claim 1, characterized in that it includes means for generating visual information that allows the user to experience as if they were in the real world, using augmented reality or virtual reality technology installed in the user terminal.

Images