System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting a user's emotional response, facial expression, heart rate, voice tone, behavioral pattern, and biometric data; means for encrypting the collected data and transmitting the encrypted data to a server; means for integrating and analyzing the received data at the server; means for authenticating the user based on the analysis result; and means for providing access permission to the user based on the authentication result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional authentication methods in the metaverse cannot fully reflect a user's personality, emotions, behavioral patterns, biometric characteristics, or environmental conditions, resulting in security risks and limitations in privacy protection. This makes impersonation and unauthorized access more likely, and limits the user experience. Therefore, it is necessary to provide a more accurate and secure authentication system that comprehensively analyzes a user's emotional responses, behavioral patterns, and biometric data. [Means for solving the problem]

[0005] The above-mentioned problem is solved by a system including a means for collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, a means for encrypting the collected data and sending it to a server, a server means for integrating and analyzing the received data, a server means for authenticating the user based on the analysis results, a means for notifying the terminal of the authentication results, and a means for providing the user with access permission based on the authentication results.

[0006] "User" refers to an individual who uses the system, and whose emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data are collected.

[0007] "Emotional response" refers to the psychological and physiological reaction a user has to a stimulus or situation, and is primarily derived from data such as facial expressions, tone of voice, and behavior.

[0008] "Facial expressions" are changes and movements of a user's face, and are an important element in expressing emotions and intentions. They are analyzed using facial recognition algorithms.

[0009] "Heart rate" refers to the number of times a user's heart beats per minute, and is part of the biometric data measured through biometric sensors.

[0010] "Tone of voice" refers to the quality or intonation of a user's voice, and is an element that is analyzed using a voice recognition algorithm.

[0011] "Behavioral patterns" refer to a user's movements and behavioral tendencies, and are part of the data analyzed using behavior tracking algorithms.

[0012] "Biometric data" refers to information collected based on a user's physical characteristics, such as heart rate, fingerprint, or iris.

[0013] "Collection Methods" refers to devices or technologies used to detect and acquire a user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data.

[0014] "Encryption Methods" refers to technology that encrypts collected data to protect it from unauthorized access.

[0015] "Server" refers to a computer system that manages, analyzes, and stores data on a network.

[0016] "Synthesis" refers to the process of combining multiple collected data into one comprehensive data set.

[0017] "Analysis means" refers to algorithms or technologies used to analyze the aggregated user data and identify individuals.

[0018] "Authentication" refers to the process of verifying a user's identity and granting access rights.

[0019] "Notification means" refers to the communication means or technology used to notify the user of the authentication result.

[0020] "Access permission" refers to the authority that allows a successfully authenticated user to access a system or service. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] Explaining program processing in natural language

[0043] Collection and transmission of user data

[0044] When a user puts on a VR headset and attempts to access the Metaverse, the device collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using facial recognition cameras, heart rate sensors, microphones, etc. The collected data is then stored in temporary memory by the device.

[0045] Data encryption and transmission

[0046] The device encrypts the collected data using an encryption algorithm (e.g., AES-256), and then transmits the encrypted data to the server using a secure communication protocol (e.g., HTTPS).

[0047] Data reception and analysis

[0048] The server receives and decrypts the encrypted data sent from the device. The decrypted data is separated into different categories (emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data). This data is then integrated using multimodal generative AI to generate a comprehensive user profile.

[0049] Analysis of individual data

[0050] The server analyzes the individual data in the following way:

[0051] Emotional responses and facial expressions are run through a facial recognition algorithm.

[0052] The tone of voice is analyzed using a voice recognition algorithm.

[0053] The heart rate is analyzed by the biosensor's data analysis algorithm.

[0054] The behavioral patterns are analyzed based on a motion prediction algorithm.

[0055] User Authentication

[0056] The server compares the generated user profile with existing user data in its internal database. Based on the comparison result, it determines whether the user is legitimate. If the user is legitimate, authentication is successful and a notification of this is sent to the terminal.

[0057] Notification of access permission to users

[0058] The terminal receives the authentication result from the server and, if successful, notifies the user of the access permission, allowing the user to log in to the metaverse. If the authentication fails, a corresponding warning message is displayed.

[0059] Specific examples

[0060] User logging into the metaverse scenario

[0061] 1. A user puts on a VR headset and attempts to access the Metaverse.

[0062] The user enters their ID and password and confirms the login screen.

[0063] 2. The device activates sensors and collects user data.

[0064] The device will enable a facial recognition camera, heart rate sensor, and microphone to collect the user's facial expressions, heart rate, and voice data.

[0065] 3. The device encrypts the data and sends it to the server.

[0066] The terminal encrypts the data with AES-256 and sends it to the server via the HTTPS protocol.

[0067] 4. The server receives and decrypts the data.

[0068] The server receives the data and decrypts it with AES-256.

[0069] 5. The server consolidates and analyzes the data.

[0070] The server performs facial recognition, voice recognition, and heart rate analysis to obtain an integrated result.

[0071] 6. The server authenticates the user against its user database.

[0072] The server checks its user database for a matching record.

[0073] 7. The server notifies the terminal of the authentication result.

[0074] The server notifies the terminal of successful authentication.

[0075] 8. The device notifies the user of the access permission.

[0076] The device displays the access permissions to the user and logs them into the metaverse.

[0077] In this way, the present invention can accurately and securely identify users, increasing safety and engagement within the metaverse.

[0078] The processing flow will be explained below.

[0079] Step 1:

[0080] A user puts on a VR headset and attempts to access the Metaverse.

[0081] The user enters their ID and password and proceeds to the login screen.

[0082] Step 2:

[0083] The device activates sensors and collects user data.

[0084] The device uses a facial recognition camera to collect facial expression data from the user.

[0085] The device uses a heart rate sensor to collect the user's heart rate data.

[0086] The terminal uses a microphone to collect data on the tone of the user's voice.

[0087] The device uses motion sensors to collect data on the user's behavioral patterns.

[0088] Step 3:

[0089] The device encrypts the collected data.

[0090] The device uses the AES-256 encryption algorithm to encrypt collected facial expression data, heart rate data, tone of voice data, and behavioral pattern data.

[0091] Step 4:

[0092] The device sends the encrypted data to the server.

[0093] The device uses the HTTPS protocol to securely send encrypted data to the server.

[0094] Step 5:

[0095] The server decrypts the received data.

[0096] The server decrypts the received encrypted data using the AES-256 encryption algorithm.

[0097] Step 6:

[0098] The server consolidates the decrypted data.

[0099] The server integrates facial recognition data, heart rate data, tone of voice data, and behavioral pattern data using multimodal generation AI to generate a user profile.

[0100] Step 7:

[0101] The server analyzes the individual data.

[0102] The server analyzes the facial expression data using a facial recognition algorithm.

[0103] The server analyzes the tone of voice data using a voice recognition algorithm.

[0104] The server analyzes the heart rate data using a heart rate analysis algorithm.

[0105] The server analyzes the behavioral pattern data using a behavioral prediction algorithm.

[0106] Step 8:

[0107] The server authenticates the user based on the analysis results.

[0108] The server matches the generated user profile with existing data in its internal database.

[0109] Based on the result of the comparison, the server determines whether the user is legitimate.

[0110] Step 9:

[0111] The server notifies the terminal of the authentication result.

[0112] If the authentication is successful, the server notifies the terminal of access permission.

[0113] If the authentication fails, the server notifies the terminal of access denial.

[0114] Step 10:

[0115] The terminal notifies the user of the authentication result.

[0116] If the authentication is successful, the terminal allows the user to access the metaverse.

[0117] If the authentication fails, the terminal displays a warning message to the user.

[0118] Example 1

[0119] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0120] Conventional virtual reality (VR) systems have faced problems with insufficiently secure user authentication methods, resulting in a high risk of easy unauthorized access. Furthermore, there was a lack of a means for centralized and detailed analysis of individual user data, making it difficult to improve the quality of the user experience. Furthermore, there was no established secure method for transferring collected data, creating a risk of data leakage. Therefore, providing a highly accurate and secure authentication system that users could use with confidence was a challenge.

[0121] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0122] In this invention, the server includes means for collecting emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using a facial recognition camera, heart rate sensor, and microphone when a user wears a VR headset and attempts to access the metaverse, means for encrypting the collected data using the AES-256 algorithm and transmitting it to the server via HTTPS protocol, and server means for decrypting the received data and integrating and analyzing the emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using multimodal generation AI, thereby enabling secure transfer of the collected data and highly accurate user authentication.

[0123] "User" refers to an individual who wears a VR headset and accesses the Metaverse.

[0124] "Device" refers to the device including the VR headset worn by the user and its related equipment.

[0125] "Server" refers to a computer on the system side that receives data sent from a user and analyzes and authenticates it.

[0126] A "face recognition camera" refers to a photographic device that captures a user's facial expressions and features.

[0127] "Heart rate sensor" refers to a detection device for measuring a user's heart rate.

[0128] "Microphone" refers to a recording device for collecting the user's voice.

[0129] "Emotional response" refers to changes in the emotions shown by the user, and is determined from facial expressions, tone of voice, etc.

[0130] "Facial expressions" refer to movements that show emotions and reactions on the user's face.

[0131] "Heart rate" refers to the number of times a user's heart beats within a certain period of time.

[0132] "Tone of voice" refers to characteristics such as the pitch and intonation of a user's voice.

[0133] "Behavioral patterns" refer to a series of actions and tendencies of a user's reactions.

[0134] "Biometric data" refers to various information collected from a user's body (e.g., heart rate, pulse, body temperature).

[0135] The "AES-256 algorithm" is one of the Advanced Encryption Standards for encrypting data, and uses a 256-bit key.

[0136] "HTTPS protocol" refers to a communications protocol for securely transferring data over the Internet.

[0137] "Multimodal generative AI" refers to an artificial intelligence model that integrates and analyzes multiple data modalities (e.g., audio, images, text).

[0138] "Encryption" refers to the process of transforming data so that it cannot be read by third parties.

[0139] "Decryption" refers to the process of restoring encrypted data to its original state.

[0140] "User profile" refers to a collection of information about an individual user that is generated by integrating various collected data.

[0141] "Authentication" refers to the process of verifying that a user is a specific individual.

[0142] "Access permission" refers to granting a user the right to access a particular system or information.

[0143] "Warning message" refers to the notification displayed to the user when authentication fails.

[0144] This invention is a system for accurately and securely authenticating users when they wear a VR headset and access the Metaverse. This system works through several steps, mainly involving data collection, encryption, transmission, reception, analysis, and authentication processes between the device, server, and user.

[0145] User Data Collection

[0146] When a user puts on a VR headset and starts an application, the device first activates sensors, including a facial recognition camera, heart rate sensor, and microphone. These devices collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. For example, the facial recognition camera captures facial expression data, the heart rate sensor monitors heart rate, and the microphone records tone of voice.

[0147] Data encryption and transmission

[0148] The collected data is temporarily stored on the device and then encrypted using the AES-256 encryption algorithm, and then transmitted to the server using the HTTPS protocol. This process ensures the security of the data.

[0149] Data analysis on the server

[0150] Once the data arrives at the server, it is first decoded. After decoding, the data is separated into categories (emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data). The server then uses multimodal generative AI to integrate and analyze this data. This results in a comprehensive user profile, which includes, for example, current emotional state, heart rate trends, and usual behavioral patterns.

[0151] User Authentication

[0152] The generated user profile is compared with existing user data stored in the server's internal database. This comparison determines whether the user has valid access rights. If authentication is successful, the server notifies the terminal of the result. If authentication fails, the terminal displays a warning message to the user.

[0153] User notification and access permissions

[0154] The terminal receives the authentication result from the server and notifies the user of the access permission if successful, allowing the user to log in to the metaverse and access interactive content. If the authentication fails, a corresponding warning message will be displayed to the user, prompting them to try again.

[0155] The system will improve the user experience and ensure safety within the metaverse by collecting, analyzing, and authenticating user data with high accuracy and security.

[0156] Prompt Sentence Examples

[0157] Finally, the following are some specific examples of prompts for generative AI models:

[0158] "Please explain in natural language the system that collects biometric data from users using a VR headset and analyzes it to authenticate the user."

[0159] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0160] Step 1:

[0161] The user puts on a VR headset and prepares to access the Metaverse.

[0162] The user turns on the VR headset, launches the application, and proceeds to the login screen where they enter their ID and password.

[0163] Input: User ID and password

[0164] Output: Access to login screen

[0165] Step 2:

[0166] The device activates sensors and collects user data.

[0167] The device activates a facial recognition camera, heart rate sensor and microphone to simultaneously collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns and biometric data.

[0168] Input: User's current biometric and behavioral data

[0169] Data processing / data calculation: Analyzes images taken with a facial recognition camera and extracts facial expression data. Records heart rate measured with a heart rate sensor. Analyzes audio data collected with a microphone and identifies voice tone.

[0170] Output: Multiple types of biometric data collected

[0171] Step 3:

[0172] Encrypt the data collected by the device.

[0173] The device encrypts all collected data using the AES-256 encryption algorithm.

[0174] Input: Unencrypted biometric data

[0175] Data processing / data calculation: Encrypt data using the AES-256 algorithm

[0176] Output: Encrypted biometric data

[0177] Step 4:

[0178] The encrypted data is sent to the server.

[0179] The device uses the HTTPS protocol to securely transmit encrypted data to the server.

[0180] Input: Encrypted biometric data

[0181] Output: Send data to the server

[0182] Step 5:

[0183] The server receives and decrypts the data.

[0184] The server receives the data using the HTTPS protocol and decrypts it using the AES-256 algorithm.

[0185] Input: Encrypted biometric data

[0186] Data processing / data calculation: Receive data using HTTPS protocol and decrypt it using AES-256 algorithm

[0187] Output: Decoded biometric data

[0188] Step 6:

[0189] The server categorizes and analyzes the data.

[0190] The server separates the decoded data into emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, and then uses multimodal generative AI to integrate and analyze this data.

[0191] Input: Decoded biometric data

[0192] Data processing / data calculation: Categorize data and perform integrated analysis using multimodal generation AI

[0193] Output: Comprehensive user profile

[0194] Step 7:

[0195] The server checks the user profile against an internal database.

[0196] The server compares the generated user profile with existing user data in its internal database and authenticates the user based on the comparison result.

[0197] Input: User profile and existing user database

[0198] Data processing / data calculation: Compare and collate profile data with existing databases

[0199] Output: Authentication result (success or failure)

[0200] Step 8:

[0201] The authentication result is notified to the terminal, and access permission is provided to the user.

[0202] The server sends the authentication result (success or failure) to the terminal. If the authentication is successful, the terminal notifies the user of the access permission and allows them to log in to the metaverse. If the authentication fails, the terminal displays a warning message.

[0203] Input: Authentication result

[0204] Output: Notification of permission or warning message

[0205] As described above, this system collects user data and securely analyzes and authenticates it, enabling accurate and secure user authentication.

[0206] (Application example 1)

[0207] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0208] In recent years, the importance of security authentication in companies has increased, and traditional authentication using IDs and passwords is often insufficient. In particular, more advanced authentication systems are required to prevent unauthorized employee access and data leaks. However, currently used biometric authentication systems have room for improvement in terms of user experience and authentication accuracy. For these reasons, there is a demand for more secure and accurate authentication methods in corporate security authentication systems.

[0209] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0210] In this invention, the server includes means for collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, means for encrypting the collected data and transmitting it to the server, server means for integrating and analyzing the received data, server means for authenticating the user based on the analysis result, means for notifying the terminal of the authentication result, means for providing the user with access permission based on the authentication result, and means for collecting, analyzing, and authenticating user data using a VR headset in a corporate security authentication system. This enables companies to achieve more accurate and secure security authentication and reduce the risk of unauthorized access and data leakage.

[0211] "User" refers to a person who uses the system and is an individual or employee who is subject to security authentication.

[0212] "Emotional responses" are information that indicates a user's mental state, and collected data includes facial expressions and vocal tone.

[0213] "Facial expressions" are signs of emotion that appear on a user's face and are part of the data collected using a facial recognition camera.

[0214] "Heart rate" refers to the number of times the heart beats within a certain period of time and is part of the biometric data collected using biometric sensors.

[0215] "Tone of voice" refers to the quality and pitch of the user's voice and is part of the data collected using a microphone.

[0216] "Behavioral patterns" are part of the data collected by sensors and indicate characteristics of a user's movements and behavior.

[0217] "Biometric data" refers to information that indicates a user's physical characteristics, and refers to a wide range of data including heart rate, tone of voice, etc.

[0218] "Encryption" refers to the process of transforming collected data using specific algorithms to protect it.

[0219] "Server" refers to a central computer system that receives collected data, analyzes it, and manages authentication results.

[0220] "Analysis" refers to the process of synthesizing collected data and analyzing the information to authenticate a user.

[0221] "Authentication" refers to the process of verifying a user's identity and granting access rights based on collected data.

[0222] "Access permission" refers to allowing a user who has been successfully authenticated to use a specific system or service.

[0223] A "VR headset" is a device for experiencing virtual reality and refers to a terminal used to collect user data.

[0224] "Secure communications protocol" refers to a communications protocol used to encrypt and protect data transmissions.

[0225] The present invention relates to a method and system for enhancing a company's security authentication system by using a VR headset worn by a user to collect, analyze, and authenticate biometric and behavioral data to enhance security authentication.

[0226] System configuration

[0227] Hardware and Software Configuration

[0228] 1. VR headset (device)

[0229] Equipment: VR headsets are equipped with sensors to collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data.

[0230] Sensors: Facial recognition camera, heart rate sensor, microphone.

[0231] 2. Server

[0232] Function: Receives collected data, decrypts encrypted data, and has the processing power to perform analysis.

[0233] Software: Multimodal generative AI (e.g., OpenAI® GPT-4®), facial recognition algorithms (e.g., OpenCV), speech recognition algorithms (e.g., Google® Cloud Speech-to-Text), and biometric data analysis algorithms.

[0234] 3. Communication Protocol

[0235] Protocol: A secure communication protocol (e.g. HTTPS) is used to send and receive data.

[0236] Program processing flow

[0237] 1. Collection of User Data

[0238] When a user puts on a VR headset to access a company's authentication system, sensors are activated to collect emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, which are then temporarily stored in memory.

[0239] 2. Data Encryption and Transmission

[0240] The collected data is encrypted using the AES-256 encryption algorithm and transmitted to the server using the HTTPS protocol.

[0241] 3. Data Receipt and Analysis

[0242] The server decrypts the received data and analyzes it in the following way:

[0243] Emotional responses and facial expressions are analyzed using a facial recognition algorithm (OpenCV).

[0244] The tone of voice is analyzed using a voice recognition algorithm (Google Cloud Speech-to-Text).

[0245] Heart rate is analyzed using biometric data analysis algorithms.

[0246] Behavioral patterns are integrated and analyzed using a multimodal generative AI model.

[0247] 4. User Authentication

[0248] Based on the parsed data, the server authenticates the user by matching the user profile with existing data in a database.

[0249] 5. Notification of authentication results and access permission

[0250] The server notifies the terminal of the authentication result, and if the authentication is successful, the user is granted access permission, ensuring the prevention of unauthorized access and the management of users.

[0251] Examples and prompts

[0252] Examples:

[0253] When the installed VR headset is used, it attempts to access the company's security authentication system.

[0254] For example: "Put on the VR headset to access the company's authentication system. Once you put it on, the sensors will automatically activate and collect data."

[0255] Example prompt sentence:

[0256] "Please put on your Oculus Quest and log in to the corporate authentication app. After logging in to the app, sensors will be activated to collect your emotional response, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. Based on this, you will be authenticated, so please just wait."

[0257] This process enables companies to achieve highly accurate and reliable security authentication, reducing the risk of unauthorized access and data leakage.

[0258] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0259] Step 1:

[0260] A user puts on a VR headset and accesses a company's authentication system.

[0261] Input: A user puts on a VR headset and attempts to log into the authentication system.

[0262] Action: A user launches a VR app and visits the login page.

[0263] Step 2:

[0264] The device activates sensors and collects user data.

[0265] Input: The sensors activated by the VR headset (face recognition camera, heart rate sensor, microphone).

[0266] Data processing: The device collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data and stores them in temporary memory.

[0267] How it works: A facial recognition camera captures your facial expressions, a heart rate sensor measures your heart rate, and a microphone records your tone of voice.

[0268] Step 3:

[0269] The terminal encrypts the collected user data and sends it to the server.

[0270] Input: Collected user data (facial expression data, heart rate data, voice data).

[0271] Data operation: The device encrypts the data using the AES-256 encryption algorithm.

[0272] Output: Encrypted user data.

[0273] How it works: The device sends encrypted data to the server using the HTTPS protocol.

[0274] Step 4:

[0275] The server receives the encrypted data and decrypts it.

[0276] Input: Encrypted user data from the device.

[0277] Data Calculation: The server decrypts the data using the AES-256 algorithm.

[0278] Output: Decrypted user data.

[0279] How it works: The server decrypts the data it receives using the HTTPS protocol.

[0280] Step 5:

[0281] The server analyzes and integrates various data.

[0282] Input: Decoded user data (facial expression data, heart rate data, voice data).

[0283] Data processing: The server analyzes the individual data using facial recognition algorithms (e.g., OpenCV), voice recognition algorithms (e.g., Google Cloud Speech-to-Text), and biometric data analysis algorithms, and then synthesizes the data using a generative AI model.

[0284] Output: A comprehensive user profile.

[0285] How it works: The server analyzes emotional responses and facial expressions using facial recognition algorithms, tone of voice using speech recognition algorithms, and heart rate using biometric data analysis algorithms.

[0286] Step 6:

[0287] The server authenticates the user based on their profile.

[0288] Input: A pooled and analyzed user profile.

[0289] Data calculation: The server checks the data against the user database and performs authentication.

[0290] Output: Authentication result (success / failure).

[0291] How it works: The server checks the generated user profile against its existing user database to determine if there is a match.

[0292] Step 7:

[0293] The server notifies the terminal of the authentication result.

[0294] Input: Authentication result (success / failure).

[0295] Output: A signal notifying the authentication result.

[0296] Operation: The server sends the authentication result to the terminal.

[0297] Step 8:

[0298] The terminal displays the authentication result to the user and provides access permission.

[0299] Input: Authentication result from the server.

[0300] Output: Notification to user (access permission / warning message).

[0301] Behavior: If the device is successfully authenticated, it displays the access permission and allows you to log in to the metaverse. If authentication fails, it displays a warning message.

[0302] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0303] Explaining program processing in natural language

[0304] Collection and transmission of user data

[0305] When a user puts on a VR headset and attempts to access the Metaverse, the device first collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. This collection is done using a facial recognition camera, heart rate sensor, microphone, and motion sensors. The device also uses an emotion engine to recognize and collect the user's emotional data in real time. This data is then stored in temporary memory.

[0306] Data encryption and transmission

[0307] The device encrypts the collected data using an encryption algorithm (e.g., AES-256), and the encrypted data is sent to the server using a secure communication protocol (e.g., HTTPS).

[0308] Data reception and analysis

[0309] The server receives and decrypts the encrypted data sent from the device. The decrypted data is categorized into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data, including emotional data obtained from the emotion engine. This data is then integrated using multimodal generative AI to generate a comprehensive user profile.

[0310] Analysis of individual data

[0311] The server analyzes the individual data in the following way:

[0312] Analyze the emotion recognition data obtained by the emotion engine.

[0313] Facial expression data is analyzed using a facial recognition algorithm.

[0314] A voice recognition algorithm is used to analyze tone of voice data.

[0315] Heart rate data is analyzed using a heart rate analysis algorithm.

[0316] Behavioral pattern data is analyzed using a behavioral prediction algorithm.

[0317] User Authentication

[0318] The server compares the generated user profile with existing user data in its internal database. Based on the comparison result, it determines whether the user is legitimate. If the user is legitimate, authentication is successful and a notification of this is sent to the terminal.

[0319] Notification of access permission to users

[0320] The terminal receives the authentication result from the server and, if successful, notifies the user of the access permission, allowing the user to log in to the metaverse. If the authentication fails, a corresponding warning message is displayed.

[0321] Specific examples

[0322] User logging into the metaverse scenario

[0323] 1. A user puts on a VR headset and attempts to access the Metaverse.

[0324] The user enters their ID and password and confirms the login screen.

[0325] 2. The device activates sensors and collects user data.

[0326] The device activates a facial recognition camera, heart rate sensor, microphone, emotion engine, and motion sensors to collect the user's facial expressions, heart rate, tone of voice, emotional data, and behavioral patterns.

[0327] 3. The device encrypts the data and sends it to the server.

[0328] The terminal encrypts the data with AES-256 and sends it to the server via the HTTPS protocol.

[0329] 4. The server receives and decrypts the data.

[0330] The server receives the data and decrypts it with AES-256.

[0331] 5. The server consolidates and analyzes the data.

[0332] The server performs emotion data recognized by the emotion engine, face recognition, voice recognition, heart rate analysis, and movement analysis to obtain the integrated results.

[0333] 6. The server authenticates the user against its user database.

[0334] The server checks its user database for a matching record.

[0335] 7. The server notifies the terminal of the authentication result.

[0336] The server notifies the terminal of successful authentication.

[0337] 8. The device notifies the user of the access permission.

[0338] The device displays the access permissions to the user and logs them into the metaverse.

[0339] In this way, the present invention can accurately and securely identify users and increase safety and engagement within the metaverse. The introduction of an emotion engine can more accurately grasp the user's psychological state, further strengthening security.

[0340] The processing flow will be explained below.

[0341] Step 1:

[0342] A user puts on a VR headset and attempts to access the Metaverse.

[0343] The user enters their ID and password on the login screen within the metaverse.

[0344] Step 2:

[0345] The device activates sensors and collects user data.

[0346] The device uses a facial recognition camera to collect facial expression data from the user.

[0347] The device uses a heart rate sensor to collect the user's heart rate data.

[0348] The terminal uses a microphone to collect data on the tone of the user's voice.

[0349] The device uses motion sensors to collect data on the user's behavioral patterns.

[0350] The terminal uses an emotion engine to collect the user's emotional response data.

[0351] Step 3:

[0352] The device encrypts the collected data.

[0353] The device uses the AES-256 encryption algorithm to encrypt collected facial expression data, heart rate data, tone of voice data, behavioral pattern data, and emotional response data.

[0354] Step 4:

[0355] The device sends the encrypted data to the server.

[0356] The device uses the HTTPS protocol to securely send encrypted data to the server.

[0357] Step 5:

[0358] The server decrypts the received data.

[0359] The server decrypts the received encrypted data using the AES-256 encryption algorithm.

[0360] Step 6:

[0361] The server classifies the decrypted data.

[0362] The server categorizes the decoded data into facial expression data, heart rate data, tone of voice data, behavioral pattern data, and emotional response data.

[0363] Step 7:

[0364] The server consolidates the data.

[0365] The server uses multimodal generative AI to integrate the categorized data and generate a comprehensive profile of the user.

[0366] Step 8:

[0367] The server analyzes the individual data.

[0368] The server analyzes the facial expression data using a facial recognition algorithm.

[0369] The server analyzes the tone of voice data using a voice recognition algorithm.

[0370] The server analyzes the heart rate data using a heart rate analysis algorithm.

[0371] The server analyzes the behavioral pattern data using a behavioral prediction algorithm.

[0372] The server uses an emotion engine to analyze the emotional response data.

[0373] Step 9:

[0374] The server authenticates the user based on the analysis results.

[0375] The server matches the generated user profile with existing data in its internal database.

[0376] Based on the result of the comparison, the server determines whether the user is legitimate.

[0377] Step 10:

[0378] The server notifies the terminal of the authentication result.

[0379] If the authentication is successful, the server notifies the terminal of access permission.

[0380] If the authentication fails, the server notifies the terminal of access denial.

[0381] Step 11:

[0382] The terminal notifies the user of the authentication result.

[0383] If the authentication is successful, the terminal allows the user to access the metaverse.

[0384] If the authentication fails, the terminal displays a warning message to the user.

[0385] In this way, the system combined with the emotion engine integrates and analyzes users' emotions, behaviors, and biometric data to achieve accurate and secure authentication, which can significantly improve safety and user engagement within the metaverse.

[0386] Example 2

[0387] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0388] In conventional systems, the process of collecting and authenticating users' personal information is complicated, and security is often insufficient. In particular, there is a lack of means to integrate and analyze multimodal data such as emotional and behavioral data, which leads to problems with the accuracy of user authentication. This leads to a poor user experience in metaverse and virtual reality environments, and safety is not guaranteed.

[0389] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0390] In this invention, the server includes means for collecting a user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, means for storing the collected data in a temporary storage device, means for encrypting the collected data using the AES-256 algorithm, means for transmitting the encrypted data to the server using a secure communication protocol, means for decrypting the received data using the AES-256 algorithm, means for classifying the decrypted data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral patterns, and biometric data and integrating them using multimodal generation AI, means for analyzing the integrated data, means for authenticating the user based on the analysis result, means for notifying the terminal of the authentication result, and means for providing access permission to the user based on the authentication result. This allows the process from collecting user data to granting access permission to be performed safely and efficiently, thereby improving user safety and engagement in the metaverse.

[0391] A "user" is a person who uses a client device to access the system and use the services.

[0392] "Emotional response" is data that indicates the user's psychological state, and is information that is analyzed in real time using an emotion engine.

[0393] "Facial expression" is data that indicates the state of the user's face, and is information collected by a face recognition camera.

[0394] "Heart rate" is data indicating the frequency of the user's heartbeat, and is information measured using a heart rate sensor.

[0395] "Voice tone" is data that indicates the quality and pitch of a user's voice when speaking, and is information collected using a microphone.

[0396] "Behavior patterns" are data that indicate the user's movements and behavioral tendencies, and are information captured using a movement sensor.

[0397] "Biometric data" refers to data related to the user's physical characteristics, including information such as heart rate and facial recognition data.

[0398] A "temporary storage device" is a storage device for temporarily storing collected data.

[0399] "AES-256 Algorithm" means an advanced encryption algorithm and technology used to encrypt data.

[0400] A "secure communication protocol" is a communication protocol for ensuring the safe transfer of data, such as HTTPS.

[0401] A "server" is a computer system that receives, decrypts, analyzes collected data, and performs user authentication.

[0402] "Emotion recognition data" is data relating to the user's emotional state, analyzed by the emotion engine.

[0403] "Facial expression data" is data relating to the state of the user's face, and is collected by a face recognition camera.

[0404] "Heart rate data" is data relating to the user's heart rate, and is acquired by a heart rate sensor.

[0405] "Voice tone data" is data relating to the quality and pitch of the user's voice, and is collected by a microphone.

[0406] "Behavior pattern data" is data relating to the user's movements and behaviors, and is captured by a movement sensor.

[0407] "Multimodal generative AI" is an artificial intelligence model for integrating and analyzing multiple different types of data.

[0408] "User authentication" is the process of verifying that a user is a legitimate user by comparing collected data with existing database information.

[0409] "Access permission" is a control that allows a user who has been successfully authenticated to access a system or service.

[0410] This invention relates to a system that safely and efficiently collects and analyzes personal data and performs accurate authentication when a user wears a VR headset and accesses the metaverse.

[0411] Hardware and software used

[0412] When a user puts on a VR headset, access to the Metaverse begins. At this time, the following hardware on the device collects data:

[0413] Facial recognition camera: Captures the user's facial expressions.

[0414] Heart rate sensor: Measures the user's heart rate.

[0415] Microphone: Collects the user's tone of voice.

[0416] Motion sensors: Capture user behavior patterns.

[0417] Emotion Engine: Analyzes users' emotional responses in real time.

[0418] This data is temporarily stored in the device's temporary storage, after which the device encrypts the data using the AES-256 algorithm and transmits it to the server via the secure HTTPS communication protocol.

[0419] The server does the following:

[0420] Decrypting received data: Decrypts encrypted data using the AES-256 algorithm.

[0421] Data Classification and Integration: Classify the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data using multimodal generative AI.

[0422] Data analysis: The server analyzes the data using various algorithms. Specifically, an emotion engine is used to analyze emotion recognition data, a face recognition algorithm is used to analyze facial expression data, and a voice recognition algorithm is used to analyze tone of voice data.

[0423] Perform user authentication: Based on the integrated data, the user is authenticated. This process involves checking against an existing user database.

[0424] Notification of authentication result: The server notifies the terminal of the authentication result.

[0425] Specific examples

[0426] For example, the following specific operations are performed.

[0427] 1. A user puts on a VR headset and attempts to access the Metaverse.

[0428] The user enters their ID and password and confirms the login screen.

[0429] 2. The device activates sensors and collects user data.

[0430] The device activates a facial recognition camera, heart rate sensor, microphone, emotion engine, and motion sensors to collect the user's facial expressions, heart rate, tone of voice, emotional data, and behavioral patterns.

[0431] 3. The device encrypts the data and sends it to the server.

[0432] The device encrypts the collected data with AES-256 and sends it to the server via the HTTPS protocol.

[0433] 4. The server receives and decrypts the data.

[0434] The server receives the data and decrypts it with AES-256.

[0435] 5. The server consolidates and analyzes the data.

[0436] The server performs emotion data recognized by the emotion engine, facial recognition, voice recognition, heart rate analysis, and movement analysis to obtain the integrated results.

[0437] 6. The server authenticates the user against its user database.

[0438] The server checks its user database for a matching record.

[0439] 7. The server notifies the terminal of the authentication result.

[0440] The server notifies the terminal of successful authentication.

[0441] 8. The device notifies the user of the access permission.

[0442] The device displays the access permissions to the user and logs them into the metaverse.

[0443] This system provides a secure and efficient authentication process, from collecting user data to granting access to the metaverse. An example prompt is as follows:

[0444] "Please describe in natural language a scenario in which a user logs into the Metaverse using a VR headset. Please specifically describe the process from user data collection to authentication and access authorization."

[0445] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0446] Step 1:

[0447] A user puts on a VR headset and attempts to access the Metaverse.

[0448] Specifically, this includes the user entering their ID and password to confirm the login screen, which generates an access request and triggers the device to start the authentication process.

[0449] Input: User login information (ID, password)

[0450] Output: Trigger to start the authentication process

[0451] Step 2:

[0452] The device activates sensors and collects user data.

[0453] Specifically, the device's built-in facial recognition camera, heart rate sensor, microphone, motion sensor, and emotion engine are all activated to collect the user's facial expressions, heart rate, tone of voice, behavioral patterns, and emotional data in real time.

[0454] Input: User's biometric information (face, heart rate, voice, movement, emotion)

[0455] Output: Collected bio-behavioral data

[0456] Step 3:

[0457] The terminal stores the collected data in temporary storage.

[0458] Specifically, this involves caching the collected data in temporary storage.

[0459] Input: Collected biometric and behavioral data

[0460] Output: Data stored in temporary storage

[0461] Step 4:

[0462] The device encrypts the collected data using the AES-256 algorithm.

[0463] Specifically, this involves automatically passing data stored in temporary storage to an encryption module where it is encrypted using AES-256.

[0464] Input: Data stored in temporary storage

[0465] Output: Encrypted data

[0466] Step 5:

[0467] The device sends encrypted data to the server using the secure HTTPS protocol.

[0468] Specifically, this includes operations in which encrypted data is transmitted using the HTTPS protocol.

[0469] Input: Encrypted data

[0470] Output: Send data to the server

[0471] Step 6:

[0472] The server receives the encrypted data and decrypts it using the AES-256 algorithm.

[0473] Specifically, the encrypted data is passed to the receiving module and decrypted using AES-256.

[0474] Input: Encrypted data

[0475] Output: Decrypted data

[0476] Step 7:

[0477] The server classifies the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data, and integrates them using multimodal generative AI.

[0478] Specifically, the decoded data is classified into categories, sent to the respective analysis modules, and an integration process is carried out by the multimodal generation AI.

[0479] Input: Decrypted data

[0480] Output: Unified user profile

[0481] Step 8:

[0482] The server analyzes the combined data and authenticates the user.

[0483] Specifically, this involves checking the integrated user profile against an internal database to determine legitimacy.

[0484] Input: Unified User Profile

[0485] Output: Authentication result

[0486] Step 9:

[0487] The server notifies the terminal of the authentication result.

[0488] Specifically, this includes an operation of transmitting the result of authentication success or failure to the terminal.

[0489] Input: Authentication result

[0490] Output: Authentication result notification to the terminal

[0491] Step 10:

[0492] The terminal notifies the user of the access permission.

[0493] Specifically, if authentication is successful, the terminal will display the access permission to the metaverse to the user, and if authentication fails, a corresponding warning message will be displayed.

[0494] Input: Authentication result notification to the terminal

[0495] Output: Allow access to user or warning message

[0496] (Application example 2)

[0497] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0498] Modern authentication systems have adopted a wide range of biometric and behavioral authentication methods for security reasons. However, because these methods rely on individual data, it is difficult to completely eliminate the risk of impersonation and unauthorized access. In particular, when using emotional or biometric data alone, they are susceptible to temporary changes in the user's state and environmental factors, making it difficult to obtain consistent authentication results. Furthermore, there are insufficient means to integrate these data and perform accurate and secure authentication based on a comprehensive user profile.

[0499] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, means for determining the legitimacy of the user based on a user profile generated using the collected biometric data and emotion recognition data, and means for providing access permission to the user based on the authentication result. This provides a consistent authentication result that integrates multiple data, eliminates impersonation and unauthorized access, and enables more secure and reliable authentication.

[0500] Below are definitions of important terms extracted from claim 1:

[0501] "User data" is a general term for information about a user, such as emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data.

[0502] "Encryption" refers to the process of transforming data using specific algorithms to make it unreadable to third parties.

[0503] A "server" is a computer system that stores, analyzes, and processes data over a network.

[0504] "Integration" refers to the act of bringing together multiple types of data and processing them in a consistent format or with consistent standards.

[0505] "Analysis" is the process of examining collected data in detail to find meaning and patterns.

[0506] "Authentication" is the process of verifying that a user has valid access rights.

[0507] A "terminal" is an electronic device that is directly operated by a user, and includes smartphones, personal computers, etc.

[0508] "Access permission" refers to granting access to a system or data.

[0509] "Biometric data" refers to information obtained from the user's body, such as heart rate, fingerprints, and facial shape.

[0510] "Emotion recognition data" is data used to analyze a user's mental state and emotions.

[0511] A "user profile" is a consistent set of information about a user that is obtained by integrating various collected data.

[0512] The present invention provides an advanced authentication system by collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, and integrating and analyzing this data. Specific embodiments for carrying out the present invention will be described below.

[0513] System Configuration

[0514] The system consists of the following main components:

[0515] Terminal: A device that is directly used by a user, such as a smartphone, smart glasses, a head-mounted display, or a robot.

[0516] Sensors: Face recognition camera, heart rate sensor, microphone, and motion sensor installed on the device.

[0517] Server: A computer system that stores, analyzes, and integrates data.

[0518] Communication Protocol: A secure communication protocol (e.g. HTTPS) for sending and receiving data.

[0519] Encryption algorithm: The encryption method used to ensure secure transmission of data (e.g., AES-256).

[0520] Emotion Engine: An API for analyzing emotions from collected data.

[0521] Program processing overview

[0522] When a user attempts to log in using a device, the device first collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using a facial recognition camera, heart rate sensor, microphone, and motion sensors. Using an emotion engine, the device analyzes the collected data in real time to generate emotional data. This data is then stored in temporary memory.

[0523] Next, the device encrypts the collected data with an encryption algorithm (e.g., AES-256) and sends it to the server using a secure communication protocol (e.g., HTTPS). The server receives and decrypts the encrypted data.

[0524] The server analyzes the decoded data using an emotion engine, face recognition algorithm, voice recognition algorithm, heart rate analysis algorithm, and behavior prediction algorithm to classify it into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavior pattern data, and biometric data. A comprehensive user profile is generated from the integrated data.

[0525] Based on the generated user profile, the server compares it with existing user data in its internal database to determine the user's legitimacy. Based on the comparison result, the server notifies the terminal of the authentication result, and the terminal displays the result to the user. If authentication is successful, the user is granted access permission.

[0526] Specific examples of hardware and software used

[0527] Device: Compatible smartphones and head-mounted displays.

[0528] Emotion engine: For example, Affectiva or Microsoft® Azure® Face API.

[0529] Encryption algorithm: AES-256.

[0530] Communication protocol: HTTPS.

[0531] Analysis algorithms: Multimodal generative AI, OpenCV (face recognition), Google Cloud Speech-to-Text (voice recognition).

[0532] Specific examples

[0533] When a user launches the security app, the app uses the smartphone's camera to capture the user's facial expressions, the heart rate sensor to obtain their heart rate, the microphone to record audio and analyze their tone of voice, and the motion sensor to record their behavioral patterns. This data is then analyzed in real time as emotional data using an emotion engine. The data is then encrypted with AES-256 and sent to a secure server via HTTPS. The server receives, decrypts, and analyzes the data to generate a comprehensive user profile. The server then authenticates the user and notifies the smartphone of the authentication result.

[0534] Prompt Sentence Examples

[0535] Here are some example prompts to input to a generative AI model:

[0536] Describe a system that collects a user's facial recognition, heart rate, tone of voice, behavioral patterns, and biometric data, and then uses an emotion engine to analyze the emotional data in real time. This data is integrated to generate a comprehensive user profile and perform user authentication.

[0537] In this way, the present invention integrates data from multiple data sources and provides a high level of security, thereby improving the reliability and accuracy of authentication.

[0538] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0539] Step 1: Collect user data

[0540] When a user attempts to log in using a device, the device activates various sensors, including a facial recognition camera, heart rate sensor, microphone, and motion sensor, which are used to collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. The input is the user's biometric data, and the output is a set of collected biometric data.

[0541] Step 2: Analyze the emotion data

[0542] The collected biometric data is analyzed in real time into emotional data using an emotion engine, where each data point is mapped to an emotional state. The input is the collected biometric data, and the output is the analyzed emotional data.

[0543] Step 3: Temporarily save data

[0544] The terminal stores the collected data and analyzed emotional data in a temporary memory. The input of this process is the analyzed emotional data and biometric data, and the output is the set of data in the temporary memory.

[0545] Step 4: Encrypt the data

[0546] The terminal encrypts the data in temporary memory using an encryption algorithm (e.g., AES-256). The input is the data stored in temporary memory, and the output is the encrypted data.

[0547] Step 5: Sending data

[0548] The encrypted data is sent to the server using a secure communication protocol (e.g., HTTPS). The input is the encrypted data and the output is the data sent to the server.

[0549] Step 6: Receiving and Decrypting Data

[0550] The server receives the encrypted data and decrypts it with a decryption algorithm (e.g., AES-256). The input is the encrypted data and the output is the decrypted data.

[0551] Step 7: Classify the data

[0552] The server classifies the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data. The input to this process is the decoded data, and the output is the various classified data.

[0553] Step 8: Analyze the data

[0554] The various classified data are analyzed using an emotion engine, face recognition algorithm, voice recognition algorithm, heart rate analysis algorithm, and behavior prediction algorithm. The input is the classified data, and the output is the analysis results.

[0555] Step 9: Generate an Integration Profile

[0556] The server integrates the analysis results and generates a comprehensive user profile. The input is the analyzed data, and the output is the integrated user profile.

[0557] Step 10: User Authentication

[0558] The server compares the generated user profile with existing user data in its internal database to determine its validity. The input is the integrated user profile, and the output is the authentication result.

[0559] Step 11: Notification of authentication results

[0560] The server sends the authentication result to the terminal, and the terminal notifies the user of the result. The input is the authentication result, and the output is the notification to the user.

[0561] Step 12: Access Permissions

[0562] If the authentication is successful, the terminal grants the user access permission. The input is the authentication result, and the output is the granting of access permission.

[0563] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0564] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0565] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0566] [Second embodiment]

[0567] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0568] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0569] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0570] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0571] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0572] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0573] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0574] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0575] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0576] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0577] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0578] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0579] Explaining program processing in natural language

[0580] Collection and transmission of user data

[0581] When a user puts on a VR headset and attempts to access the Metaverse, the device collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using facial recognition cameras, heart rate sensors, microphones, etc. The collected data is then stored in temporary memory by the device.

[0582] Data encryption and transmission

[0583] The device encrypts the collected data using an encryption algorithm (e.g., AES-256), and then transmits the encrypted data to the server using a secure communication protocol (e.g., HTTPS).

[0584] Data reception and analysis

[0585] The server receives and decrypts the encrypted data sent from the device. The decrypted data is separated into different categories (emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data). This data is then integrated using multimodal generative AI to generate a comprehensive user profile.

[0586] Analysis of individual data

[0587] The server analyzes the individual data in the following way:

[0588] Emotional responses and facial expressions are run through a facial recognition algorithm.

[0589] The tone of voice is analyzed using a voice recognition algorithm.

[0590] The heart rate is analyzed by the biosensor's data analysis algorithm.

[0591] The behavioral patterns are analyzed based on a motion prediction algorithm.

[0592] User Authentication

[0593] The server compares the generated user profile with existing user data in its internal database. Based on the comparison result, it determines whether the user is legitimate. If the user is legitimate, authentication is successful and a notification of this is sent to the terminal.

[0594] Notification of access permission to users

[0595] The terminal receives the authentication result from the server and, if successful, notifies the user of the access permission, allowing the user to log in to the metaverse. If the authentication fails, a corresponding warning message is displayed.

[0596] Specific examples

[0597] User logging into the metaverse scenario

[0598] 1. A user puts on a VR headset and attempts to access the Metaverse.

[0599] The user enters their ID and password and confirms the login screen.

[0600] 2. The device activates sensors and collects user data.

[0601] The device will enable a facial recognition camera, heart rate sensor, and microphone to collect the user's facial expressions, heart rate, and voice data.

[0602] 3. The device encrypts the data and sends it to the server.

[0603] The terminal encrypts the data with AES-256 and sends it to the server via the HTTPS protocol.

[0604] 4. The server receives and decrypts the data.

[0605] The server receives the data and decrypts it with AES-256.

[0606] 5. The server consolidates and analyzes the data.

[0607] The server performs facial recognition, voice recognition, and heart rate analysis to obtain an integrated result.

[0608] 6. The server authenticates the user against its user database.

[0609] The server checks its user database for a matching record.

[0610] 7. The server notifies the terminal of the authentication result.

[0611] The server notifies the terminal of successful authentication.

[0612] 8. The device notifies the user of the access permission.

[0613] The device displays the access permissions to the user and logs them into the metaverse.

[0614] In this way, the present invention can accurately and securely identify users, increasing safety and engagement within the metaverse.

[0615] The processing flow will be explained below.

[0616] Step 1:

[0617] A user puts on a VR headset and attempts to access the Metaverse.

[0618] The user enters their ID and password and proceeds to the login screen.

[0619] Step 2:

[0620] The device activates sensors and collects user data.

[0621] The device uses a facial recognition camera to collect facial expression data from the user.

[0622] The device uses a heart rate sensor to collect the user's heart rate data.

[0623] The terminal uses a microphone to collect data on the tone of the user's voice.

[0624] The device uses motion sensors to collect data on the user's behavioral patterns.

[0625] Step 3:

[0626] The device encrypts the collected data.

[0627] The device uses the AES-256 encryption algorithm to encrypt collected facial expression data, heart rate data, tone of voice data, and behavioral pattern data.

[0628] Step 4:

[0629] The device sends the encrypted data to the server.

[0630] The device uses the HTTPS protocol to securely send encrypted data to the server.

[0631] Step 5:

[0632] The server decrypts the received data.

[0633] The server decrypts the received encrypted data using the AES-256 encryption algorithm.

[0634] Step 6:

[0635] The server consolidates the decrypted data.

[0636] The server integrates facial recognition data, heart rate data, tone of voice data, and behavioral pattern data using multimodal generation AI to generate a user profile.

[0637] Step 7:

[0638] The server analyzes the individual data.

[0639] The server analyzes the facial expression data using a facial recognition algorithm.

[0640] The server analyzes the tone of voice data using a voice recognition algorithm.

[0641] The server analyzes the heart rate data using a heart rate analysis algorithm.

[0642] The server analyzes the behavioral pattern data using a behavioral prediction algorithm.

[0643] Step 8:

[0644] The server authenticates the user based on the analysis results.

[0645] The server matches the generated user profile with existing data in its internal database.

[0646] Based on the result of the comparison, the server determines whether the user is legitimate.

[0647] Step 9:

[0648] The server notifies the terminal of the authentication result.

[0649] If the authentication is successful, the server notifies the terminal of access permission.

[0650] If the authentication fails, the server notifies the terminal of access denial.

[0651] Step 10:

[0652] The terminal notifies the user of the authentication result.

[0653] If the authentication is successful, the terminal allows the user to access the metaverse.

[0654] If the authentication fails, the terminal displays a warning message to the user.

[0655] Example 1

[0656] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0657] Conventional virtual reality (VR) systems have faced problems with insufficiently secure user authentication methods, resulting in a high risk of easy unauthorized access. Furthermore, there was a lack of a means for centralized and detailed analysis of individual user data, making it difficult to improve the quality of the user experience. Furthermore, there was no established secure method for transferring collected data, creating a risk of data leakage. Therefore, providing a highly accurate and secure authentication system that users could use with confidence was a challenge.

[0658] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0659] In this invention, the server includes means for collecting emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using a facial recognition camera, heart rate sensor, and microphone when a user wears a VR headset and attempts to access the metaverse, means for encrypting the collected data using the AES-256 algorithm and transmitting it to the server via HTTPS protocol, and server means for decrypting the received data and integrating and analyzing the emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using multimodal generation AI, thereby enabling secure transfer of the collected data and highly accurate user authentication.

[0660] "User" refers to an individual who wears a VR headset and accesses the Metaverse.

[0661] "Device" refers to the device including the VR headset worn by the user and its related equipment.

[0662] "Server" refers to a computer on the system side that receives data sent from a user and analyzes and authenticates it.

[0663] A "face recognition camera" refers to a photographic device that captures a user's facial expressions and features.

[0664] "Heart rate sensor" refers to a detection device for measuring a user's heart rate.

[0665] "Microphone" refers to a recording device for collecting the user's voice.

[0666] "Emotional response" refers to changes in the emotions shown by the user, and is determined from facial expressions, tone of voice, etc.

[0667] "Facial expressions" refer to movements that show emotions and reactions on the user's face.

[0668] "Heart rate" refers to the number of times a user's heart beats within a certain period of time.

[0669] "Tone of voice" refers to characteristics such as the pitch and intonation of a user's voice.

[0670] "Behavioral patterns" refer to a series of actions and tendencies of a user's reactions.

[0671] "Biometric data" refers to various information collected from a user's body (e.g., heart rate, pulse, body temperature).

[0672] The "AES-256 algorithm" is one of the Advanced Encryption Standards for encrypting data, and uses a 256-bit key.

[0673] "HTTPS protocol" refers to a communications protocol for securely transferring data over the Internet.

[0674] "Multimodal generative AI" refers to an artificial intelligence model that integrates and analyzes multiple data modalities (e.g., audio, images, text).

[0675] "Encryption" refers to the process of transforming data so that it cannot be read by third parties.

[0676] "Decryption" refers to the process of restoring encrypted data to its original state.

[0677] "User profile" refers to a collection of information about an individual user that is generated by integrating various collected data.

[0678] "Authentication" refers to the process of verifying that a user is a specific individual.

[0679] "Access permission" refers to granting a user the right to access a particular system or information.

[0680] "Warning message" refers to the notification displayed to the user when authentication fails.

[0681] This invention is a system for accurately and securely authenticating users when they wear a VR headset and access the Metaverse. This system works through several steps, mainly involving data collection, encryption, transmission, reception, analysis, and authentication processes between the device, server, and user.

[0682] User Data Collection

[0683] When a user puts on a VR headset and starts an application, the device first activates sensors, including a facial recognition camera, heart rate sensor, and microphone. These devices collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. For example, the facial recognition camera captures facial expression data, the heart rate sensor monitors heart rate, and the microphone records tone of voice.

[0684] Data encryption and transmission

[0685] The collected data is temporarily stored on the device and then encrypted using the AES-256 encryption algorithm, and then transmitted to the server using the HTTPS protocol. This process ensures the security of the data.

[0686] Data analysis on the server

[0687] Once the data arrives at the server, it is first decoded. After decoding, the data is separated into categories (emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data). The server then uses multimodal generative AI to integrate and analyze this data. This results in a comprehensive user profile, which includes, for example, current emotional state, heart rate trends, and usual behavioral patterns.

[0688] User Authentication

[0689] The generated user profile is compared with existing user data stored in the server's internal database. This comparison determines whether the user has valid access rights. If authentication is successful, the server notifies the terminal of the result. If authentication fails, the terminal displays a warning message to the user.

[0690] User notification and access permissions

[0691] The terminal receives the authentication result from the server and notifies the user of the access permission if successful, allowing the user to log in to the metaverse and access interactive content. If the authentication fails, a corresponding warning message will be displayed to the user, prompting them to try again.

[0692] The system will improve the user experience and ensure safety within the metaverse by collecting, analyzing, and authenticating user data with high accuracy and security.

[0693] Prompt Sentence Examples

[0694] Finally, the following are some specific examples of prompts for generative AI models:

[0695] "Please explain in natural language the system that collects biometric data from users using a VR headset and analyzes it to authenticate the user."

[0696] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0697] Step 1:

[0698] The user puts on a VR headset and prepares to access the Metaverse.

[0699] The user turns on the VR headset, launches the application, and proceeds to the login screen where they enter their ID and password.

[0700] Input: User ID and password

[0701] Output: Access to login screen

[0702] Step 2:

[0703] The device activates sensors and collects user data.

[0704] The device activates a facial recognition camera, heart rate sensor and microphone to simultaneously collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns and biometric data.

[0705] Input: User's current biometric and behavioral data

[0706] Data processing / data calculation: Analyzes images taken with a facial recognition camera and extracts facial expression data. Records heart rate measured with a heart rate sensor. Analyzes audio data collected with a microphone and identifies voice tone.

[0707] Output: Multiple types of biometric data collected

[0708] Step 3:

[0709] Encrypt the data collected by the device.

[0710] The device encrypts all collected data using the AES-256 encryption algorithm.

[0711] Input: Unencrypted biometric data

[0712] Data processing / data calculation: Encrypt data using the AES-256 algorithm

[0713] Output: Encrypted biometric data

[0714] Step 4:

[0715] The encrypted data is sent to the server.

[0716] The device uses the HTTPS protocol to securely transmit encrypted data to the server.

[0717] Input: Encrypted biometric data

[0718] Output: Send data to the server

[0719] Step 5:

[0720] The server receives and decrypts the data.

[0721] The server receives the data using the HTTPS protocol and decrypts it using the AES-256 algorithm.

[0722] Input: Encrypted biometric data

[0723] Data processing / data calculation: Receive data using HTTPS protocol and decrypt it using AES-256 algorithm

[0724] Output: Decoded biometric data

[0725] Step 6:

[0726] The server categorizes and analyzes the data.

[0727] The server separates the decoded data into emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, and then uses multimodal generative AI to integrate and analyze this data.

[0728] Input: Decoded biometric data

[0729] Data processing / data calculation: Categorize data and perform integrated analysis using multimodal generation AI

[0730] Output: Comprehensive user profile

[0731] Step 7:

[0732] The server checks the user profile against an internal database.

[0733] The server compares the generated user profile with existing user data in its internal database and authenticates the user based on the comparison result.

[0734] Input: User profile and existing user database

[0735] Data processing / data calculation: Compare and collate profile data with existing databases

[0736] Output: Authentication result (success or failure)

[0737] Step 8:

[0738] The authentication result is notified to the terminal, and access permission is provided to the user.

[0739] The server sends the authentication result (success or failure) to the terminal. If the authentication is successful, the terminal notifies the user of the access permission and allows them to log in to the metaverse. If the authentication fails, the terminal displays a warning message.

[0740] Input: Authentication result

[0741] Output: Notification of permission or warning message

[0742] As described above, this system collects user data and securely analyzes and authenticates it, enabling accurate and secure user authentication.

[0743] (Application example 1)

[0744] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0745] In recent years, the importance of security authentication in companies has increased, and traditional authentication using IDs and passwords is often insufficient. In particular, more advanced authentication systems are required to prevent unauthorized employee access and data leaks. However, currently used biometric authentication systems have room for improvement in terms of user experience and authentication accuracy. For these reasons, there is a demand for more secure and accurate authentication methods in corporate security authentication systems.

[0746] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0747] In this invention, the server includes means for collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, means for encrypting the collected data and transmitting it to the server, server means for integrating and analyzing the received data, server means for authenticating the user based on the analysis result, means for notifying the terminal of the authentication result, means for providing the user with access permission based on the authentication result, and means for collecting, analyzing, and authenticating user data using a VR headset in a corporate security authentication system. This enables companies to achieve more accurate and secure security authentication and reduce the risk of unauthorized access and data leakage.

[0748] "User" refers to a person who uses the system and is an individual or employee who is subject to security authentication.

[0749] "Emotional responses" are information that indicates a user's mental state, and collected data includes facial expressions and vocal tone.

[0750] "Facial expressions" are signs of emotion that appear on a user's face and are part of the data collected using a facial recognition camera.

[0751] "Heart rate" refers to the number of times the heart beats within a certain period of time and is part of the biometric data collected using biometric sensors.

[0752] "Tone of voice" refers to the quality and pitch of the user's voice and is part of the data collected using a microphone.

[0753] "Behavioral patterns" are part of the data collected by sensors and indicate characteristics of a user's movements and behavior.

[0754] "Biometric data" refers to information that indicates a user's physical characteristics, and refers to a wide range of data including heart rate, tone of voice, etc.

[0755] "Encryption" refers to the process of transforming collected data using specific algorithms to protect it.

[0756] "Server" refers to a central computer system that receives collected data, analyzes it, and manages authentication results.

[0757] "Analysis" refers to the process of synthesizing collected data and analyzing the information to authenticate a user.

[0758] "Authentication" refers to the process of verifying a user's identity and granting access rights based on collected data.

[0759] "Access permission" refers to allowing a user who has been successfully authenticated to use a specific system or service.

[0760] A "VR headset" is a device for experiencing virtual reality and refers to a terminal used to collect user data.

[0761] "Secure communications protocol" refers to a communications protocol used to encrypt and protect data transmissions.

[0762] The present invention relates to a method and system for enhancing a company's security authentication system by using a VR headset worn by a user to collect, analyze, and authenticate biometric and behavioral data to enhance security authentication.

[0763] System configuration

[0764] Hardware and Software Configuration

[0765] 1. VR headset (device)

[0766] Equipment: VR headsets are equipped with sensors to collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data.

[0767] Sensors: Facial recognition camera, heart rate sensor, microphone.

[0768] 2. Server

[0769] Function: Receives collected data, decrypts encrypted data, and has the processing power to perform analysis.

[0770] Software: Multimodal generative AI (e.g., OpenAI GPT-4), facial recognition algorithms (e.g., OpenCV), speech recognition algorithms (e.g., Google Cloud Speech-to-Text), and biometric data analysis algorithms.

[0771] 3. Communication Protocol

[0772] Protocol: A secure communication protocol (e.g. HTTPS) is used to send and receive data.

[0773] Program processing flow

[0774] 1. Collection of User Data

[0775] When a user puts on a VR headset to access a company's authentication system, sensors are activated to collect emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, which are then temporarily stored in memory.

[0776] 2. Data Encryption and Transmission

[0777] The collected data is encrypted using the AES-256 encryption algorithm and transmitted to the server using the HTTPS protocol.

[0778] 3. Data Receipt and Analysis

[0779] The server decrypts the received data and analyzes it in the following way:

[0780] Emotional responses and facial expressions are analyzed using a facial recognition algorithm (OpenCV).

[0781] The tone of voice is analyzed using a voice recognition algorithm (Google Cloud Speech-to-Text).

[0782] Heart rate is analyzed using biometric data analysis algorithms.

[0783] Behavioral patterns are integrated and analyzed using a multimodal generative AI model.

[0784] 4. User Authentication

[0785] Based on the parsed data, the server authenticates the user by matching the user profile with existing data in a database.

[0786] 5. Notification of authentication results and access permission

[0787] The server notifies the terminal of the authentication result, and if the authentication is successful, the user is granted access permission, ensuring the prevention of unauthorized access and the management of users.

[0788] Examples and prompts

[0789] Examples:

[0790] When the installed VR headset is used, it attempts to access the company's security authentication system.

[0791] For example: "Put on the VR headset to access the company's authentication system. Once you put it on, the sensors will automatically activate and collect data."

[0792] Example prompt sentence:

[0793] "Please put on your Oculus Quest and log in to the corporate authentication app. After logging in to the app, sensors will be activated to collect your emotional response, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. Based on this, you will be authenticated, so please just wait."

[0794] This process enables companies to achieve highly accurate and reliable security authentication, reducing the risk of unauthorized access and data leakage.

[0795] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0796] Step 1:

[0797] A user puts on a VR headset and accesses a company's authentication system.

[0798] Input: A user puts on a VR headset and attempts to log into the authentication system.

[0799] Action: A user launches a VR app and visits the login page.

[0800] Step 2:

[0801] The device activates sensors and collects user data.

[0802] Input: The sensors activated by the VR headset (face recognition camera, heart rate sensor, microphone).

[0803] Data processing: The device collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data and stores them in temporary memory.

[0804] How it works: A facial recognition camera captures your facial expressions, a heart rate sensor measures your heart rate, and a microphone records your tone of voice.

[0805] Step 3:

[0806] The terminal encrypts the collected user data and sends it to the server.

[0807] Input: Collected user data (facial expression data, heart rate data, voice data).

[0808] Data operation: The device encrypts the data using the AES-256 encryption algorithm.

[0809] Output: Encrypted user data.

[0810] How it works: The device sends encrypted data to the server using the HTTPS protocol.

[0811] Step 4:

[0812] The server receives the encrypted data and decrypts it.

[0813] Input: Encrypted user data from the device.

[0814] Data Calculation: The server decrypts the data using the AES-256 algorithm.

[0815] Output: Decrypted user data.

[0816] How it works: The server decrypts the data it receives using the HTTPS protocol.

[0817] Step 5:

[0818] The server analyzes and integrates various data.

[0819] Input: Decoded user data (facial expression data, heart rate data, voice data).

[0820] Data processing: The server analyzes the individual data using facial recognition algorithms (e.g., OpenCV), voice recognition algorithms (e.g., Google Cloud Speech-to-Text), and biometric data analysis algorithms, and then synthesizes the data using a generative AI model.

[0821] Output: A comprehensive user profile.

[0822] How it works: The server analyzes emotional responses and facial expressions using facial recognition algorithms, tone of voice using speech recognition algorithms, and heart rate using biometric data analysis algorithms.

[0823] Step 6:

[0824] The server authenticates the user based on their profile.

[0825] Input: A pooled and analyzed user profile.

[0826] Data calculation: The server checks the data against the user database and performs authentication.

[0827] Output: Authentication result (success / failure).

[0828] How it works: The server checks the generated user profile against its existing user database to determine if there is a match.

[0829] Step 7:

[0830] The server notifies the terminal of the authentication result.

[0831] Input: Authentication result (success / failure).

[0832] Output: A signal notifying the authentication result.

[0833] Operation: The server sends the authentication result to the terminal.

[0834] Step 8:

[0835] The terminal displays the authentication result to the user and provides access permission.

[0836] Input: Authentication result from the server.

[0837] Output: Notification to user (access permission / warning message).

[0838] Behavior: If the device is successfully authenticated, it displays the access permission and allows you to log in to the metaverse. If authentication fails, it displays a warning message.

[0839] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0840] Explaining program processing in natural language

[0841] Collection and transmission of user data

[0842] When a user puts on a VR headset and attempts to access the Metaverse, the device first collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. This collection is done using a facial recognition camera, heart rate sensor, microphone, and motion sensors. The device also uses an emotion engine to recognize and collect the user's emotional data in real time. This data is then stored in temporary memory.

[0843] Data encryption and transmission

[0844] The device encrypts the collected data using an encryption algorithm (e.g., AES-256), and the encrypted data is sent to the server using a secure communication protocol (e.g., HTTPS).

[0845] Data reception and analysis

[0846] The server receives and decrypts the encrypted data sent from the device. The decrypted data is categorized into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data, including emotional data obtained from the emotion engine. This data is then integrated using multimodal generative AI to generate a comprehensive user profile.

[0847] Analysis of individual data

[0848] The server analyzes the individual data in the following way:

[0849] Analyze the emotion recognition data obtained by the emotion engine.

[0850] Facial expression data is analyzed using a facial recognition algorithm.

[0851] A voice recognition algorithm is used to analyze tone of voice data.

[0852] Heart rate data is analyzed using a heart rate analysis algorithm.

[0853] Behavioral pattern data is analyzed using a behavioral prediction algorithm.

[0854] User Authentication

[0855] The server compares the generated user profile with existing user data in its internal database. Based on the comparison result, it determines whether the user is legitimate. If the user is legitimate, authentication is successful and a notification of this is sent to the terminal.

[0856] Notification of access permission to users

[0857] The terminal receives the authentication result from the server and, if successful, notifies the user of the access permission, allowing the user to log in to the metaverse. If the authentication fails, a corresponding warning message is displayed.

[0858] Specific examples

[0859] User logging into the metaverse scenario

[0860] 1. A user puts on a VR headset and attempts to access the Metaverse.

[0861] The user enters their ID and password and confirms the login screen.

[0862] 2. The device activates sensors and collects user data.

[0863] The device activates a facial recognition camera, heart rate sensor, microphone, emotion engine, and motion sensors to collect the user's facial expressions, heart rate, tone of voice, emotional data, and behavioral patterns.

[0864] 3. The device encrypts the data and sends it to the server.

[0865] The terminal encrypts the data with AES-256 and sends it to the server via the HTTPS protocol.

[0866] 4. The server receives and decrypts the data.

[0867] The server receives the data and decrypts it with AES-256.

[0868] 5. The server consolidates and analyzes the data.

[0869] The server performs emotion data recognized by the emotion engine, face recognition, voice recognition, heart rate analysis, and movement analysis to obtain the integrated results.

[0870] 6. The server authenticates the user against its user database.

[0871] The server checks its user database for a matching record.

[0872] 7. The server notifies the terminal of the authentication result.

[0873] The server notifies the terminal of successful authentication.

[0874] 8. The device notifies the user of the access permission.

[0875] The device displays the access permissions to the user and logs them into the metaverse.

[0876] In this way, the present invention can accurately and securely identify users and increase safety and engagement within the metaverse. The introduction of an emotion engine can more accurately grasp the user's psychological state, further strengthening security.

[0877] The processing flow will be explained below.

[0878] Step 1:

[0879] A user puts on a VR headset and attempts to access the Metaverse.

[0880] The user enters their ID and password on the login screen within the metaverse.

[0881] Step 2:

[0882] The device activates sensors and collects user data.

[0883] The device uses a facial recognition camera to collect facial expression data from the user.

[0884] The device uses a heart rate sensor to collect the user's heart rate data.

[0885] The terminal uses a microphone to collect data on the tone of the user's voice.

[0886] The device uses motion sensors to collect data on the user's behavioral patterns.

[0887] The terminal uses an emotion engine to collect the user's emotional response data.

[0888] Step 3:

[0889] The device encrypts the collected data.

[0890] The device uses the AES-256 encryption algorithm to encrypt collected facial expression data, heart rate data, tone of voice data, behavioral pattern data, and emotional response data.

[0891] Step 4:

[0892] The device sends the encrypted data to the server.

[0893] The device uses the HTTPS protocol to securely send encrypted data to the server.

[0894] Step 5:

[0895] The server decrypts the received data.

[0896] The server decrypts the received encrypted data using the AES-256 encryption algorithm.

[0897] Step 6:

[0898] The server classifies the decrypted data.

[0899] The server categorizes the decoded data into facial expression data, heart rate data, tone of voice data, behavioral pattern data, and emotional response data.

[0900] Step 7:

[0901] The server consolidates the data.

[0902] The server uses multimodal generative AI to integrate the categorized data and generate a comprehensive profile of the user.

[0903] Step 8:

[0904] The server analyzes the individual data.

[0905] The server analyzes the facial expression data using a facial recognition algorithm.

[0906] The server analyzes the tone of voice data using a voice recognition algorithm.

[0907] The server analyzes the heart rate data using a heart rate analysis algorithm.

[0908] The server analyzes the behavioral pattern data using a behavioral prediction algorithm.

[0909] The server uses an emotion engine to analyze the emotional response data.

[0910] Step 9:

[0911] The server authenticates the user based on the analysis results.

[0912] The server matches the generated user profile with existing data in its internal database.

[0913] Based on the result of the comparison, the server determines whether the user is legitimate.

[0914] Step 10:

[0915] The server notifies the terminal of the authentication result.

[0916] If the authentication is successful, the server notifies the terminal of access permission.

[0917] If the authentication fails, the server notifies the terminal of access denial.

[0918] Step 11:

[0919] The terminal notifies the user of the authentication result.

[0920] If the authentication is successful, the terminal allows the user to access the metaverse.

[0921] If the authentication fails, the terminal displays a warning message to the user.

[0922] In this way, the system combined with the emotion engine integrates and analyzes users' emotions, behaviors, and biometric data to achieve accurate and secure authentication, which can significantly improve safety and user engagement within the metaverse.

[0923] Example 2

[0924] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0925] In conventional systems, the process of collecting and authenticating users' personal information is complicated, and security is often insufficient. In particular, there is a lack of means to integrate and analyze multimodal data such as emotional and behavioral data, which leads to problems with the accuracy of user authentication. This leads to a poor user experience in metaverse and virtual reality environments, and safety is not guaranteed.

[0926] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0927] In this invention, the server includes means for collecting a user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, means for storing the collected data in a temporary storage device, means for encrypting the collected data using the AES-256 algorithm, means for transmitting the encrypted data to the server using a secure communication protocol, means for decrypting the received data using the AES-256 algorithm, means for classifying the decrypted data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral patterns, and biometric data and integrating them using multimodal generation AI, means for analyzing the integrated data, means for authenticating the user based on the analysis result, means for notifying the terminal of the authentication result, and means for providing access permission to the user based on the authentication result. This allows the process from collecting user data to granting access permission to be performed safely and efficiently, thereby improving user safety and engagement in the metaverse.

[0928] A "user" is a person who uses a client device to access the system and use the services.

[0929] "Emotional response" is data that indicates the user's psychological state, and is information that is analyzed in real time using an emotion engine.

[0930] "Facial expression" is data that indicates the state of the user's face, and is information collected by a face recognition camera.

[0931] "Heart rate" is data indicating the frequency of the user's heartbeat, and is information measured using a heart rate sensor.

[0932] "Voice tone" is data that indicates the quality and pitch of a user's voice when speaking, and is information collected using a microphone.

[0933] "Behavior patterns" are data that indicate the user's movements and behavioral tendencies, and are information captured using a movement sensor.

[0934] "Biometric data" refers to data related to the user's physical characteristics, including information such as heart rate and facial recognition data.

[0935] A "temporary storage device" is a storage device for temporarily storing collected data.

[0936] "AES-256 Algorithm" means an advanced encryption algorithm and technology used to encrypt data.

[0937] A "secure communication protocol" is a communication protocol for ensuring the safe transfer of data, such as HTTPS.

[0938] A "server" is a computer system that receives, decrypts, analyzes collected data, and performs user authentication.

[0939] "Emotion recognition data" is data relating to the user's emotional state, analyzed by the emotion engine.

[0940] "Facial expression data" is data relating to the state of the user's face, and is collected by a face recognition camera.

[0941] "Heart rate data" is data relating to the user's heart rate, and is acquired by a heart rate sensor.

[0942] "Voice tone data" is data relating to the quality and pitch of the user's voice, and is collected by a microphone.

[0943] "Behavior pattern data" is data relating to the user's movements and behaviors, and is captured by a movement sensor.

[0944] "Multimodal generative AI" is an artificial intelligence model for integrating and analyzing multiple different types of data.

[0945] "User authentication" is the process of verifying that a user is a legitimate user by comparing collected data with existing database information.

[0946] "Access permission" is a control that allows a user who has been successfully authenticated to access a system or service.

[0947] This invention relates to a system that safely and efficiently collects and analyzes personal data and performs accurate authentication when a user wears a VR headset and accesses the metaverse.

[0948] Hardware and software used

[0949] When a user puts on a VR headset, access to the Metaverse begins. At this time, the following hardware on the device collects data:

[0950] Facial recognition camera: Captures the user's facial expressions.

[0951] Heart rate sensor: Measures the user's heart rate.

[0952] Microphone: Collects the user's tone of voice.

[0953] Motion sensors: Capture user behavior patterns.

[0954] Emotion Engine: Analyzes users' emotional responses in real time.

[0955] This data is temporarily stored in the device's temporary storage, after which the device encrypts the data using the AES-256 algorithm and transmits it to the server via the secure HTTPS communication protocol.

[0956] The server does the following:

[0957] Decrypting received data: Decrypts encrypted data using the AES-256 algorithm.

[0958] Data Classification and Integration: Classify the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data using multimodal generative AI.

[0959] Data analysis: The server analyzes the data using various algorithms. Specifically, an emotion engine is used to analyze emotion recognition data, a face recognition algorithm is used to analyze facial expression data, and a voice recognition algorithm is used to analyze tone of voice data.

[0960] Perform user authentication: Based on the integrated data, the user is authenticated. This process involves checking against an existing user database.

[0961] Notification of authentication result: The server notifies the terminal of the authentication result.

[0962] Specific examples

[0963] For example, the following specific operations are performed.

[0964] 1. A user puts on a VR headset and attempts to access the Metaverse.

[0965] The user enters their ID and password and confirms the login screen.

[0966] 2. The device activates sensors and collects user data.

[0967] The device activates a facial recognition camera, heart rate sensor, microphone, emotion engine, and motion sensors to collect the user's facial expressions, heart rate, tone of voice, emotional data, and behavioral patterns.

[0968] 3. The device encrypts the data and sends it to the server.

[0969] The device encrypts the collected data with AES-256 and sends it to the server via the HTTPS protocol.

[0970] 4. The server receives and decrypts the data.

[0971] The server receives the data and decrypts it with AES-256.

[0972] 5. The server consolidates and analyzes the data.

[0973] The server performs emotion data recognized by the emotion engine, facial recognition, voice recognition, heart rate analysis, and movement analysis to obtain the integrated results.

[0974] 6. The server authenticates the user against its user database.

[0975] The server checks its user database for a matching record.

[0976] 7. The server notifies the terminal of the authentication result.

[0977] The server notifies the terminal of successful authentication.

[0978] 8. The device notifies the user of the access permission.

[0979] The device displays the access permissions to the user and logs them into the metaverse.

[0980] This system provides a secure and efficient authentication process, from collecting user data to granting access to the metaverse. An example prompt is as follows:

[0981] "Please describe in natural language a scenario in which a user logs into the Metaverse using a VR headset. Please specifically describe the process from user data collection to authentication and access authorization."

[0982] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0983] Step 1:

[0984] A user puts on a VR headset and attempts to access the Metaverse.

[0985] Specifically, this includes the user entering their ID and password to confirm the login screen, which generates an access request and triggers the device to start the authentication process.

[0986] Input: User login information (ID, password)

[0987] Output: Trigger to start the authentication process

[0988] Step 2:

[0989] The device activates sensors and collects user data.

[0990] Specifically, the device's built-in facial recognition camera, heart rate sensor, microphone, motion sensor, and emotion engine are all activated to collect the user's facial expressions, heart rate, tone of voice, behavioral patterns, and emotional data in real time.

[0991] Input: User's biometric information (face, heart rate, voice, movement, emotion)

[0992] Output: Collected bio-behavioral data

[0993] Step 3:

[0994] The terminal stores the collected data in temporary storage.

[0995] Specifically, this involves caching the collected data in temporary storage.

[0996] Input: Collected biometric and behavioral data

[0997] Output: Data stored in temporary storage

[0998] Step 4:

[0999] The device encrypts the collected data using the AES-256 algorithm.

[1000] Specifically, this involves automatically passing data stored in temporary storage to an encryption module where it is encrypted using AES-256.

[1001] Input: Data stored in temporary storage

[1002] Output: Encrypted data

[1003] Step 5:

[1004] The device sends encrypted data to the server using the secure HTTPS protocol.

[1005] Specifically, this includes operations in which encrypted data is transmitted using the HTTPS protocol.

[1006] Input: Encrypted data

[1007] Output: Send data to the server

[1008] Step 6:

[1009] The server receives the encrypted data and decrypts it using the AES-256 algorithm.

[1010] Specifically, the encrypted data is passed to the receiving module and decrypted using AES-256.

[1011] Input: Encrypted data

[1012] Output: Decrypted data

[1013] Step 7:

[1014] The server classifies the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data, and integrates them using multimodal generative AI.

[1015] Specifically, the decoded data is classified into categories, sent to the respective analysis modules, and an integration process is carried out by the multimodal generation AI.

[1016] Input: Decrypted data

[1017] Output: Unified user profile

[1018] Step 8:

[1019] The server analyzes the combined data and authenticates the user.

[1020] Specifically, this involves checking the integrated user profile against an internal database to determine legitimacy.

[1021] Input: Unified User Profile

[1022] Output: Authentication result

[1023] Step 9:

[1024] The server notifies the terminal of the authentication result.

[1025] Specifically, this includes an operation of transmitting the result of authentication success or failure to the terminal.

[1026] Input: Authentication result

[1027] Output: Authentication result notification to the terminal

[1028] Step 10:

[1029] The terminal notifies the user of the access permission.

[1030] Specifically, if authentication is successful, the terminal will display the access permission to the metaverse to the user, and if authentication fails, a corresponding warning message will be displayed.

[1031] Input: Authentication result notification to the terminal

[1032] Output: Allow access to user or warning message

[1033] (Application example 2)

[1034] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1035] Modern authentication systems have adopted a wide range of biometric and behavioral authentication methods for security reasons. However, because these methods rely on individual data, it is difficult to completely eliminate the risk of impersonation and unauthorized access. In particular, when using emotional or biometric data alone, they are susceptible to temporary changes in the user's state and environmental factors, making it difficult to obtain consistent authentication results. Furthermore, there are insufficient means to integrate these data and perform accurate and secure authentication based on a comprehensive user profile.

[1036] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, means for determining the legitimacy of the user based on a user profile generated using the collected biometric data and emotion recognition data, and means for providing access permission to the user based on the authentication result. This provides a consistent authentication result that integrates multiple data, eliminates impersonation and unauthorized access, and enables more secure and reliable authentication.

[1037] Below are definitions of important terms extracted from claim 1:

[1038] "User data" is a general term for information about a user, such as emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data.

[1039] "Encryption" refers to the process of transforming data using specific algorithms to make it unreadable to third parties.

[1040] A "server" is a computer system that stores, analyzes, and processes data over a network.

[1041] "Integration" refers to the act of bringing together multiple types of data and processing them in a consistent format or with consistent standards.

[1042] "Analysis" is the process of examining collected data in detail to find meaning and patterns.

[1043] "Authentication" is the process of verifying that a user has valid access rights.

[1044] A "terminal" is an electronic device that is directly operated by a user, and includes smartphones, personal computers, etc.

[1045] "Access permission" refers to granting access to a system or data.

[1046] "Biometric data" refers to information obtained from the user's body, such as heart rate, fingerprints, and facial shape.

[1047] "Emotion recognition data" is data used to analyze a user's mental state and emotions.

[1048] A "user profile" is a consistent set of information about a user that is obtained by integrating various collected data.

[1049] The present invention provides an advanced authentication system by collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, and integrating and analyzing this data. Specific embodiments for carrying out the present invention will be described below.

[1050] System Configuration

[1051] The system consists of the following main components:

[1052] Terminal: A device that is directly used by a user, such as a smartphone, smart glasses, a head-mounted display, or a robot.

[1053] Sensors: Face recognition camera, heart rate sensor, microphone, and motion sensor installed on the device.

[1054] Server: A computer system that stores, analyzes, and integrates data.

[1055] Communication Protocol: A secure communication protocol (e.g. HTTPS) for sending and receiving data.

[1056] Encryption algorithm: The encryption method used to ensure secure transmission of data (e.g., AES-256).

[1057] Emotion Engine: An API for analyzing emotions from collected data.

[1058] Program processing overview

[1059] When a user attempts to log in using a device, the device first collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using a facial recognition camera, heart rate sensor, microphone, and motion sensors. Using an emotion engine, the device analyzes the collected data in real time to generate emotional data. This data is then stored in temporary memory.

[1060] Next, the device encrypts the collected data with an encryption algorithm (e.g., AES-256) and sends it to the server using a secure communication protocol (e.g., HTTPS). The server receives and decrypts the encrypted data.

[1061] The server analyzes the decoded data using an emotion engine, face recognition algorithm, voice recognition algorithm, heart rate analysis algorithm, and behavior prediction algorithm to classify it into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavior pattern data, and biometric data. A comprehensive user profile is generated from the integrated data.

[1062] Based on the generated user profile, the server compares it with existing user data in its internal database to determine the user's legitimacy. Based on the comparison result, the server notifies the terminal of the authentication result, and the terminal displays the result to the user. If authentication is successful, the user is granted access permission.

[1063] Specific examples of hardware and software used

[1064] Device: Compatible smartphones and head-mounted displays.

[1065] Emotion engines: for example, Affectiva or Microsoft Azure Face API.

[1066] Encryption algorithm: AES-256.

[1067] Communication protocol: HTTPS.

[1068] Analysis algorithms: Multimodal generative AI, OpenCV (face recognition), Google Cloud Speech-to-Text (voice recognition).

[1069] Specific examples

[1070] When a user launches the security app, the app uses the smartphone's camera to capture the user's facial expressions, the heart rate sensor to obtain their heart rate, the microphone to record audio and analyze their tone of voice, and the motion sensor to record their behavioral patterns. This data is then analyzed in real time as emotional data using an emotion engine. The data is then encrypted with AES-256 and sent to a secure server via HTTPS. The server receives, decrypts, and analyzes the data to generate a comprehensive user profile. The server then authenticates the user and notifies the smartphone of the authentication result.

[1071] Prompt Sentence Examples

[1072] Here are some example prompts to input to a generative AI model:

[1073] Describe a system that collects a user's facial recognition, heart rate, tone of voice, behavioral patterns, and biometric data, and then uses an emotion engine to analyze the emotional data in real time. This data is integrated to generate a comprehensive user profile and perform user authentication.

[1074] In this way, the present invention integrates data from multiple data sources and provides a high level of security, thereby improving the reliability and accuracy of authentication.

[1075] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1076] Step 1: Collect user data

[1077] When a user attempts to log in using a device, the device activates various sensors, including a facial recognition camera, heart rate sensor, microphone, and motion sensor, which are used to collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. The input is the user's biometric data, and the output is a set of collected biometric data.

[1078] Step 2: Analyze the emotion data

[1079] The collected biometric data is analyzed in real time into emotional data using an emotion engine, where each data point is mapped to an emotional state. The input is the collected biometric data, and the output is the analyzed emotional data.

[1080] Step 3: Temporarily save data

[1081] The terminal stores the collected data and analyzed emotional data in a temporary memory. The input of this process is the analyzed emotional data and biometric data, and the output is the set of data in the temporary memory.

[1082] Step 4: Encrypt the data

[1083] The terminal encrypts the data in temporary memory using an encryption algorithm (e.g., AES-256). The input is the data stored in temporary memory, and the output is the encrypted data.

[1084] Step 5: Sending data

[1085] The encrypted data is sent to the server using a secure communication protocol (e.g., HTTPS). The input is the encrypted data and the output is the data sent to the server.

[1086] Step 6: Receiving and Decrypting Data

[1087] The server receives the encrypted data and decrypts it with a decryption algorithm (e.g., AES-256). The input is the encrypted data and the output is the decrypted data.

[1088] Step 7: Classify the data

[1089] The server classifies the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data. The input to this process is the decoded data, and the output is the various classified data.

[1090] Step 8: Analyze the data

[1091] The various classified data are analyzed using an emotion engine, face recognition algorithm, voice recognition algorithm, heart rate analysis algorithm, and behavior prediction algorithm. The input is the classified data, and the output is the analysis results.

[1092] Step 9: Generate an Integration Profile

[1093] The server integrates the analysis results and generates a comprehensive user profile. The input is the analyzed data, and the output is the integrated user profile.

[1094] Step 10: User Authentication

[1095] The server compares the generated user profile with existing user data in its internal database to determine its validity. The input is the integrated user profile, and the output is the authentication result.

[1096] Step 11: Notification of authentication results

[1097] The server sends the authentication result to the terminal, and the terminal notifies the user of the result. The input is the authentication result, and the output is the notification to the user.

[1098] Step 12: Access Permissions

[1099] If the authentication is successful, the terminal grants the user access permission. The input is the authentication result, and the output is the granting of access permission.

[1100] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1101] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1102] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1103] [Third embodiment]

[1104] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1105] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1106] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1107] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1108] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1109] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1110] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1111] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1112] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1113] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1114] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1115] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1116] Explaining program processing in natural language

[1117] Collection and transmission of user data

[1118] When a user puts on a VR headset and attempts to access the Metaverse, the device collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using facial recognition cameras, heart rate sensors, microphones, etc. The collected data is then stored in temporary memory by the device.

[1119] Data encryption and transmission

[1120] The device encrypts the collected data using an encryption algorithm (e.g., AES-256), and then transmits the encrypted data to the server using a secure communication protocol (e.g., HTTPS).

[1121] Data reception and analysis

[1122] The server receives and decrypts the encrypted data sent from the device. The decrypted data is separated into different categories (emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data). This data is then integrated using multimodal generative AI to generate a comprehensive user profile.

[1123] Analysis of individual data

[1124] The server analyzes the individual data in the following way:

[1125] Emotional responses and facial expressions are run through a facial recognition algorithm.

[1126] The tone of voice is analyzed using a voice recognition algorithm.

[1127] The heart rate is analyzed by the biosensor's data analysis algorithm.

[1128] The behavioral patterns are analyzed based on a motion prediction algorithm.

[1129] User Authentication

[1130] The server compares the generated user profile with existing user data in its internal database. Based on the comparison result, it determines whether the user is legitimate. If the user is legitimate, authentication is successful and a notification of this is sent to the terminal.

[1131] Notification of access permission to users

[1132] The terminal receives the authentication result from the server and, if successful, notifies the user of the access permission, allowing the user to log in to the metaverse. If the authentication fails, a corresponding warning message is displayed.

[1133] Specific examples

[1134] User logging into the metaverse scenario

[1135] 1. A user puts on a VR headset and attempts to access the Metaverse.

[1136] The user enters their ID and password and confirms the login screen.

[1137] 2. The device activates sensors and collects user data.

[1138] The device will enable a facial recognition camera, heart rate sensor, and microphone to collect the user's facial expressions, heart rate, and voice data.

[1139] 3. The device encrypts the data and sends it to the server.

[1140] The terminal encrypts the data with AES-256 and sends it to the server via the HTTPS protocol.

[1141] 4. The server receives and decrypts the data.

[1142] The server receives the data and decrypts it with AES-256.

[1143] 5. The server consolidates and analyzes the data.

[1144] The server performs facial recognition, voice recognition, and heart rate analysis to obtain an integrated result.

[1145] 6. The server authenticates the user against its user database.

[1146] The server checks its user database for a matching record.

[1147] 7. The server notifies the terminal of the authentication result.

[1148] The server notifies the terminal of successful authentication.

[1149] 8. The device notifies the user of the access permission.

[1150] The device displays the access permissions to the user and logs them into the metaverse.

[1151] In this way, the present invention can accurately and securely identify users, increasing safety and engagement within the metaverse.

[1152] The processing flow will be explained below.

[1153] Step 1:

[1154] A user puts on a VR headset and attempts to access the Metaverse.

[1155] The user enters their ID and password and proceeds to the login screen.

[1156] Step 2:

[1157] The device activates sensors and collects user data.

[1158] The device uses a facial recognition camera to collect facial expression data from the user.

[1159] The device uses a heart rate sensor to collect the user's heart rate data.

[1160] The terminal uses a microphone to collect data on the tone of the user's voice.

[1161] The device uses motion sensors to collect data on the user's behavioral patterns.

[1162] Step 3:

[1163] The device encrypts the collected data.

[1164] The device uses the AES-256 encryption algorithm to encrypt collected facial expression data, heart rate data, tone of voice data, and behavioral pattern data.

[1165] Step 4:

[1166] The device sends the encrypted data to the server.

[1167] The device uses the HTTPS protocol to securely send encrypted data to the server.

[1168] Step 5:

[1169] The server decrypts the received data.

[1170] The server decrypts the received encrypted data using the AES-256 encryption algorithm.

[1171] Step 6:

[1172] The server consolidates the decrypted data.

[1173] The server integrates facial recognition data, heart rate data, tone of voice data, and behavioral pattern data using multimodal generation AI to generate a user profile.

[1174] Step 7:

[1175] The server analyzes the individual data.

[1176] The server analyzes the facial expression data using a facial recognition algorithm.

[1177] The server analyzes the tone of voice data using a voice recognition algorithm.

[1178] The server analyzes the heart rate data using a heart rate analysis algorithm.

[1179] The server analyzes the behavioral pattern data using a behavioral prediction algorithm.

[1180] Step 8:

[1181] The server authenticates the user based on the analysis results.

[1182] The server matches the generated user profile with existing data in its internal database.

[1183] Based on the result of the comparison, the server determines whether the user is legitimate.

[1184] Step 9:

[1185] The server notifies the terminal of the authentication result.

[1186] If the authentication is successful, the server notifies the terminal of access permission.

[1187] If the authentication fails, the server notifies the terminal of access denial.

[1188] Step 10:

[1189] The terminal notifies the user of the authentication result.

[1190] If the authentication is successful, the terminal allows the user to access the metaverse.

[1191] If the authentication fails, the terminal displays a warning message to the user.

[1192] Example 1

[1193] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1194] Conventional virtual reality (VR) systems have faced problems with insufficiently secure user authentication methods, resulting in a high risk of easy unauthorized access. Furthermore, there was a lack of a means for centralized and detailed analysis of individual user data, making it difficult to improve the quality of the user experience. Furthermore, there was no established secure method for transferring collected data, creating a risk of data leakage. Therefore, providing a highly accurate and secure authentication system that users could use with confidence was a challenge.

[1195] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1196] In this invention, the server includes means for collecting emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using a facial recognition camera, heart rate sensor, and microphone when a user wears a VR headset and attempts to access the metaverse, means for encrypting the collected data using the AES-256 algorithm and transmitting it to the server via HTTPS protocol, and server means for decrypting the received data and integrating and analyzing the emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using multimodal generation AI, thereby enabling secure transfer of the collected data and highly accurate user authentication.

[1197] "User" refers to an individual who wears a VR headset and accesses the Metaverse.

[1198] "Device" refers to the device including the VR headset worn by the user and its related equipment.

[1199] "Server" refers to a computer on the system side that receives data sent from a user and analyzes and authenticates it.

[1200] A "face recognition camera" refers to a photographic device that captures a user's facial expressions and features.

[1201] "Heart rate sensor" refers to a detection device for measuring a user's heart rate.

[1202] "Microphone" refers to a recording device for collecting the user's voice.

[1203] "Emotional response" refers to changes in the emotions shown by the user, and is determined from facial expressions, tone of voice, etc.

[1204] "Facial expressions" refer to movements that show emotions and reactions on the user's face.

[1205] "Heart rate" refers to the number of times a user's heart beats within a certain period of time.

[1206] "Tone of voice" refers to characteristics such as the pitch and intonation of a user's voice.

[1207] "Behavioral patterns" refer to a series of actions and tendencies of a user's reactions.

[1208] "Biometric data" refers to various information collected from a user's body (e.g., heart rate, pulse, body temperature).

[1209] The "AES-256 algorithm" is one of the Advanced Encryption Standards for encrypting data, and uses a 256-bit key.

[1210] "HTTPS protocol" refers to a communications protocol for securely transferring data over the Internet.

[1211] "Multimodal generative AI" refers to an artificial intelligence model that integrates and analyzes multiple data modalities (e.g., audio, images, text).

[1212] "Encryption" refers to the process of transforming data so that it cannot be read by third parties.

[1213] "Decryption" refers to the process of restoring encrypted data to its original state.

[1214] "User profile" refers to a collection of information about an individual user that is generated by integrating various collected data.

[1215] "Authentication" refers to the process of verifying that a user is a specific individual.

[1216] "Access permission" refers to granting a user the right to access a particular system or information.

[1217] "Warning message" refers to the notification displayed to the user when authentication fails.

[1218] This invention is a system for accurately and securely authenticating users when they wear a VR headset and access the Metaverse. This system works through several steps, mainly involving data collection, encryption, transmission, reception, analysis, and authentication processes between the device, server, and user.

[1219] User Data Collection

[1220] When a user puts on a VR headset and starts an application, the device first activates sensors, including a facial recognition camera, heart rate sensor, and microphone. These devices collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. For example, the facial recognition camera captures facial expression data, the heart rate sensor monitors heart rate, and the microphone records tone of voice.

[1221] Data encryption and transmission

[1222] The collected data is temporarily stored on the device and then encrypted using the AES-256 encryption algorithm, and then transmitted to the server using the HTTPS protocol. This process ensures the security of the data.

[1223] Data analysis on the server

[1224] Once the data arrives at the server, it is first decoded. After decoding, the data is separated into categories (emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data). The server then uses multimodal generative AI to integrate and analyze this data. This results in a comprehensive user profile, which includes, for example, current emotional state, heart rate trends, and usual behavioral patterns.

[1225] User Authentication

[1226] The generated user profile is compared with existing user data stored in the server's internal database. This comparison determines whether the user has valid access rights. If authentication is successful, the server notifies the terminal of the result. If authentication fails, the terminal displays a warning message to the user.

[1227] User notification and access permissions

[1228] The terminal receives the authentication result from the server and notifies the user of the access permission if successful, allowing the user to log in to the metaverse and access interactive content. If the authentication fails, a corresponding warning message will be displayed to the user, prompting them to try again.

[1229] The system will improve the user experience and ensure safety within the metaverse by collecting, analyzing, and authenticating user data with high accuracy and security.

[1230] Prompt Sentence Examples

[1231] Finally, the following are some specific examples of prompts for generative AI models:

[1232] "Please explain in natural language the system that collects biometric data from users using a VR headset and analyzes it to authenticate the user."

[1233] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1234] Step 1:

[1235] The user puts on a VR headset and prepares to access the Metaverse.

[1236] The user turns on the VR headset, launches the application, and proceeds to the login screen where they enter their ID and password.

[1237] Input: User ID and password

[1238] Output: Access to login screen

[1239] Step 2:

[1240] The device activates sensors and collects user data.

[1241] The device activates a facial recognition camera, heart rate sensor and microphone to simultaneously collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns and biometric data.

[1242] Input: User's current biometric and behavioral data

[1243] Data processing / data calculation: Analyzes images taken with a facial recognition camera and extracts facial expression data. Records heart rate measured with a heart rate sensor. Analyzes audio data collected with a microphone and identifies voice tone.

[1244] Output: Multiple types of biometric data collected

[1245] Step 3:

[1246] Encrypt the data collected by the device.

[1247] The device encrypts all collected data using the AES-256 encryption algorithm.

[1248] Input: Unencrypted biometric data

[1249] Data processing / data calculation: Encrypt data using the AES-256 algorithm

[1250] Output: Encrypted biometric data

[1251] Step 4:

[1252] The encrypted data is sent to the server.

[1253] The device uses the HTTPS protocol to securely transmit encrypted data to the server.

[1254] Input: Encrypted biometric data

[1255] Output: Send data to the server

[1256] Step 5:

[1257] The server receives and decrypts the data.

[1258] The server receives the data using the HTTPS protocol and decrypts it using the AES-256 algorithm.

[1259] Input: Encrypted biometric data

[1260] Data processing / data calculation: Receive data using HTTPS protocol and decrypt it using AES-256 algorithm

[1261] Output: Decoded biometric data

[1262] Step 6:

[1263] The server categorizes and analyzes the data.

[1264] The server separates the decoded data into emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, and then uses multimodal generative AI to integrate and analyze this data.

[1265] Input: Decoded biometric data

[1266] Data processing / data calculation: Categorize data and perform integrated analysis using multimodal generation AI

[1267] Output: Comprehensive user profile

[1268] Step 7:

[1269] The server checks the user profile against an internal database.

[1270] The server compares the generated user profile with existing user data in its internal database and authenticates the user based on the comparison result.

[1271] Input: User profile and existing user database

[1272] Data processing / data calculation: Compare and collate profile data with existing databases

[1273] Output: Authentication result (success or failure)

[1274] Step 8:

[1275] The authentication result is notified to the terminal, and access permission is provided to the user.

[1276] The server sends the authentication result (success or failure) to the terminal. If the authentication is successful, the terminal notifies the user of the access permission and allows them to log in to the metaverse. If the authentication fails, the terminal displays a warning message.

[1277] Input: Authentication result

[1278] Output: Notification of permission or warning message

[1279] As described above, this system collects user data and securely analyzes and authenticates it, enabling accurate and secure user authentication.

[1280] (Application example 1)

[1281] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1282] In recent years, the importance of security authentication in companies has increased, and traditional authentication using IDs and passwords is often insufficient. In particular, more advanced authentication systems are required to prevent unauthorized employee access and data leaks. However, currently used biometric authentication systems have room for improvement in terms of user experience and authentication accuracy. For these reasons, there is a demand for more secure and accurate authentication methods in corporate security authentication systems.

[1283] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1284] In this invention, the server includes means for collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, means for encrypting the collected data and transmitting it to the server, server means for integrating and analyzing the received data, server means for authenticating the user based on the analysis result, means for notifying the terminal of the authentication result, means for providing the user with access permission based on the authentication result, and means for collecting, analyzing, and authenticating user data using a VR headset in a corporate security authentication system. This enables companies to achieve more accurate and secure security authentication and reduce the risk of unauthorized access and data leakage.

[1285] "User" refers to a person who uses the system and is an individual or employee who is subject to security authentication.

[1286] "Emotional responses" are information that indicates a user's mental state, and collected data includes facial expressions and vocal tone.

[1287] "Facial expressions" are signs of emotion that appear on a user's face and are part of the data collected using a facial recognition camera.

[1288] "Heart rate" refers to the number of times the heart beats within a certain period of time and is part of the biometric data collected using biometric sensors.

[1289] "Tone of voice" refers to the quality and pitch of the user's voice and is part of the data collected using a microphone.

[1290] "Behavioral patterns" are part of the data collected by sensors and indicate characteristics of a user's movements and behavior.

[1291] "Biometric data" refers to information that indicates a user's physical characteristics, and refers to a wide range of data including heart rate, tone of voice, etc.

[1292] "Encryption" refers to the process of transforming collected data using specific algorithms to protect it.

[1293] "Server" refers to a central computer system that receives collected data, analyzes it, and manages authentication results.

[1294] "Analysis" refers to the process of synthesizing collected data and analyzing the information to authenticate a user.

[1295] "Authentication" refers to the process of verifying a user's identity and granting access rights based on collected data.

[1296] "Access permission" refers to allowing a user who has been successfully authenticated to use a specific system or service.

[1297] A "VR headset" is a device for experiencing virtual reality and refers to a terminal used to collect user data.

[1298] "Secure communications protocol" refers to a communications protocol used to encrypt and protect data transmissions.

[1299] The present invention relates to a method and system for enhancing a company's security authentication system by using a VR headset worn by a user to collect, analyze, and authenticate biometric and behavioral data to enhance security authentication.

[1300] System configuration

[1301] Hardware and Software Configuration

[1302] 1. VR headset (device)

[1303] Equipment: VR headsets are equipped with sensors to collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data.

[1304] Sensors: Facial recognition camera, heart rate sensor, microphone.

[1305] 2. Server

[1306] Function: Receives collected data, decrypts encrypted data, and has the processing power to perform analysis.

[1307] Software: Multimodal generative AI (e.g., OpenAI GPT-4), facial recognition algorithms (e.g., OpenCV), speech recognition algorithms (e.g., Google Cloud Speech-to-Text), and biometric data analysis algorithms.

[1308] 3. Communication Protocol

[1309] Protocol: A secure communication protocol (e.g. HTTPS) is used to send and receive data.

[1310] Program processing flow

[1311] 1. Collection of User Data

[1312] When a user puts on a VR headset to access a company's authentication system, sensors are activated to collect emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, which are then temporarily stored in memory.

[1313] 2. Data Encryption and Transmission

[1314] The collected data is encrypted using the AES-256 encryption algorithm and transmitted to the server using the HTTPS protocol.

[1315] 3. Data Receipt and Analysis

[1316] The server decrypts the received data and analyzes it in the following way:

[1317] Emotional responses and facial expressions are analyzed using a facial recognition algorithm (OpenCV).

[1318] The tone of voice is analyzed using a voice recognition algorithm (Google Cloud Speech-to-Text).

[1319] Heart rate is analyzed using biometric data analysis algorithms.

[1320] Behavioral patterns are integrated and analyzed using a multimodal generative AI model.

[1321] 4. User Authentication

[1322] Based on the parsed data, the server authenticates the user by matching the user profile with existing data in a database.

[1323] 5. Notification of authentication results and access permission

[1324] The server notifies the terminal of the authentication result, and if the authentication is successful, the user is granted access permission, ensuring the prevention of unauthorized access and the management of users.

[1325] Examples and prompts

[1326] Examples:

[1327] When the installed VR headset is used, it attempts to access the company's security authentication system.

[1328] For example: "Put on the VR headset to access the company's authentication system. Once you put it on, the sensors will automatically activate and collect data."

[1329] Example prompt sentence:

[1330] "Please put on your Oculus Quest and log in to the corporate authentication app. After logging in to the app, sensors will be activated to collect your emotional response, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. Based on this, you will be authenticated, so please just wait."

[1331] This process enables companies to achieve highly accurate and reliable security authentication, reducing the risk of unauthorized access and data leakage.

[1332] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1333] Step 1:

[1334] A user puts on a VR headset and accesses a company's authentication system.

[1335] Input: A user puts on a VR headset and attempts to log into the authentication system.

[1336] Action: A user launches a VR app and visits the login page.

[1337] Step 2:

[1338] The device activates sensors and collects user data.

[1339] Input: The sensors activated by the VR headset (face recognition camera, heart rate sensor, microphone).

[1340] Data processing: The device collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data and stores them in temporary memory.

[1341] How it works: A facial recognition camera captures your facial expressions, a heart rate sensor measures your heart rate, and a microphone records your tone of voice.

[1342] Step 3:

[1343] The terminal encrypts the collected user data and sends it to the server.

[1344] Input: Collected user data (facial expression data, heart rate data, voice data).

[1345] Data operation: The device encrypts the data using the AES-256 encryption algorithm.

[1346] Output: Encrypted user data.

[1347] How it works: The device sends encrypted data to the server using the HTTPS protocol.

[1348] Step 4:

[1349] The server receives the encrypted data and decrypts it.

[1350] Input: Encrypted user data from the device.

[1351] Data Calculation: The server decrypts the data using the AES-256 algorithm.

[1352] Output: Decrypted user data.

[1353] How it works: The server decrypts the data it receives using the HTTPS protocol.

[1354] Step 5:

[1355] The server analyzes and integrates various data.

[1356] Input: Decoded user data (facial expression data, heart rate data, voice data).

[1357] Data processing: The server analyzes the individual data using facial recognition algorithms (e.g., OpenCV), voice recognition algorithms (e.g., Google Cloud Speech-to-Text), and biometric data analysis algorithms, and then synthesizes the data using a generative AI model.

[1358] Output: A comprehensive user profile.

[1359] How it works: The server analyzes emotional responses and facial expressions using facial recognition algorithms, tone of voice using speech recognition algorithms, and heart rate using biometric data analysis algorithms.

[1360] Step 6:

[1361] The server authenticates the user based on their profile.

[1362] Input: A pooled and analyzed user profile.

[1363] Data calculation: The server checks the data against the user database and performs authentication.

[1364] Output: Authentication result (success / failure).

[1365] How it works: The server checks the generated user profile against its existing user database to determine if there is a match.

[1366] Step 7:

[1367] The server notifies the terminal of the authentication result.

[1368] Input: Authentication result (success / failure).

[1369] Output: A signal notifying the authentication result.

[1370] Operation: The server sends the authentication result to the terminal.

[1371] Step 8:

[1372] The terminal displays the authentication result to the user and provides access permission.

[1373] Input: Authentication result from the server.

[1374] Output: Notification to user (access permission / warning message).

[1375] Behavior: If the device is successfully authenticated, it displays the access permission and allows you to log in to the metaverse. If authentication fails, it displays a warning message.

[1376] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1377] Explaining program processing in natural language

[1378] Collection and transmission of user data

[1379] When a user puts on a VR headset and attempts to access the Metaverse, the device first collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. This collection is done using a facial recognition camera, heart rate sensor, microphone, and motion sensors. The device also uses an emotion engine to recognize and collect the user's emotional data in real time. This data is then stored in temporary memory.

[1380] Data encryption and transmission

[1381] The device encrypts the collected data using an encryption algorithm (e.g., AES-256), and the encrypted data is sent to the server using a secure communication protocol (e.g., HTTPS).

[1382] Data reception and analysis

[1383] The server receives and decrypts the encrypted data sent from the device. The decrypted data is categorized into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data, including emotional data obtained from the emotion engine. This data is then integrated using multimodal generative AI to generate a comprehensive user profile.

[1384] Analysis of individual data

[1385] The server analyzes the individual data in the following way:

[1386] Analyze the emotion recognition data obtained by the emotion engine.

[1387] Facial expression data is analyzed using a facial recognition algorithm.

[1388] A voice recognition algorithm is used to analyze tone of voice data.

[1389] Heart rate data is analyzed using a heart rate analysis algorithm.

[1390] Behavioral pattern data is analyzed using a behavioral prediction algorithm.

[1391] User Authentication

[1392] The server compares the generated user profile with existing user data in its internal database. Based on the comparison result, it determines whether the user is legitimate. If the user is legitimate, authentication is successful and a notification of this is sent to the terminal.

[1393] Notification of access permission to users

[1394] The terminal receives the authentication result from the server and, if successful, notifies the user of the access permission, allowing the user to log in to the metaverse. If the authentication fails, a corresponding warning message is displayed.

[1395] Specific examples

[1396] User logging into the metaverse scenario

[1397] 1. A user puts on a VR headset and attempts to access the Metaverse.

[1398] The user enters their ID and password and confirms the login screen.

[1399] 2. The device activates sensors and collects user data.

[1400] The device activates a facial recognition camera, heart rate sensor, microphone, emotion engine, and motion sensors to collect the user's facial expressions, heart rate, tone of voice, emotional data, and behavioral patterns.

[1401] 3. The device encrypts the data and sends it to the server.

[1402] The terminal encrypts the data with AES-256 and sends it to the server via the HTTPS protocol.

[1403] 4. The server receives and decrypts the data.

[1404] The server receives the data and decrypts it with AES-256.

[1405] 5. The server consolidates and analyzes the data.

[1406] The server performs emotion data recognized by the emotion engine, face recognition, voice recognition, heart rate analysis, and movement analysis to obtain the integrated results.

[1407] 6. The server authenticates the user against its user database.

[1408] The server checks its user database for a matching record.

[1409] 7. The server notifies the terminal of the authentication result.

[1410] The server notifies the terminal of successful authentication.

[1411] 8. The device notifies the user of the access permission.

[1412] The device displays the access permissions to the user and logs them into the metaverse.

[1413] In this way, the present invention can accurately and securely identify users and increase safety and engagement within the metaverse. The introduction of an emotion engine can more accurately grasp the user's psychological state, further strengthening security.

[1414] The processing flow will be explained below.

[1415] Step 1:

[1416] A user puts on a VR headset and attempts to access the Metaverse.

[1417] The user enters their ID and password on the login screen within the metaverse.

[1418] Step 2:

[1419] The device activates sensors and collects user data.

[1420] The device uses a facial recognition camera to collect facial expression data from the user.

[1421] The device uses a heart rate sensor to collect the user's heart rate data.

[1422] The terminal uses a microphone to collect data on the tone of the user's voice.

[1423] The device uses motion sensors to collect data on the user's behavioral patterns.

[1424] The terminal uses an emotion engine to collect the user's emotional response data.

[1425] Step 3:

[1426] The device encrypts the collected data.

[1427] The device uses the AES-256 encryption algorithm to encrypt collected facial expression data, heart rate data, tone of voice data, behavioral pattern data, and emotional response data.

[1428] Step 4:

[1429] The device sends the encrypted data to the server.

[1430] The device uses the HTTPS protocol to securely send encrypted data to the server.

[1431] Step 5:

[1432] The server decrypts the received data.

[1433] The server decrypts the received encrypted data using the AES-256 encryption algorithm.

[1434] Step 6:

[1435] The server classifies the decrypted data.

[1436] The server categorizes the decoded data into facial expression data, heart rate data, tone of voice data, behavioral pattern data, and emotional response data.

[1437] Step 7:

[1438] The server consolidates the data.

[1439] The server uses multimodal generative AI to integrate the categorized data and generate a comprehensive profile of the user.

[1440] Step 8:

[1441] The server analyzes the individual data.

[1442] The server analyzes the facial expression data using a facial recognition algorithm.

[1443] The server analyzes the tone of voice data using a voice recognition algorithm.

[1444] The server analyzes the heart rate data using a heart rate analysis algorithm.

[1445] The server analyzes the behavioral pattern data using a behavioral prediction algorithm.

[1446] The server uses an emotion engine to analyze the emotional response data.

[1447] Step 9:

[1448] The server authenticates the user based on the analysis results.

[1449] The server matches the generated user profile with existing data in its internal database.

[1450] Based on the result of the comparison, the server determines whether the user is legitimate.

[1451] Step 10:

[1452] The server notifies the terminal of the authentication result.

[1453] If the authentication is successful, the server notifies the terminal of access permission.

[1454] If the authentication fails, the server notifies the terminal of access denial.

[1455] Step 11:

[1456] The terminal notifies the user of the authentication result.

[1457] If the authentication is successful, the terminal allows the user to access the metaverse.

[1458] If the authentication fails, the terminal displays a warning message to the user.

[1459] In this way, the system combined with the emotion engine integrates and analyzes users' emotions, behaviors, and biometric data to achieve accurate and secure authentication, which can significantly improve safety and user engagement within the metaverse.

[1460] Example 2

[1461] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1462] In conventional systems, the process of collecting and authenticating users' personal information is complicated, and security is often insufficient. In particular, there is a lack of means to integrate and analyze multimodal data such as emotional and behavioral data, which leads to problems with the accuracy of user authentication. This leads to a poor user experience in metaverse and virtual reality environments, and safety is not guaranteed.

[1463] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1464] In this invention, the server includes means for collecting a user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, means for storing the collected data in a temporary storage device, means for encrypting the collected data using the AES-256 algorithm, means for transmitting the encrypted data to the server using a secure communication protocol, means for decrypting the received data using the AES-256 algorithm, means for classifying the decrypted data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral patterns, and biometric data and integrating them using multimodal generation AI, means for analyzing the integrated data, means for authenticating the user based on the analysis result, means for notifying the terminal of the authentication result, and means for providing access permission to the user based on the authentication result. This allows the process from collecting user data to granting access permission to be performed safely and efficiently, thereby improving user safety and engagement in the metaverse.

[1465] A "user" is a person who uses a client device to access the system and use the services.

[1466] "Emotional response" is data that indicates the user's psychological state, and is information that is analyzed in real time using an emotion engine.

[1467] "Facial expression" is data that indicates the state of the user's face, and is information collected by a face recognition camera.

[1468] "Heart rate" is data indicating the frequency of the user's heartbeat, and is information measured using a heart rate sensor.

[1469] "Voice tone" is data that indicates the quality and pitch of a user's voice when speaking, and is information collected using a microphone.

[1470] "Behavior patterns" are data that indicate the user's movements and behavioral tendencies, and are information captured using a movement sensor.

[1471] "Biometric data" refers to data related to the user's physical characteristics, including information such as heart rate and facial recognition data.

[1472] A "temporary storage device" is a storage device for temporarily storing collected data.

[1473] "AES-256 Algorithm" means an advanced encryption algorithm and technology used to encrypt data.

[1474] A "secure communication protocol" is a communication protocol for ensuring the safe transfer of data, such as HTTPS.

[1475] A "server" is a computer system that receives, decrypts, analyzes collected data, and performs user authentication.

[1476] "Emotion recognition data" is data relating to the user's emotional state, analyzed by the emotion engine.

[1477] "Facial expression data" is data relating to the state of the user's face, and is collected by a face recognition camera.

[1478] "Heart rate data" is data relating to the user's heart rate, and is acquired by a heart rate sensor.

[1479] "Voice tone data" is data relating to the quality and pitch of the user's voice, and is collected by a microphone.

[1480] "Behavior pattern data" is data relating to the user's movements and behaviors, and is captured by a movement sensor.

[1481] "Multimodal generative AI" is an artificial intelligence model for integrating and analyzing multiple different types of data.

[1482] "User authentication" is the process of verifying that a user is a legitimate user by comparing collected data with existing database information.

[1483] "Access permission" is a control that allows a user who has been successfully authenticated to access a system or service.

[1484] This invention relates to a system that safely and efficiently collects and analyzes personal data and performs accurate authentication when a user wears a VR headset and accesses the metaverse.

[1485] Hardware and software used

[1486] When a user puts on a VR headset, access to the Metaverse begins. At this time, the following hardware on the device collects data:

[1487] Facial recognition camera: Captures the user's facial expressions.

[1488] Heart rate sensor: Measures the user's heart rate.

[1489] Microphone: Collects the user's tone of voice.

[1490] Motion sensors: Capture user behavior patterns.

[1491] Emotion Engine: Analyzes users' emotional responses in real time.

[1492] This data is temporarily stored in the device's temporary storage, after which the device encrypts the data using the AES-256 algorithm and transmits it to the server via the secure HTTPS communication protocol.

[1493] The server does the following:

[1494] Decrypting received data: Decrypts encrypted data using the AES-256 algorithm.

[1495] Data Classification and Integration: Classify the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data using multimodal generative AI.

[1496] Data analysis: The server analyzes the data using various algorithms. Specifically, an emotion engine is used to analyze emotion recognition data, a face recognition algorithm is used to analyze facial expression data, and a voice recognition algorithm is used to analyze tone of voice data.

[1497] Perform user authentication: Based on the integrated data, the user is authenticated. This process involves checking against an existing user database.

[1498] Notification of authentication result: The server notifies the terminal of the authentication result.

[1499] Specific examples

[1500] For example, the following specific operations are performed.

[1501] 1. A user puts on a VR headset and attempts to access the Metaverse.

[1502] The user enters their ID and password and confirms the login screen.

[1503] 2. The device activates sensors and collects user data.

[1504] The device activates a facial recognition camera, heart rate sensor, microphone, emotion engine, and motion sensors to collect the user's facial expressions, heart rate, tone of voice, emotional data, and behavioral patterns.

[1505] 3. The device encrypts the data and sends it to the server.

[1506] The device encrypts the collected data with AES-256 and sends it to the server via the HTTPS protocol.

[1507] 4. The server receives and decrypts the data.

[1508] The server receives the data and decrypts it with AES-256.

[1509] 5. The server consolidates and analyzes the data.

[1510] The server performs emotion data recognized by the emotion engine, facial recognition, voice recognition, heart rate analysis, and movement analysis to obtain the integrated results.

[1511] 6. The server authenticates the user against its user database.

[1512] The server checks its user database for a matching record.

[1513] 7. The server notifies the terminal of the authentication result.

[1514] The server notifies the terminal of successful authentication.

[1515] 8. The device notifies the user of the access permission.

[1516] The device displays the access permissions to the user and logs them into the metaverse.

[1517] This system provides a secure and efficient authentication process, from collecting user data to granting access to the metaverse. An example prompt is as follows:

[1518] "Please describe in natural language a scenario in which a user logs into the Metaverse using a VR headset. Please specifically describe the process from user data collection to authentication and access authorization."

[1519] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1520] Step 1:

[1521] A user puts on a VR headset and attempts to access the Metaverse.

[1522] Specifically, this includes the user entering their ID and password to confirm the login screen, which generates an access request and triggers the device to start the authentication process.

[1523] Input: User login information (ID, password)

[1524] Output: Trigger to start the authentication process

[1525] Step 2:

[1526] The device activates sensors and collects user data.

[1527] Specifically, the device's built-in facial recognition camera, heart rate sensor, microphone, motion sensor, and emotion engine are all activated to collect the user's facial expressions, heart rate, tone of voice, behavioral patterns, and emotional data in real time.

[1528] Input: User's biometric information (face, heart rate, voice, movement, emotion)

[1529] Output: Collected bio-behavioral data

[1530] Step 3:

[1531] The terminal stores the collected data in temporary storage.

[1532] Specifically, this involves caching the collected data in temporary storage.

[1533] Input: Collected biometric and behavioral data

[1534] Output: Data stored in temporary storage

[1535] Step 4:

[1536] The device encrypts the collected data using the AES-256 algorithm.

[1537] Specifically, this involves automatically passing data stored in temporary storage to an encryption module where it is encrypted using AES-256.

[1538] Input: Data stored in temporary storage

[1539] Output: Encrypted data

[1540] Step 5:

[1541] The device sends encrypted data to the server using the secure HTTPS protocol.

[1542] Specifically, this includes operations in which encrypted data is transmitted using the HTTPS protocol.

[1543] Input: Encrypted data

[1544] Output: Send data to the server

[1545] Step 6:

[1546] The server receives the encrypted data and decrypts it using the AES-256 algorithm.

[1547] Specifically, the encrypted data is passed to the receiving module and decrypted using AES-256.

[1548] Input: Encrypted data

[1549] Output: Decrypted data

[1550] Step 7:

[1551] The server classifies the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data, and integrates them using multimodal generative AI.

[1552] Specifically, the decoded data is classified into categories, sent to the respective analysis modules, and an integration process is carried out by the multimodal generation AI.

[1553] Input: Decrypted data

[1554] Output: Unified user profile

[1555] Step 8:

[1556] The server analyzes the combined data and authenticates the user.

[1557] Specifically, this involves checking the integrated user profile against an internal database to determine legitimacy.

[1558] Input: Unified User Profile

[1559] Output: Authentication result

[1560] Step 9:

[1561] The server notifies the terminal of the authentication result.

[1562] Specifically, this includes an operation of transmitting the result of authentication success or failure to the terminal.

[1563] Input: Authentication result

[1564] Output: Authentication result notification to the terminal

[1565] Step 10:

[1566] The terminal notifies the user of the access permission.

[1567] Specifically, if authentication is successful, the terminal will display the access permission to the metaverse to the user, and if authentication fails, a corresponding warning message will be displayed.

[1568] Input: Authentication result notification to the terminal

[1569] Output: Allow access to user or warning message

[1570] (Application example 2)

[1571] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1572] Modern authentication systems have adopted a wide range of biometric and behavioral authentication methods for security reasons. However, because these methods rely on individual data, it is difficult to completely eliminate the risk of impersonation and unauthorized access. In particular, when using emotional or biometric data alone, they are susceptible to temporary changes in the user's state and environmental factors, making it difficult to obtain consistent authentication results. Furthermore, there are insufficient means to integrate these data and perform accurate and secure authentication based on a comprehensive user profile.

[1573] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, means for determining the legitimacy of the user based on a user profile generated using the collected biometric data and emotion recognition data, and means for providing access permission to the user based on the authentication result. This provides a consistent authentication result that integrates multiple data, eliminates impersonation and unauthorized access, and enables more secure and reliable authentication.

[1574] Below are definitions of important terms extracted from claim 1:

[1575] "User data" is a general term for information about a user, such as emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data.

[1576] "Encryption" refers to the process of transforming data using specific algorithms to make it unreadable to third parties.

[1577] A "server" is a computer system that stores, analyzes, and processes data over a network.

[1578] "Integration" refers to the act of bringing together multiple types of data and processing them in a consistent format or with consistent standards.

[1579] "Analysis" is the process of examining collected data in detail to find meaning and patterns.

[1580] "Authentication" is the process of verifying that a user has valid access rights.

[1581] A "terminal" is an electronic device that is directly operated by a user, and includes smartphones, personal computers, etc.

[1582] "Access permission" refers to granting access to a system or data.

[1583] "Biometric data" refers to information obtained from the user's body, such as heart rate, fingerprints, and facial shape.

[1584] "Emotion recognition data" is data used to analyze a user's mental state and emotions.

[1585] A "user profile" is a consistent set of information about a user that is obtained by integrating various collected data.

[1586] The present invention provides an advanced authentication system by collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, and integrating and analyzing this data. Specific embodiments for carrying out the present invention will be described below.

[1587] System Configuration

[1588] The system consists of the following main components:

[1589] Terminal: A device that is directly used by a user, such as a smartphone, smart glasses, a head-mounted display, or a robot.

[1590] Sensors: Face recognition camera, heart rate sensor, microphone, and motion sensor installed on the device.

[1591] Server: A computer system that stores, analyzes, and integrates data.

[1592] Communication Protocol: A secure communication protocol (e.g. HTTPS) for sending and receiving data.

[1593] Encryption algorithm: The encryption method used to ensure secure transmission of data (e.g., AES-256).

[1594] Emotion Engine: An API for analyzing emotions from collected data.

[1595] Program processing overview

[1596] When a user attempts to log in using a device, the device first collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using a facial recognition camera, heart rate sensor, microphone, and motion sensors. Using an emotion engine, the device analyzes the collected data in real time to generate emotional data. This data is then stored in temporary memory.

[1597] Next, the device encrypts the collected data with an encryption algorithm (e.g., AES-256) and sends it to the server using a secure communication protocol (e.g., HTTPS). The server receives and decrypts the encrypted data.

[1598] The server analyzes the decoded data using an emotion engine, face recognition algorithm, voice recognition algorithm, heart rate analysis algorithm, and behavior prediction algorithm to classify it into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavior pattern data, and biometric data. A comprehensive user profile is generated from the integrated data.

[1599] Based on the generated user profile, the server compares it with existing user data in its internal database to determine the user's legitimacy. Based on the comparison result, the server notifies the terminal of the authentication result, and the terminal displays the result to the user. If authentication is successful, the user is granted access permission.

[1600] Specific examples of hardware and software used

[1601] Device: Compatible smartphones and head-mounted displays.

[1602] Emotion engines: for example, Affectiva or Microsoft Azure Face API.

[1603] Encryption algorithm: AES-256.

[1604] Communication protocol: HTTPS.

[1605] Analysis algorithms: Multimodal generative AI, OpenCV (face recognition), Google Cloud Speech-to-Text (voice recognition).

[1606] Specific examples

[1607] When a user launches the security app, the app uses the smartphone's camera to capture the user's facial expressions, the heart rate sensor to obtain their heart rate, the microphone to record audio and analyze their tone of voice, and the motion sensor to record their behavioral patterns. This data is then analyzed in real time as emotional data using an emotion engine. The data is then encrypted with AES-256 and sent to a secure server via HTTPS. The server receives, decrypts, and analyzes the data to generate a comprehensive user profile. The server then authenticates the user and notifies the smartphone of the authentication result.

[1608] Prompt Sentence Examples

[1609] Here are some example prompts to input to a generative AI model:

[1610] Describe a system that collects a user's facial recognition, heart rate, tone of voice, behavioral patterns, and biometric data, and then uses an emotion engine to analyze the emotional data in real time. This data is integrated to generate a comprehensive user profile and perform user authentication.

[1611] In this way, the present invention integrates data from multiple data sources and provides a high level of security, thereby improving the reliability and accuracy of authentication.

[1612] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1613] Step 1: Collect user data

[1614] When a user attempts to log in using a device, the device activates various sensors, including a facial recognition camera, heart rate sensor, microphone, and motion sensor, which are used to collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. The input is the user's biometric data, and the output is a set of collected biometric data.

[1615] Step 2: Analyze the emotion data

[1616] The collected biometric data is analyzed in real time into emotional data using an emotion engine, where each data point is mapped to an emotional state. The input is the collected biometric data, and the output is the analyzed emotional data.

[1617] Step 3: Temporarily save data

[1618] The terminal stores the collected data and analyzed emotional data in a temporary memory. The input of this process is the analyzed emotional data and biometric data, and the output is the set of data in the temporary memory.

[1619] Step 4: Encrypt the data

[1620] The terminal encrypts the data in temporary memory using an encryption algorithm (e.g., AES-256). The input is the data stored in temporary memory, and the output is the encrypted data.

[1621] Step 5: Sending data

[1622] The encrypted data is sent to the server using a secure communication protocol (e.g., HTTPS). The input is the encrypted data and the output is the data sent to the server.

[1623] Step 6: Receiving and Decrypting Data

[1624] The server receives the encrypted data and decrypts it with a decryption algorithm (e.g., AES-256). The input is the encrypted data and the output is the decrypted data.

[1625] Step 7: Classify the data

[1626] The server classifies the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data. The input to this process is the decoded data, and the output is the various classified data.

[1627] Step 8: Analyze the data

[1628] The various classified data are analyzed using an emotion engine, face recognition algorithm, voice recognition algorithm, heart rate analysis algorithm, and behavior prediction algorithm. The input is the classified data, and the output is the analysis results.

[1629] Step 9: Generate an Integration Profile

[1630] The server integrates the analysis results and generates a comprehensive user profile. The input is the analyzed data, and the output is the integrated user profile.

[1631] Step 10: User Authentication

[1632] The server compares the generated user profile with existing user data in its internal database to determine its validity. The input is the integrated user profile, and the output is the authentication result.

[1633] Step 11: Notification of authentication results

[1634] The server sends the authentication result to the terminal, and the terminal notifies the user of the result. The input is the authentication result, and the output is the notification to the user.

[1635] Step 12: Access Permissions

[1636] If the authentication is successful, the terminal grants the user access permission. The input is the authentication result, and the output is the granting of access permission.

[1637] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1638] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1639] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1640] [Fourth embodiment]

[1641] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1642] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1643] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1644] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1645] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1646] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1647] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1648] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1649] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1650] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1651] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1652] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1653] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1654] Explaining program processing in natural language

[1655] Collection and transmission of user data

[1656] When a user puts on a VR headset and attempts to access the Metaverse, the device collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using facial recognition cameras, heart rate sensors, microphones, etc. The collected data is then stored in temporary memory by the device.

[1657] Data encryption and transmission

[1658] The device encrypts the collected data using an encryption algorithm (e.g., AES-256), and then transmits the encrypted data to the server using a secure communication protocol (e.g., HTTPS).

[1659] Data reception and analysis

[1660] The server receives and decrypts the encrypted data sent from the device. The decrypted data is separated into different categories (emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data). This data is then integrated using multimodal generative AI to generate a comprehensive user profile.

[1661] Analysis of individual data

[1662] The server analyzes the individual data in the following way:

[1663] Emotional responses and facial expressions are run through a facial recognition algorithm.

[1664] The tone of voice is analyzed using a voice recognition algorithm.

[1665] The heart rate is analyzed by the biosensor's data analysis algorithm.

[1666] The behavioral patterns are analyzed based on a motion prediction algorithm.

[1667] User Authentication

[1668] The server compares the generated user profile with existing user data in its internal database. Based on the comparison result, it determines whether the user is legitimate. If the user is legitimate, authentication is successful and a notification of this is sent to the terminal.

[1669] Notification of access permission to users

[1670] The terminal receives the authentication result from the server and, if successful, notifies the user of the access permission, allowing the user to log in to the metaverse. If the authentication fails, a corresponding warning message is displayed.

[1671] Specific examples

[1672] User logging into the metaverse scenario

[1673] 1. A user puts on a VR headset and attempts to access the Metaverse.

[1674] The user enters their ID and password and confirms the login screen.

[1675] 2. The device activates sensors and collects user data.

[1676] The device will enable a facial recognition camera, heart rate sensor, and microphone to collect the user's facial expressions, heart rate, and voice data.

[1677] 3. The device encrypts the data and sends it to the server.

[1678] The terminal encrypts the data with AES-256 and sends it to the server via the HTTPS protocol.

[1679] 4. The server receives and decrypts the data.

[1680] The server receives the data and decrypts it with AES-256.

[1681] 5. The server consolidates and analyzes the data.

[1682] The server performs facial recognition, voice recognition, and heart rate analysis to obtain an integrated result.

[1683] 6. The server authenticates the user against its user database.

[1684] The server checks its user database for a matching record.

[1685] 7. The server notifies the terminal of the authentication result.

[1686] The server notifies the terminal of successful authentication.

[1687] 8. The device notifies the user of the access permission.

[1688] The device displays the access permissions to the user and logs them into the metaverse.

[1689] In this way, the present invention can accurately and securely identify users, increasing safety and engagement within the metaverse.

[1690] The processing flow will be explained below.

[1691] Step 1:

[1692] A user puts on a VR headset and attempts to access the Metaverse.

[1693] The user enters their ID and password and proceeds to the login screen.

[1694] Step 2:

[1695] The device activates sensors and collects user data.

[1696] The device uses a facial recognition camera to collect facial expression data from the user.

[1697] The device uses a heart rate sensor to collect the user's heart rate data.

[1698] The terminal uses a microphone to collect data on the tone of the user's voice.

[1699] The device uses motion sensors to collect data on the user's behavioral patterns.

[1700] Step 3:

[1701] The device encrypts the collected data.

[1702] The device uses the AES-256 encryption algorithm to encrypt collected facial expression data, heart rate data, tone of voice data, and behavioral pattern data.

[1703] Step 4:

[1704] The device sends the encrypted data to the server.

[1705] The device uses the HTTPS protocol to securely send encrypted data to the server.

[1706] Step 5:

[1707] The server decrypts the received data.

[1708] The server decrypts the received encrypted data using the AES-256 encryption algorithm.

[1709] Step 6:

[1710] The server consolidates the decrypted data.

[1711] The server integrates facial recognition data, heart rate data, tone of voice data, and behavioral pattern data using multimodal generation AI to generate a user profile.

[1712] Step 7:

[1713] The server analyzes the individual data.

[1714] The server analyzes the facial expression data using a facial recognition algorithm.

[1715] The server analyzes the tone of voice data using a voice recognition algorithm.

[1716] The server analyzes the heart rate data using a heart rate analysis algorithm.

[1717] The server analyzes the behavioral pattern data using a behavioral prediction algorithm.

[1718] Step 8:

[1719] The server authenticates the user based on the analysis results.

[1720] The server matches the generated user profile with existing data in its internal database.

[1721] Based on the result of the comparison, the server determines whether the user is legitimate.

[1722] Step 9:

[1723] The server notifies the terminal of the authentication result.

[1724] If the authentication is successful, the server notifies the terminal of access permission.

[1725] If the authentication fails, the server notifies the terminal of access denial.

[1726] Step 10:

[1727] The terminal notifies the user of the authentication result.

[1728] If the authentication is successful, the terminal allows the user to access the metaverse.

[1729] If the authentication fails, the terminal displays a warning message to the user.

[1730] Example 1

[1731] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1732] Conventional virtual reality (VR) systems have faced problems with insufficiently secure user authentication methods, resulting in a high risk of easy unauthorized access. Furthermore, there was a lack of a means for centralized and detailed analysis of individual user data, making it difficult to improve the quality of the user experience. Furthermore, there was no established secure method for transferring collected data, creating a risk of data leakage. Therefore, providing a highly accurate and secure authentication system that users could use with confidence was a challenge.

[1733] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1734] In this invention, the server includes means for collecting emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using a facial recognition camera, heart rate sensor, and microphone when a user wears a VR headset and attempts to access the metaverse, means for encrypting the collected data using the AES-256 algorithm and transmitting it to the server via HTTPS protocol, and server means for decrypting the received data and integrating and analyzing the emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using multimodal generation AI, thereby enabling secure transfer of the collected data and highly accurate user authentication.

[1735] "User" refers to an individual who wears a VR headset and accesses the Metaverse.

[1736] "Device" refers to the device including the VR headset worn by the user and its related equipment.

[1737] "Server" refers to a computer on the system side that receives data sent from a user and analyzes and authenticates it.

[1738] A "face recognition camera" refers to a photographic device that captures a user's facial expressions and features.

[1739] "Heart rate sensor" refers to a detection device for measuring a user's heart rate.

[1740] "Microphone" refers to a recording device for collecting the user's voice.

[1741] "Emotional response" refers to changes in the emotions shown by the user, and is determined from facial expressions, tone of voice, etc.

[1742] "Facial expressions" refer to movements that show emotions and reactions on the user's face.

[1743] "Heart rate" refers to the number of times a user's heart beats within a certain period of time.

[1744] "Tone of voice" refers to characteristics such as the pitch and intonation of a user's voice.

[1745] "Behavioral patterns" refer to a series of actions and tendencies of a user's reactions.

[1746] "Biometric data" refers to various information collected from a user's body (e.g., heart rate, pulse, body temperature).

[1747] The "AES-256 algorithm" is one of the Advanced Encryption Standards for encrypting data, and uses a 256-bit key.

[1748] "HTTPS protocol" refers to a communications protocol for securely transferring data over the Internet.

[1749] "Multimodal generative AI" refers to an artificial intelligence model that integrates and analyzes multiple data modalities (e.g., audio, images, text).

[1750] "Encryption" refers to the process of transforming data so that it cannot be read by third parties.

[1751] "Decryption" refers to the process of restoring encrypted data to its original state.

[1752] "User profile" refers to a collection of information about an individual user that is generated by integrating various collected data.

[1753] "Authentication" refers to the process of verifying that a user is a specific individual.

[1754] "Access permission" refers to granting a user the right to access a particular system or information.

[1755] "Warning message" refers to the notification displayed to the user when authentication fails.

[1756] This invention is a system for accurately and securely authenticating users when they wear a VR headset and access the Metaverse. This system works through several steps, mainly involving data collection, encryption, transmission, reception, analysis, and authentication processes between the device, server, and user.

[1757] User Data Collection

[1758] When a user puts on a VR headset and starts an application, the device first activates sensors, including a facial recognition camera, heart rate sensor, and microphone. These devices collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. For example, the facial recognition camera captures facial expression data, the heart rate sensor monitors heart rate, and the microphone records tone of voice.

[1759] Data encryption and transmission

[1760] The collected data is temporarily stored on the device and then encrypted using the AES-256 encryption algorithm, and then transmitted to the server using the HTTPS protocol. This process ensures the security of the data.

[1761] Data analysis on the server

[1762] Once the data arrives at the server, it is first decoded. After decoding, the data is separated into categories (emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data). The server then uses multimodal generative AI to integrate and analyze this data. This results in a comprehensive user profile, which includes, for example, current emotional state, heart rate trends, and usual behavioral patterns.

[1763] User Authentication

[1764] The generated user profile is compared with existing user data stored in the server's internal database. This comparison determines whether the user has valid access rights. If authentication is successful, the server notifies the terminal of the result. If authentication fails, the terminal displays a warning message to the user.

[1765] User notification and access permissions

[1766] The terminal receives the authentication result from the server and notifies the user of the access permission if successful, allowing the user to log in to the metaverse and access interactive content. If the authentication fails, a corresponding warning message will be displayed to the user, prompting them to try again.

[1767] The system will improve the user experience and ensure safety within the metaverse by collecting, analyzing, and authenticating user data with high accuracy and security.

[1768] Prompt Sentence Examples

[1769] Finally, the following are some specific examples of prompts for generative AI models:

[1770] "Please explain in natural language the system that collects biometric data from users using a VR headset and analyzes it to authenticate the user."

[1771] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1772] Step 1:

[1773] The user puts on a VR headset and prepares to access the Metaverse.

[1774] The user turns on the VR headset, launches the application, and proceeds to the login screen where they enter their ID and password.

[1775] Input: User ID and password

[1776] Output: Access to login screen

[1777] Step 2:

[1778] The device activates sensors and collects user data.

[1779] The device activates a facial recognition camera, heart rate sensor and microphone to simultaneously collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns and biometric data.

[1780] Input: User's current biometric and behavioral data

[1781] Data processing / data calculation: Analyzes images taken with a facial recognition camera and extracts facial expression data. Records heart rate measured with a heart rate sensor. Analyzes audio data collected with a microphone and identifies voice tone.

[1782] Output: Multiple types of biometric data collected

[1783] Step 3:

[1784] Encrypt the data collected by the device.

[1785] The device encrypts all collected data using the AES-256 encryption algorithm.

[1786] Input: Unencrypted biometric data

[1787] Data processing / data calculation: Encrypt data using the AES-256 algorithm

[1788] Output: Encrypted biometric data

[1789] Step 4:

[1790] The encrypted data is sent to the server.

[1791] The device uses the HTTPS protocol to securely transmit encrypted data to the server.

[1792] Input: Encrypted biometric data

[1793] Output: Send data to the server

[1794] Step 5:

[1795] The server receives and decrypts the data.

[1796] The server receives the data using the HTTPS protocol and decrypts it using the AES-256 algorithm.

[1797] Input: Encrypted biometric data

[1798] Data processing / data calculation: Receive data using HTTPS protocol and decrypt it using AES-256 algorithm

[1799] Output: Decoded biometric data

[1800] Step 6:

[1801] The server categorizes and analyzes the data.

[1802] The server separates the decoded data into emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, and then uses multimodal generative AI to integrate and analyze this data.

[1803] Input: Decoded biometric data

[1804] Data processing / data calculation: Categorize data and perform integrated analysis using multimodal generation AI

[1805] Output: Comprehensive user profile

[1806] Step 7:

[1807] The server checks the user profile against an internal database.

[1808] The server compares the generated user profile with existing user data in its internal database and authenticates the user based on the comparison result.

[1809] Input: User profile and existing user database

[1810] Data processing / data calculation: Compare and collate profile data with existing databases

[1811] Output: Authentication result (success or failure)

[1812] Step 8:

[1813] The authentication result is notified to the terminal, and access permission is provided to the user.

[1814] The server sends the authentication result (success or failure) to the terminal. If the authentication is successful, the terminal notifies the user of the access permission and allows them to log in to the metaverse. If the authentication fails, the terminal displays a warning message.

[1815] Input: Authentication result

[1816] Output: Notification of permission or warning message

[1817] As described above, this system collects user data and securely analyzes and authenticates it, enabling accurate and secure user authentication.

[1818] (Application example 1)

[1819] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1820] In recent years, the importance of security authentication in companies has increased, and traditional authentication using IDs and passwords is often insufficient. In particular, more advanced authentication systems are required to prevent unauthorized employee access and data leaks. However, currently used biometric authentication systems have room for improvement in terms of user experience and authentication accuracy. For these reasons, there is a demand for more secure and accurate authentication methods in corporate security authentication systems.

[1821] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1822] In this invention, the server includes means for collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, means for encrypting the collected data and transmitting it to the server, server means for integrating and analyzing the received data, server means for authenticating the user based on the analysis result, means for notifying the terminal of the authentication result, means for providing the user with access permission based on the authentication result, and means for collecting, analyzing, and authenticating user data using a VR headset in a corporate security authentication system. This enables companies to achieve more accurate and secure security authentication and reduce the risk of unauthorized access and data leakage.

[1823] "User" refers to a person who uses the system and is an individual or employee who is subject to security authentication.

[1824] "Emotional responses" are information that indicates a user's mental state, and collected data includes facial expressions and vocal tone.

[1825] "Facial expressions" are signs of emotion that appear on a user's face and are part of the data collected using a facial recognition camera.

[1826] "Heart rate" refers to the number of times the heart beats within a certain period of time and is part of the biometric data collected using biometric sensors.

[1827] "Tone of voice" refers to the quality and pitch of the user's voice and is part of the data collected using a microphone.

[1828] "Behavioral patterns" are part of the data collected by sensors and indicate characteristics of a user's movements and behavior.

[1829] "Biometric data" refers to information that indicates a user's physical characteristics, and refers to a wide range of data including heart rate, tone of voice, etc.

[1830] "Encryption" refers to the process of transforming collected data using specific algorithms to protect it.

[1831] "Server" refers to a central computer system that receives collected data, analyzes it, and manages authentication results.

[1832] "Analysis" refers to the process of synthesizing collected data and analyzing the information to authenticate a user.

[1833] "Authentication" refers to the process of verifying a user's identity and granting access rights based on collected data.

[1834] "Access permission" refers to allowing a user who has been successfully authenticated to use a specific system or service.

[1835] A "VR headset" is a device for experiencing virtual reality and refers to a terminal used to collect user data.

[1836] "Secure communications protocol" refers to a communications protocol used to encrypt and protect data transmissions.

[1837] The present invention relates to a method and system for enhancing a company's security authentication system by using a VR headset worn by a user to collect, analyze, and authenticate biometric and behavioral data to enhance security authentication.

[1838] System configuration

[1839] Hardware and Software Configuration

[1840] 1. VR headset (device)

[1841] Equipment: VR headsets are equipped with sensors to collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data.

[1842] Sensors: Facial recognition camera, heart rate sensor, microphone.

[1843] 2. Server

[1844] Function: Receives collected data, decrypts encrypted data, and has the processing power to perform analysis.

[1845] Software: Multimodal generative AI (e.g., OpenAI GPT-4), facial recognition algorithms (e.g., OpenCV), speech recognition algorithms (e.g., Google Cloud Speech-to-Text), and biometric data analysis algorithms.

[1846] 3. Communication Protocol

[1847] Protocol: A secure communication protocol (e.g. HTTPS) is used to send and receive data.

[1848] Program processing flow

[1849] 1. Collection of User Data

[1850] When a user puts on a VR headset to access a company's authentication system, sensors are activated to collect emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, which are then temporarily stored in memory.

[1851] 2. Data Encryption and Transmission

[1852] The collected data is encrypted using the AES-256 encryption algorithm and transmitted to the server using the HTTPS protocol.

[1853] 3. Data Receipt and Analysis

[1854] The server decrypts the received data and analyzes it in the following way:

[1855] Emotional responses and facial expressions are analyzed using a facial recognition algorithm (OpenCV).

[1856] The tone of voice is analyzed using a voice recognition algorithm (Google Cloud Speech-to-Text).

[1857] Heart rate is analyzed using biometric data analysis algorithms.

[1858] Behavioral patterns are integrated and analyzed using a multimodal generative AI model.

[1859] 4. User Authentication

[1860] Based on the parsed data, the server authenticates the user by matching the user profile with existing data in a database.

[1861] 5. Notification of authentication results and access permission

[1862] The server notifies the terminal of the authentication result, and if the authentication is successful, the user is granted access permission, ensuring the prevention of unauthorized access and the management of users.

[1863] Examples and prompts

[1864] Examples:

[1865] When the installed VR headset is used, it attempts to access the company's security authentication system.

[1866] For example: "Put on the VR headset to access the company's authentication system. Once you put it on, the sensors will automatically activate and collect data."

[1867] Example prompt sentence:

[1868] "Please put on your Oculus Quest and log in to the corporate authentication app. After logging in to the app, sensors will be activated to collect your emotional response, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. Based on this, you will be authenticated, so please just wait."

[1869] This process enables companies to achieve highly accurate and reliable security authentication, reducing the risk of unauthorized access and data leakage.

[1870] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1871] Step 1:

[1872] A user puts on a VR headset and accesses a company's authentication system.

[1873] Input: A user puts on a VR headset and attempts to log into the authentication system.

[1874] Action: A user launches a VR app and visits the login page.

[1875] Step 2:

[1876] The device activates sensors and collects user data.

[1877] Input: The sensors activated by the VR headset (face recognition camera, heart rate sensor, microphone).

[1878] Data processing: The device collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data and stores them in temporary memory.

[1879] How it works: A facial recognition camera captures your facial expressions, a heart rate sensor measures your heart rate, and a microphone records your tone of voice.

[1880] Step 3:

[1881] The terminal encrypts the collected user data and sends it to the server.

[1882] Input: Collected user data (facial expression data, heart rate data, voice data).

[1883] Data operation: The device encrypts the data using the AES-256 encryption algorithm.

[1884] Output: Encrypted user data.

[1885] How it works: The device sends encrypted data to the server using the HTTPS protocol.

[1886] Step 4:

[1887] The server receives the encrypted data and decrypts it.

[1888] Input: Encrypted user data from the device.

[1889] Data Calculation: The server decrypts the data using the AES-256 algorithm.

[1890] Output: Decrypted user data.

[1891] How it works: The server decrypts the data it receives using the HTTPS protocol.

[1892] Step 5:

[1893] The server analyzes and integrates various data.

[1894] Input: Decoded user data (facial expression data, heart rate data, voice data).

[1895] Data processing: The server analyzes the individual data using facial recognition algorithms (e.g., OpenCV), voice recognition algorithms (e.g., Google Cloud Speech-to-Text), and biometric data analysis algorithms, and then synthesizes the data using a generative AI model.

[1896] Output: A comprehensive user profile.

[1897] How it works: The server analyzes emotional responses and facial expressions using facial recognition algorithms, tone of voice using speech recognition algorithms, and heart rate using biometric data analysis algorithms.

[1898] Step 6:

[1899] The server authenticates the user based on their profile.

[1900] Input: A pooled and analyzed user profile.

[1901] Data calculation: The server checks the data against the user database and performs authentication.

[1902] Output: Authentication result (success / failure).

[1903] How it works: The server checks the generated user profile against its existing user database to determine if there is a match.

[1904] Step 7:

[1905] The server notifies the terminal of the authentication result.

[1906] Input: Authentication result (success / failure).

[1907] Output: A signal notifying the authentication result.

[1908] Operation: The server sends the authentication result to the terminal.

[1909] Step 8:

[1910] The terminal displays the authentication result to the user and provides access permission.

[1911] Input: Authentication result from the server.

[1912] Output: Notification to user (access permission / warning message).

[1913] Behavior: If the device is successfully authenticated, it displays the access permission and allows you to log in to the metaverse. If authentication fails, it displays a warning message.

[1914] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1915] Explaining program processing in natural language

[1916] Collection and transmission of user data

[1917] When a user puts on a VR headset and attempts to access the Metaverse, the device first collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. This collection is done using a facial recognition camera, heart rate sensor, microphone, and motion sensors. The device also uses an emotion engine to recognize and collect the user's emotional data in real time. This data is then stored in temporary memory.

[1918] Data encryption and transmission

[1919] The device encrypts the collected data using an encryption algorithm (e.g., AES-256), and the encrypted data is sent to the server using a secure communication protocol (e.g., HTTPS).

[1920] Data reception and analysis

[1921] The server receives and decrypts the encrypted data sent from the device. The decrypted data is categorized into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data, including emotional data obtained from the emotion engine. This data is then integrated using multimodal generative AI to generate a comprehensive user profile.

[1922] Analysis of individual data

[1923] The server analyzes the individual data in the following way:

[1924] Analyze the emotion recognition data obtained by the emotion engine.

[1925] Facial expression data is analyzed using a facial recognition algorithm.

[1926] A voice recognition algorithm is used to analyze tone of voice data.

[1927] Heart rate data is analyzed using a heart rate analysis algorithm.

[1928] Behavioral pattern data is analyzed using a behavioral prediction algorithm.

[1929] User Authentication

[1930] The server compares the generated user profile with existing user data in its internal database. Based on the comparison result, it determines whether the user is legitimate. If the user is legitimate, authentication is successful and a notification of this is sent to the terminal.

[1931] Notification of access permission to users

[1932] The terminal receives the authentication result from the server and, if successful, notifies the user of the access permission, allowing the user to log in to the metaverse. If the authentication fails, a corresponding warning message is displayed.

[1933] Specific examples

[1934] User logging into the metaverse scenario

[1935] 1. A user puts on a VR headset and attempts to access the Metaverse.

[1936] The user enters their ID and password and confirms the login screen.

[1937] 2. The device activates sensors and collects user data.

[1938] The device activates a facial recognition camera, heart rate sensor, microphone, emotion engine, and motion sensors to collect the user's facial expressions, heart rate, tone of voice, emotional data, and behavioral patterns.

[1939] 3. The device encrypts the data and sends it to the server.

[1940] The terminal encrypts the data with AES-256 and sends it to the server via the HTTPS protocol.

[1941] 4. The server receives and decrypts the data.

[1942] The server receives the data and decrypts it with AES-256.

[1943] 5. The server consolidates and analyzes the data.

[1944] The server performs emotion data recognized by the emotion engine, face recognition, voice recognition, heart rate analysis, and movement analysis to obtain the integrated results.

[1945] 6. The server authenticates the user against its user database.

[1946] The server checks its user database for a matching record.

[1947] 7. The server notifies the terminal of the authentication result.

[1948] The server notifies the terminal of successful authentication.

[1949] 8. The device notifies the user of the access permission.

[1950] The device displays the access permissions to the user and logs them into the metaverse.

[1951] In this way, the present invention can accurately and securely identify users and increase safety and engagement within the metaverse. The introduction of an emotion engine can more accurately grasp the user's psychological state, further strengthening security.

[1952] The processing flow will be explained below.

[1953] Step 1:

[1954] A user puts on a VR headset and attempts to access the Metaverse.

[1955] The user enters their ID and password on the login screen within the metaverse.

[1956] Step 2:

[1957] The device activates sensors and collects user data.

[1958] The device uses a facial recognition camera to collect facial expression data from the user.

[1959] The device uses a heart rate sensor to collect the user's heart rate data.

[1960] The terminal uses a microphone to collect data on the tone of the user's voice.

[1961] The device uses motion sensors to collect data on the user's behavioral patterns.

[1962] The terminal uses an emotion engine to collect the user's emotional response data.

[1963] Step 3:

[1964] The device encrypts the collected data.

[1965] The device uses the AES-256 encryption algorithm to encrypt collected facial expression data, heart rate data, tone of voice data, behavioral pattern data, and emotional response data.

[1966] Step 4:

[1967] The device sends the encrypted data to the server.

[1968] The device uses the HTTPS protocol to securely send encrypted data to the server.

[1969] Step 5:

[1970] The server decrypts the received data.

[1971] The server decrypts the received encrypted data using the AES-256 encryption algorithm.

[1972] Step 6:

[1973] The server classifies the decrypted data.

[1974] The server categorizes the decoded data into facial expression data, heart rate data, tone of voice data, behavioral pattern data, and emotional response data.

[1975] Step 7:

[1976] The server consolidates the data.

[1977] The server uses multimodal generative AI to integrate the categorized data and generate a comprehensive profile of the user.

[1978] Step 8:

[1979] The server analyzes the individual data.

[1980] The server analyzes the facial expression data using a facial recognition algorithm.

[1981] The server analyzes the tone of voice data using a voice recognition algorithm.

[1982] The server analyzes the heart rate data using a heart rate analysis algorithm.

[1983] The server analyzes the behavioral pattern data using a behavioral prediction algorithm.

[1984] The server uses an emotion engine to analyze the emotional response data.

[1985] Step 9:

[1986] The server authenticates the user based on the analysis results.

[1987] The server matches the generated user profile with existing data in its internal database.

[1988] Based on the result of the comparison, the server determines whether the user is legitimate.

[1989] Step 10:

[1990] The server notifies the terminal of the authentication result.

[1991] If the authentication is successful, the server notifies the terminal of access permission.

[1992] If the authentication fails, the server notifies the terminal of access denial.

[1993] Step 11:

[1994] The terminal notifies the user of the authentication result.

[1995] If the authentication is successful, the terminal allows the user to access the metaverse.

[1996] If the authentication fails, the terminal displays a warning message to the user.

[1997] In this way, the system combined with the emotion engine integrates and analyzes users' emotions, behaviors, and biometric data to achieve accurate and secure authentication, which can significantly improve safety and user engagement within the metaverse.

[1998] Example 2

[1999] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2000] In conventional systems, the process of collecting and authenticating users' personal information is complicated, and security is often insufficient. In particular, there is a lack of means to integrate and analyze multimodal data such as emotional and behavioral data, which leads to problems with the accuracy of user authentication. This leads to a poor user experience in metaverse and virtual reality environments, and safety is not guaranteed.

[2001] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2002] In this invention, the server includes means for collecting a user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, means for storing the collected data in a temporary storage device, means for encrypting the collected data using the AES-256 algorithm, means for transmitting the encrypted data to the server using a secure communication protocol, means for decrypting the received data using the AES-256 algorithm, means for classifying the decrypted data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral patterns, and biometric data and integrating them using multimodal generation AI, means for analyzing the integrated data, means for authenticating the user based on the analysis result, means for notifying the terminal of the authentication result, and means for providing access permission to the user based on the authentication result. This allows the process from collecting user data to granting access permission to be performed safely and efficiently, thereby improving user safety and engagement in the metaverse.

[2003] A "user" is a person who uses a client device to access the system and use the services.

[2004] "Emotional response" is data that indicates the user's psychological state, and is information that is analyzed in real time using an emotion engine.

[2005] "Facial expression" is data that indicates the state of the user's face, and is information collected by a face recognition camera.

[2006] "Heart rate" is data indicating the frequency of the user's heartbeat, and is information measured using a heart rate sensor.

[2007] "Voice tone" is data that indicates the quality and pitch of a user's voice when speaking, and is information collected using a microphone.

[2008] "Behavior patterns" are data that indicate the user's movements and behavioral tendencies, and are information captured using a movement sensor.

[2009] "Biometric data" refers to data related to the user's physical characteristics, including information such as heart rate and facial recognition data.

[2010] A "temporary storage device" is a storage device for temporarily storing collected data.

[2011] "AES-256 Algorithm" means an advanced encryption algorithm and technology used to encrypt data.

[2012] A "secure communication protocol" is a communication protocol for ensuring the safe transfer of data, such as HTTPS.

[2013] A "server" is a computer system that receives, decrypts, analyzes collected data, and performs user authentication.

[2014] "Emotion recognition data" is data relating to the user's emotional state, analyzed by the emotion engine.

[2015] "Facial expression data" is data relating to the state of the user's face, and is collected by a face recognition camera.

[2016] "Heart rate data" is data relating to the user's heart rate, and is acquired by a heart rate sensor.

[2017] "Voice tone data" is data relating to the quality and pitch of the user's voice, and is collected by a microphone.

[2018] "Behavior pattern data" is data relating to the user's movements and behaviors, and is captured by a movement sensor.

[2019] "Multimodal generative AI" is an artificial intelligence model for integrating and analyzing multiple different types of data.

[2020] "User authentication" is the process of verifying that a user is a legitimate user by comparing collected data with existing database information.

[2021] "Access permission" is a control that allows a user who has been successfully authenticated to access a system or service.

[2022] This invention relates to a system that safely and efficiently collects and analyzes personal data and performs accurate authentication when a user wears a VR headset and accesses the metaverse.

[2023] Hardware and software used

[2024] When a user puts on a VR headset, access to the Metaverse begins. At this time, the following hardware on the device collects data:

[2025] Facial recognition camera: Captures the user's facial expressions.

[2026] Heart rate sensor: Measures the user's heart rate.

[2027] Microphone: Collects the user's tone of voice.

[2028] Motion sensors: Capture user behavior patterns.

[2029] Emotion Engine: Analyzes users' emotional responses in real time.

[2030] This data is temporarily stored in the device's temporary storage, after which the device encrypts the data using the AES-256 algorithm and transmits it to the server via the secure HTTPS communication protocol.

[2031] The server does the following:

[2032] Decrypting received data: Decrypts encrypted data using the AES-256 algorithm.

[2033] Data Classification and Integration: Classify the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data using multimodal generative AI.

[2034] Data analysis: The server analyzes the data using various algorithms. Specifically, an emotion engine is used to analyze emotion recognition data, a face recognition algorithm is used to analyze facial expression data, and a voice recognition algorithm is used to analyze tone of voice data.

[2035] Perform user authentication: Based on the integrated data, the user is authenticated. This process involves checking against an existing user database.

[2036] Notification of authentication result: The server notifies the terminal of the authentication result.

[2037] Specific examples

[2038] For example, the following specific operations are performed.

[2039] 1. A user puts on a VR headset and attempts to access the Metaverse.

[2040] The user enters their ID and password and confirms the login screen.

[2041] 2. The device activates sensors and collects user data.

[2042] The device activates a facial recognition camera, heart rate sensor, microphone, emotion engine, and motion sensors to collect the user's facial expressions, heart rate, tone of voice, emotional data, and behavioral patterns.

[2043] 3. The device encrypts the data and sends it to the server.

[2044] The device encrypts the collected data with AES-256 and sends it to the server via the HTTPS protocol.

[2045] 4. The server receives and decrypts the data.

[2046] The server receives the data and decrypts it with AES-256.

[2047] 5. The server consolidates and analyzes the data.

[2048] The server performs emotion data recognized by the emotion engine, facial recognition, voice recognition, heart rate analysis, and movement analysis to obtain the integrated results.

[2049] 6. The server authenticates the user against its user database.

[2050] The server checks its user database for a matching record.

[2051] 7. The server notifies the terminal of the authentication result.

[2052] The server notifies the terminal of successful authentication.

[2053] 8. The device notifies the user of the access permission.

[2054] The device displays the access permissions to the user and logs them into the metaverse.

[2055] This system provides a secure and efficient authentication process, from collecting user data to granting access to the metaverse. An example prompt is as follows:

[2056] "Please describe in natural language a scenario in which a user logs into the Metaverse using a VR headset. Please specifically describe the process from user data collection to authentication and access authorization."

[2057] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2058] Step 1:

[2059] A user puts on a VR headset and attempts to access the Metaverse.

[2060] Specifically, this includes the user entering their ID and password to confirm the login screen, which generates an access request and triggers the device to start the authentication process.

[2061] Input: User login information (ID, password)

[2062] Output: Trigger to start the authentication process

[2063] Step 2:

[2064] The device activates sensors and collects user data.

[2065] Specifically, the device's built-in facial recognition camera, heart rate sensor, microphone, motion sensor, and emotion engine are all activated to collect the user's facial expressions, heart rate, tone of voice, behavioral patterns, and emotional data in real time.

[2066] Input: User's biometric information (face, heart rate, voice, movement, emotion)

[2067] Output: Collected bio-behavioral data

[2068] Step 3:

[2069] The terminal stores the collected data in temporary storage.

[2070] Specifically, this involves caching the collected data in temporary storage.

[2071] Input: Collected biometric and behavioral data

[2072] Output: Data stored in temporary storage

[2073] Step 4:

[2074] The device encrypts the collected data using the AES-256 algorithm.

[2075] Specifically, this involves automatically passing data stored in temporary storage to an encryption module where it is encrypted using AES-256.

[2076] Input: Data stored in temporary storage

[2077] Output: Encrypted data

[2078] Step 5:

[2079] The device sends encrypted data to the server using the secure HTTPS protocol.

[2080] Specifically, this includes operations in which encrypted data is transmitted using the HTTPS protocol.

[2081] Input: Encrypted data

[2082] Output: Send data to the server

[2083] Step 6:

[2084] The server receives the encrypted data and decrypts it using the AES-256 algorithm.

[2085] Specifically, the encrypted data is passed to the receiving module and decrypted using AES-256.

[2086] Input: Encrypted data

[2087] Output: Decrypted data

[2088] Step 7:

[2089] The server classifies the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data, and integrates them using multimodal generative AI.

[2090] Specifically, the decoded data is classified into categories, sent to the respective analysis modules, and an integration process is carried out by the multimodal generation AI.

[2091] Input: Decrypted data

[2092] Output: Unified user profile

[2093] Step 8:

[2094] The server analyzes the combined data and authenticates the user.

[2095] Specifically, this involves checking the integrated user profile against an internal database to determine legitimacy.

[2096] Input: Unified User Profile

[2097] Output: Authentication result

[2098] Step 9:

[2099] The server notifies the terminal of the authentication result.

[2100] Specifically, this includes an operation of transmitting the result of authentication success or failure to the terminal.

[2101] Input: Authentication result

[2102] Output: Authentication result notification to the terminal

[2103] Step 10:

[2104] The terminal notifies the user of the access permission.

[2105] Specifically, if authentication is successful, the terminal will display the access permission to the metaverse to the user, and if authentication fails, a corresponding warning message will be displayed.

[2106] Input: Authentication result notification to the terminal

[2107] Output: Allow access to user or warning message

[2108] (Application example 2)

[2109] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2110] Modern authentication systems have adopted a wide range of biometric and behavioral authentication methods for security reasons. However, because these methods rely on individual data, it is difficult to completely eliminate the risk of impersonation and unauthorized access. In particular, when using emotional or biometric data alone, they are susceptible to temporary changes in the user's state and environmental factors, making it difficult to obtain consistent authentication results. Furthermore, there are insufficient means to integrate these data and perform accurate and secure authentication based on a comprehensive user profile.

[2111] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, means for determining the legitimacy of the user based on a user profile generated using the collected biometric data and emotion recognition data, and means for providing access permission to the user based on the authentication result. This provides a consistent authentication result that integrates multiple data, eliminates impersonation and unauthorized access, and enables more secure and reliable authentication.

[2112] Below are definitions of important terms extracted from claim 1:

[2113] "User data" is a general term for information about a user, such as emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data.

[2114] "Encryption" refers to the process of transforming data using specific algorithms to make it unreadable to third parties.

[2115] A "server" is a computer system that stores, analyzes, and processes data over a network.

[2116] "Integration" refers to the act of bringing together multiple types of data and processing them in a consistent format or with consistent standards.

[2117] "Analysis" is the process of examining collected data in detail to find meaning and patterns.

[2118] "Authentication" is the process of verifying that a user has valid access rights.

[2119] A "terminal" is an electronic device that is directly operated by a user, and includes smartphones, personal computers, etc.

[2120] "Access permission" refers to granting access to a system or data.

[2121] "Biometric data" refers to information obtained from the user's body, such as heart rate, fingerprints, and facial shape.

[2122] "Emotion recognition data" is data used to analyze a user's mental state and emotions.

[2123] A "user profile" is a consistent set of information about a user that is obtained by integrating various collected data.

[2124] The present invention provides an advanced authentication system by collecting a user's emotional reactions, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data, and integrating and analyzing this data. Specific embodiments for carrying out the present invention will be described below.

[2125] System Configuration

[2126] The system consists of the following main components:

[2127] Terminal: A device that is directly used by a user, such as a smartphone, smart glasses, a head-mounted display, or a robot.

[2128] Sensors: Face recognition camera, heart rate sensor, microphone, and motion sensor installed on the device.

[2129] Server: A computer system that stores, analyzes, and integrates data.

[2130] Communication Protocol: A secure communication protocol (e.g. HTTPS) for sending and receiving data.

[2131] Encryption algorithm: The encryption method used to ensure secure transmission of data (e.g., AES-256).

[2132] Emotion Engine: An API for analyzing emotions from collected data.

[2133] Program processing overview

[2134] When a user attempts to log in using a device, the device first collects the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using a facial recognition camera, heart rate sensor, microphone, and motion sensors. Using an emotion engine, the device analyzes the collected data in real time to generate emotional data. This data is then stored in temporary memory.

[2135] Next, the device encrypts the collected data with an encryption algorithm (e.g., AES-256) and sends it to the server using a secure communication protocol (e.g., HTTPS). The server receives and decrypts the encrypted data.

[2136] The server analyzes the decoded data using an emotion engine, face recognition algorithm, voice recognition algorithm, heart rate analysis algorithm, and behavior prediction algorithm to classify it into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavior pattern data, and biometric data. A comprehensive user profile is generated from the integrated data.

[2137] Based on the generated user profile, the server compares it with existing user data in its internal database to determine the user's legitimacy. Based on the comparison result, the server notifies the terminal of the authentication result, and the terminal displays the result to the user. If authentication is successful, the user is granted access permission.

[2138] Specific examples of hardware and software used

[2139] Device: Compatible smartphones and head-mounted displays.

[2140] Emotion engines: for example, Affectiva or Microsoft Azure Face API.

[2141] Encryption algorithm: AES-256.

[2142] Communication protocol: HTTPS.

[2143] Analysis algorithms: Multimodal generative AI, OpenCV (face recognition), Google Cloud Speech-to-Text (voice recognition).

[2144] Specific examples

[2145] When a user launches the security app, the app uses the smartphone's camera to capture the user's facial expressions, the heart rate sensor to obtain their heart rate, the microphone to record audio and analyze their tone of voice, and the motion sensor to record their behavioral patterns. This data is then analyzed in real time as emotional data using an emotion engine. The data is then encrypted with AES-256 and sent to a secure server via HTTPS. The server receives, decrypts, and analyzes the data to generate a comprehensive user profile. The server then authenticates the user and notifies the smartphone of the authentication result.

[2146] Prompt Sentence Examples

[2147] Here are some example prompts to input to a generative AI model:

[2148] Describe a system that collects a user's facial recognition, heart rate, tone of voice, behavioral patterns, and biometric data, and then uses an emotion engine to analyze the emotional data in real time. This data is integrated to generate a comprehensive user profile and perform user authentication.

[2149] In this way, the present invention integrates data from multiple data sources and provides a high level of security, thereby improving the reliability and accuracy of authentication.

[2150] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2151] Step 1: Collect user data

[2152] When a user attempts to log in using a device, the device activates various sensors, including a facial recognition camera, heart rate sensor, microphone, and motion sensor, which are used to collect the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data. The input is the user's biometric data, and the output is a set of collected biometric data.

[2153] Step 2: Analyze the emotion data

[2154] The collected biometric data is analyzed in real time into emotional data using an emotion engine, where each data point is mapped to an emotional state. The input is the collected biometric data, and the output is the analyzed emotional data.

[2155] Step 3: Temporarily save data

[2156] The terminal stores the collected data and analyzed emotional data in a temporary memory. The input of this process is the analyzed emotional data and biometric data, and the output is the set of data in the temporary memory.

[2157] Step 4: Encrypt the data

[2158] The terminal encrypts the data in temporary memory using an encryption algorithm (e.g., AES-256). The input is the data stored in temporary memory, and the output is the encrypted data.

[2159] Step 5: Sending data

[2160] The encrypted data is sent to the server using a secure communication protocol (e.g., HTTPS). The input is the encrypted data and the output is the data sent to the server.

[2161] Step 6: Receiving and Decrypting Data

[2162] The server receives the encrypted data and decrypts it with a decryption algorithm (e.g., AES-256). The input is the encrypted data and the output is the decrypted data.

[2163] Step 7: Classify the data

[2164] The server classifies the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data. The input to this process is the decoded data, and the output is the various classified data.

[2165] Step 8: Analyze the data

[2166] The various classified data are analyzed using an emotion engine, face recognition algorithm, voice recognition algorithm, heart rate analysis algorithm, and behavior prediction algorithm. The input is the classified data, and the output is the analysis results.

[2167] Step 9: Generate an Integration Profile

[2168] The server integrates the analysis results and generates a comprehensive user profile. The input is the analyzed data, and the output is the integrated user profile.

[2169] Step 10: User Authentication

[2170] The server compares the generated user profile with existing user data in its internal database to determine its validity. The input is the integrated user profile, and the output is the authentication result.

[2171] Step 11: Notification of authentication results

[2172] The server sends the authentication result to the terminal, and the terminal notifies the user of the result. The input is the authentication result, and the output is the notification to the user.

[2173] Step 12: Access Permissions

[2174] If the authentication is successful, the terminal grants the user access permission. The input is the authentication result, and the output is the granting of access permission.

[2175] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2176] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2177] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2178] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2179] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2180] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2181] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2182] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2183] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2184] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2185] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2186] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2187] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2188] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2189] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2190] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2191] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2192] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2193] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2194] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2195] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2196] The following is further disclosed regarding the above embodiment.

[2197] (Claim 1)

[2198] means for collecting the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data;

[2199] means for encrypting the collected data and transmitting it to a server;

[2200] a server means for integrating and analyzing the received data;

[2201] a server means for authenticating a user based on the analysis result;

[2202] a means for notifying the terminal of the authentication result;

[2203] a means for providing access permission to the user based on the authentication result;

[2204] A system including:

[2205] (Claim 2)

[2206] 10. The system of claim 1, including server means for integrating emotional data, behavioral data, and biometric data using multimodal generative AI.

[2207] (Claim 3)

[2208] 10. The system of claim 1, further comprising means for transmitting user data to the server using a secure communications protocol.

[2209] "Example 1"

[2210] (Claim 1)

[2211] When a user puts on a VR headset and attempts to access the Metaverse, facial recognition cameras, heart rate sensors, and microphones are used to collect emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data.

[2212] means for encrypting the collected data using the AES-256 algorithm and transmitting it to a server via HTTPS protocol;

[2213] A server means for decoding the received data and integrating and analyzing emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data using multimodal generative AI;

[2214] a server means for generating a user profile based on the analysis result and authenticating the user by comparing it with an internal database;

[2215] a means for notifying the terminal of the authentication result and providing the user with access permission if the authentication is successful;

[2216] a means for displaying a warning message if authentication fails;

[2217] A system including:

[2218] (Claim 2)

[2219] 2. The system of claim 1, further comprising a server means for integrating and analyzing emotional response data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data using multimodal generation AI.

[2220] (Claim 3)

[2221] 10. The system of claim 1, including means for securely transmitting user data from the terminal to the server using the HTTPS protocol.

[2222] "Application Example 1"

[2223] (Claim 1)

[2224] means for collecting the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data;

[2225] means for encrypting the collected data and transmitting it to a server;

[2226] a server means for integrating and analyzing the received data;

[2227] a server means for authenticating a user based on the analysis result;

[2228] a means for notifying the terminal of the authentication result;

[2229] a means for providing access permission to the user based on the authentication result;

[2230] A means for collecting, analyzing, and authenticating user data using a VR headset in a corporate security authentication system;

[2231] A system including:

[2232] (Claim 2)

[2233] 10. The system of claim 1, including server means for integrating emotional data, behavioral data, and biometric data using multimodal generative AI.

[2234] (Claim 3)

[2235] 10. The system of claim 1, further comprising means for transmitting user data to the server using a secure communications protocol.

[2236] "Example 2: Combining Emotion Engines"

[2237] (Claim 1)

[2238] means for collecting the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data;

[2239] means for storing the collected data in a temporary storage device;

[2240] means for encrypting the collected data using the AES-256 algorithm;

[2241] means for transmitting encrypted data to a server using a secure communications protocol;

[2242] server means for decrypting the received data using the AES-256 algorithm;

[2243] A server means for classifying the decoded data into emotion recognition data, facial expression data, heart rate data, tone of voice data, behavioral pattern data, and biometric data, and integrating them using multimodal generation AI;

[2244] a server means for analyzing the integrated data;

[2245] a server means for authenticating a user based on the analysis result;

[2246] a means for notifying the terminal of the authentication result;

[2247] a means for providing access permission to the user based on the authentication result;

[2248] A system including:

[2249] (Claim 2)

[2250] 10. The system of claim 1, including server means for integrating emotional data, behavioral data, and biometric data using multimodal generative AI.

[2251] (Claim 3)

[2252] 10. The system of claim 1, further comprising means for transmitting user data to the server using a secure communications protocol.

[2253] "Application example 2 when combining emotion engines"

[2254] (Claim 1)

[2255] means for collecting the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data;

[2256] means for encrypting the collected data and transmitting it to a server;

[2257] a server means for integrating and analyzing the received data;

[2258] a server means for authenticating a user based on the analysis result;

[2259] a means for notifying the terminal of the authentication result;

[2260] a means for providing access permission to the user based on the authentication result;

[2261] a means for determining the legitimacy of a user based on a user profile generated using the collected biometric data and emotion recognition data;

[2262] A system including:

[2263] (Claim 2)

[2264] 10. The system of claim 1, including server means for integrating emotional data, behavioral data, and biometric data using multimodal generative AI.

[2265] (Claim 3)

[2266] 10. The system of claim 1, further comprising means for transmitting user data to the server using a secure communications protocol. [Explanation of symbols]

[2267] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for collecting the user's emotional responses, facial expressions, heart rate, tone of voice, behavioral patterns, and biometric data; means for encrypting the collected data and transmitting it to a server; a server means for integrating and analyzing the received data; a server means for authenticating a user based on the analysis result; a means for notifying the terminal of the authentication result; a means for providing access permission to the user based on the authentication result; A system including:

2. 10. The system of claim 1, further comprising server means for integrating emotional data, behavioral data, and biometric data using multimodal generative AI.

3. 10. The system of claim 1, further comprising means for transmitting user data to the server using a secure communications protocol.

Images