system

Abstract

We provide the system. [Solution] A means of generating a digital twin corresponding to the purchaser, A method for linking event ticket information to the generated digital twin, Methods for obtaining facial photographs and fingerprint data as identity verification information, A method for associating and storing acquired facial photographs and fingerprint data in a digital twin, An input method to obtain the purchaser's facial photograph or fingerprint data upon entry to the event venue, A means of matching acquired facial photograph or fingerprint data with the identity verification information of the digital twin, A system that includes means to grant entry only after successful identity verification.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When tickets for events or concerts are resold, there may arise a problem that the purchaser himself / herself cannot participate in the event. As a result, consumers who purchase tickets at a legitimate price may suffer disadvantages. Also, it becomes difficult to manage the number of attendees due to resold tickets, and the security risk also increases. It is necessary to solve such problems, prevent ticket resale, and ensure that the purchaser himself / herself can surely participate in the event.

Means for Solving the Problems

[0005] This invention relates to a system that generates a digital twin corresponding to a purchaser and links event ticket information to that digital twin. This system includes means for acquiring facial photographs and fingerprint data as identity verification information and storing them in association with the digital twin. Upon entry to the event venue, the purchaser's facial photograph or fingerprint data is acquired by an input means and compared with the identity verification information in the digital twin. Entry is permitted only if this comparison is successful. The digital twin is stored in a database containing event ticket information, and the acquisition of facial photographs and fingerprint data and their comparison with the digital twin are performed using encrypted communication. This prevents ticket resale and ensures that the purchaser themselves can reliably participate in the event.

[0006] A "digital twin" is a virtual data entity linked to a user's personal information, which can uniquely identify the user.

[0007] "Event ticket information" refers to information that indicates the right to participate in a specific event, and includes the event name, date and time, seat information, etc.

[0008] "Personal identification information" refers to information used to uniquely identify a user, and primarily includes biometric information such as facial photographs and fingerprint data.

[0009] A "database" is a system that stores and manages a collection of structured data, and it is where digital twins and event ticket information are stored.

[0010] "Method and system" refers to the means and operating environment necessary for realizing the present invention, and provides a series of functions including the generation of a digital twin, the linking of event ticket information, and the means for verifying identity.

[0011] A "face photograph" is image data of a user's face, and is used as identification information.

[0012] "Fingerprint data" refers to digital data obtained by scanning a user's fingerprints, and is used as personal identification information.

[0013] "Encrypted communication" is a technology for securely sending and receiving data, protecting users' personal information and identification information from being leaked to third parties. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of the data processing device and smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] Modes for carrying out the invention

[0036] This invention is a system that prevents ticket resale by generating a digital twin corresponding to the purchaser and linking event ticket information to that digital twin. A specific embodiment of this system is shown below.

[0037] System Configuration

[0038] This system primarily consists of a server, user terminals, and event venue entry terminals. The server is responsible for generating and managing digital twins, linking event ticket information, and storing and verifying identity information. User terminals are devices used to input personal and identity information when purchasing tickets and to communicate with the server. Event venue entry terminals are devices used to verify the user's identity upon entry.

[0039] Ticket purchase processing

[0040] When a user wishes to purchase an event ticket, the device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into the device. The collected personal information is sent from the device to a server, which generates a digital twin based on the received information. The generated digital twin is linked to the information of the purchased event ticket.

[0041] Entry process

[0042] On the day of the event, a user arrives at the event venue and wishes to enter. A terminal (authentication device) installed at the entrance requests facial recognition or fingerprint authentication from the user. When the user faces the authentication device or performs a fingerprint scan, the acquired facial image or fingerprint data is sent from the terminal to the server. The server compares the received biometric information with the digital twin information, and if they match, entry is permitted. The server sends an entry permission message to the terminal, and the terminal displays the entry permission to the user. If they do not match, the server sends an entry denial message, and the terminal displays this to the user.

[0043] Specific example

[0044] As a concrete example, I will explain the process of purchasing event tickets and the entry process on the day of the event.

[0045] Example of purchasing tickets

[0046] When a user purchases event tickets using their smartphone, the device (smartphone app) prompts the user to take a photo of their face and perform a fingerprint scan. The device sends this information to a server, which generates a digital twin and stores the linked ticket information. Finally, the server sends a purchase completion notification to the user's email address.

[0047] Example of entry

[0048] On the day of the event, when a user arrives at the venue entrance, an authentication terminal will perform facial recognition or fingerprint authentication. The acquired facial image or fingerprint data is sent to a server and compared with a digital twin. If the server successfully matches the data, entry permission is issued, and the terminal displays a message granting entry to the user. If the matching fails, the terminal displays an entry denial message.

[0049] In this way, this system can effectively prevent ticket resale and ensure that legitimate purchasers can attend the event.

[0050] The following describes the processing flow.

[0051] Ticket purchase processing

[0052] Step 1:

[0053] A user wishes to purchase event tickets and opens the purchase screen on their device.

[0054] Step 2:

[0055] The device displays an interface prompting the user to enter their name, email address, facial photo, and fingerprint data.

[0056] Step 3:

[0057] The user provides personal information (name, email address), takes a photo of their face, and performs a fingerprint scan.

[0058] Step 4:

[0059] The device sends collected personal information, facial images, and fingerprint data to the server. This data is encrypted using the SSL / TLS protocol.

[0060] Step 5:

[0061] The server analyzes the personal information it receives and generates a digital twin of the user.

[0062] Step 6:

[0063] The server links event ticket information (event name, date and time, seat information, etc.) to the generated digital twin and stores it in a database.

[0064] Step 7:

[0065] The server sends a ticket purchase completion notification to the user's email address.

[0066] Entry process

[0067] Step 1:

[0068] The user arrives at the event venue and heads towards the authentication terminal located at the entrance.

[0069] Step 2:

[0070] The device displays a message requesting the user to authenticate using facial recognition or fingerprint recognition.

[0071] Step 3:

[0072] The user faces the authentication terminal or performs a fingerprint scan.

[0073] Step 4:

[0074] The device sends facial images and fingerprint data it has acquired to the server. This data is encrypted using the SSL / TLS protocol.

[0075] Step 5:

[0076] The server compares the received biometric information with the information in the digital twin.

[0077] Step 6:

[0078] The server checks the matching results and issues an entry permit if there is a match. If the matching is unsuccessful, it decides to deny entry.

[0079] Step 7:

[0080] The server sends the verification result to the terminal. This includes a permission message if successful, and a rejection message if unsuccessful.

[0081] Step 8:

[0082] The device displays the verification result to the user. If an permission message is displayed, the user is allowed to enter. If a denial message is displayed, the user is prompted to try authenticating again or to go through another procedure.

[0083] The above processing steps create a system that prevents ticket resale and ensures that legitimate purchasers can attend the event.

[0084] (Example 1)

[0085] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0086] Currently, illegal ticket reselling is a problem at many events. This makes it difficult for legitimate buyers to obtain tickets at the regular price. Furthermore, reselling can make identity verification difficult at the entrance, potentially weakening security. To address this, a system is needed that ensures that the original purchaser can enter the event venue while also preventing ticket reselling.

[0087] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0088] In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring image data and biometric data as identity verification information, means for associating and storing the acquired image data and biometric data with the digital twin, input means for acquiring the purchaser's image data or biometric data upon entry to the event venue, means for comparing the acquired image data or biometric data with the identity verification information of the digital twin, means for granting entry if identity verification is successful, and means for generating and storing the digital twin and associated ticket information using an external computing service. This prevents fraudulent ticket resale and enables legitimate purchasers to enter the event with security ensured.

[0089] A "purchaser" is a person who has purchased an event ticket and possesses certain rights.

[0090] A "digital twin" is a virtual digital profile generated based on the purchaser's personal information and biometric data.

[0091] "Event ticket information" refers to data concerning the information and rights necessary to participate in a specific event.

[0092] "Identity verification information" refers to information used to identify the purchaser, and specifically includes image data and biometric data.

[0093] "Image data" refers to data that digitally represents visual information, such as a photograph of the buyer's face.

[0094] "Biometric data" refers to data that includes unique physical characteristics used for purchaser fingerprints and facial recognition.

[0095] "Input means" refers to a device or equipment used to input the purchaser's image data or biometric data at the event venue.

[0096] "Means of matching" refers to methods for comparing acquired image data or biometric data with information from a digital twin to confirm whether they match.

[0097] "Means of granting entry" refers to the means by which a purchaser can enter the event venue if their identity verification is successful.

[0098] "External computing services" refer to external cloud services or computing resources used to generate and store digital twins and associated ticket information.

[0099] This invention is a system that prevents ticket resale and ensures that legitimate purchasers can participate in events. This system primarily includes a server, user terminals, and entry terminals for event facilities. Specific embodiments are described below.

[0100] System Configuration

[0101] This system consists of three main hardware components: a server, user terminals (smartphones, PCs, etc.), and event facility entry terminals (facial recognition cameras, fingerprint scanners, etc.).

[0102] Server: Plays a central role in generating and managing digital twins, linking event ticket information, and storing and verifying identity information. The software used will utilize external computing services (e.g., cloud services) and database systems (e.g., relational database systems (RDBMS)).

[0103] User's device: This includes smartphones and PCs, which are devices used to enter personal information and identity verification information when purchasing tickets and to communicate with the server. The software used includes dedicated applications and web browsers.

[0104] Event venue entry terminals: These are devices used for facial recognition and fingerprint authentication, thereby managing entry to event venues. The hardware used includes facial recognition cameras and fingerprint scanners.

[0105] Ticket purchase processing

[0106] When a user wishes to purchase event tickets, the device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into the device. Based on this information, the device sends the necessary data to the server. The server generates a digital twin based on the transmitted personal information and links the purchased event ticket information to the generated digital twin. In this process, the server uses an external computing service to generate the digital twin and stores the saved information in a database system.

[0107] Specific examples of ticket purchases

[0108] In the specific ticket purchase process, the user purchases event tickets using their smartphone. The smartphone app prompts the user to take a photo of their face and perform a fingerprint scan. The device sends this information to a server, which generates a digital twin and stores the linked ticket information using an external computing service. Finally, the server sends a purchase completion notification to the user's email address.

[0109] Example of a prompt

[0110] To purchase event tickets, please enter the following information: name, email address, photo, and fingerprint data. After entering the information, please press the "Submit" button.

[0111] Entry process

[0112] On the day of the event, a user arrives at the event venue and requests entry. An authentication device installed at the entry terminal requests facial recognition or fingerprint authentication from the user. When the user faces the authentication device or performs a fingerprint scan, the acquired facial image or fingerprint data is sent from the entry terminal to the server. The server compares the received biometric information with the information in the digital twin. If they match, an entry permission message is sent to the terminal and displayed to the user. If the matching fails, an entry denial message is sent and displayed to the user on the terminal.

[0113] Specific examples of entry

[0114] On the day of the event, the user goes to the venue entrance. The entry terminal (authentication device) performs facial recognition or fingerprint authentication. The facial image or fingerprint data is sent to the server and compared with a pre-generated digital twin. If the server successfully matches the user, an entry permission message is issued, and the terminal displays a message granting the user entry. If the matching fails, an entry denial message is displayed.

[0115] Example of a prompt

[0116] "Please use facial recognition or fingerprint authentication to enter. Authentication is in progress, so please wait."

[0117] Thus, this system prevents ticket resale and ensures that legitimate purchasers can safely participate in events while maintaining security.

[0118] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0119] Step 1:

[0120] When a user wishes to purchase event tickets, the device displays a ticket purchase screen. Here, the user enters personal information such as their name, email address, facial photograph, and fingerprint data. This information is retrieved and used as input for the next step.

[0121] Input: Name, email address, facial photo, fingerprint data

[0122] Output: Prepared personal data

[0123] Specific actions:

[0124] Device: The UI for purchasing tickets will be displayed in a web browser or a dedicated app.

[0125] User: Enters their name and email address into a form, takes a photo of their face with the camera, and obtains fingerprint data with a fingerprint scanner.

[0126] Step 2:

[0127] The device sends the acquired personal information to the server. The server receives this information and generates a digital twin.

[0128] Input: Personal information data

[0129] Output: Digital twin generation request

[0130] Specific actions:

[0131] Terminal: Sends personal information to the server via AJAX requests or form submissions.

[0132] Server: Establishes an API endpoint to process received personal information and receives data.

[0133] Step 3:

[0134] The server generates a digital twin based on the personal information it receives. The server uses a relational database system to generate and store the digital twin for the purchaser. Simultaneously, event ticket information is also associated with this digital twin.

[0135] Input: Personal information data

[0136] Output: Digital twin and associated ticket information

[0137] Specific actions:

[0138] Server: Calls APIs of external computing services to generate digital twins from personal information.

[0139] Server: Stores the generated digital twin in an RDBMS and associates it with event ticket information.

[0140] Step 4:

[0141] The server generates a purchase completion notification and sends it to the user's email address. The email includes details of the purchased event and verification information such as a QR code (registered trademark).

[0142] Input: Digital twin and ticket information

[0143] Output: Purchase confirmation email

[0144] Specific actions:

[0145] Server: Uses an email sending service to send purchase completion notifications to users.

[0146] Server: The email content will include event details, purchase confirmation information, QR code, etc.

[0147] Step 5:

[0148] On the day of the event, users arrive at the venue entrance and authenticate at an entry terminal. The authentication device will request facial recognition or fingerprint authentication.

[0149] Input: None

[0150] Output: Facial image or fingerprint data (real-time)

[0151] Specific actions:

[0152] User: Face the authentication terminal or perform a fingerprint scan.

[0153] Device: A biometric authentication device is used to obtain facial images and fingerprint data.

[0154] Step 6:

[0155] The device sends the acquired facial image or fingerprint data to the server. The server receives this data and compares it with the information in the digital twin. If they match, entry is permitted; otherwise, it is denied.

[0156] Input: Real-time biometric data

[0157] Output: Entry permitted message or entry denied message

[0158] Specific actions:

[0159] Terminal: Sends acquired biometric data to the server.

[0160] Server: Matches biometric data with digital twin information and generates results.

[0161] Server: Sends the authentication result to the entry terminal.

[0162] Step 7:

[0163] The terminal receives a response from the server and displays a message to the user indicating whether entry is permitted or denied.

[0164] Input: Admission permission message or admission denial message

[0165] Output: User feedback

[0166] Specific actions:

[0167] Terminal: Receives responses from the server and displays the corresponding message.

[0168] User: Act according to the displayed message (either enter or be denied).

[0169] (Application Example 1)

[0170] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0171] Conventional access control systems carry the risk of loss or leakage of card keys or PINs, highlighting the need for improved security and convenience. Furthermore, while digital twin-based identity verification systems have been used to prevent resale of event tickets, there was a desire for their application to a wider range of uses. Therefore, there is a need for technology that enables highly accurate and secure identity verification and prevents unauthorized access in office and facility access control systems.

[0172] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0173] In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring facial photographs and fingerprint data as identity verification information, means for associating and storing the acquired facial photographs and fingerprint data with the digital twin, input means for acquiring a user's facial photograph or fingerprint data when entering or leaving an office or facility, means for comparing the acquired facial photograph or fingerprint data with the identity verification information of the digital twin, and means for granting entry or exit if identity verification is successful. This enables highly accurate and secure identity verification using digital twins in office and facility access control, and prevents unauthorized access.

[0174] A "digital twin" is a digital avatar corresponding to a specific individual, a virtual entity that possesses the individual's biometric information and related data.

[0175] "Event ticket information" refers to data provided to purchasers as permission to participate in a specific event, and includes information such as the ticket number, event date and time, and seat number.

[0176] "Personal identification information" refers to biometric information used to identify a specific person, such as a person's facial photograph or fingerprint data.

[0177] An "event venue" refers to the physical location or facility where a specific event takes place.

[0178] "Input means" refers to devices or interfaces used to acquire or input data or information.

[0179] An "access control system" is a system used to monitor and manage the entry and exit of people in a specific facility or office.

[0180] "Verification means" refers to technologies and processes for comparing acquired biometric information with information associated with a digital twin to confirm whether they match.

[0181] "Encrypted communication" is a technology used to securely send and receive data, and it is a means of preventing unauthorized access by third parties by transforming data using a specific algorithm.

[0182] This invention is a system that generates a digital twin for use as an access control system for offices and facilities, and performs highly accurate and secure identity verification by using facial photographs and fingerprint data as identity verification information. Specifically, it includes a server, a user terminal, and a device for access control (hereinafter referred to as the terminal).

[0183] System Configuration

[0184] This system uses the following hardware and software:

[0185] Hardware: Terminal devices including smartphones, fingerprint scanners, and cameras.

[0186] Software: OpenCV, requests, fingerprint_scanner library

[0187] Server Functions

[0188] The server includes the following measures:

[0189] 1. Means for generating a digital twin: A digital twin is generated based on the user's personal information.

[0190] 2. Method for linking event ticket information: Link event tickets and entry / exit management information to the generated digital twin.

[0191] 3. Means of storing and verifying identity information: Facial photographs and fingerprint data are stored in association with a digital twin and verified.

[0192] Device functions

[0193] The user's device includes the following features:

[0194] 1. Method of obtaining facial photographs and fingerprint data: Identity verification information is obtained using a smartphone camera and fingerprint reader.

[0195] 2. Data transmission method: The acquired data is sent to the server.

[0196] Access control device functions

[0197] The terminal (access control device) is installed at the entrance of the office or facility and has the following functions:

[0198] 1. Input method: Acquire facial photographs and fingerprint data.

[0199] 2. Verification method: The acquired data is sent to a server and compared with the digital twin.

[0200] 3. Entry / Exit Permission Method: Entry and exit will be permitted only after successful identity verification.

[0201] Example of operation

[0202] Program processing

[0203] The basic processing flow of this system is as follows:

[0204] 1. The user takes a photo of their face with their smartphone camera.

[0205] 2. Obtain fingerprint data using a fingerprint scanner.

[0206] 3. The acquired data is sent to the server to generate and store a digital twin.

[0207] 4. When a user enters or leaves an office or facility, the installed terminal acquires a facial photograph and fingerprint data and sends it to the server.

[0208] 5. The server verifies the biometric information and returns the result to the terminal.

[0209] 6. If they match, entry and exit will be permitted; otherwise, a warning will be displayed.

[0210] Examples

[0211] Employee A enters the office in the morning and follows the following prompts:

[0212] 1. "Please take a photo of your face with your smartphone camera."

[0213] 2. "Please scan your fingerprints."

[0214] 3. "We are checking the authentication results. Please wait a moment."

[0215] The server compares the received information with the digital twin and grants access. During this process, encrypted communication is used throughout the entire system to ensure data security.

[0216] In this way, the invention provides highly accurate identity verification using digital twins, enabling secure access control for offices and facilities.

[0217] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0218] Step 1:

[0219] The user takes a photo of their face using their smartphone.

[0220] Input: User's face image.

[0221] Output: Face image file (e.g., face_image.jpg).

[0222] The user activates their smartphone camera and points it at their face to take a picture. The captured photo is saved as a file on the smartphone. This photo must be of high quality as it will be used in a later authentication process.

[0223] Step 2:

[0224] The user obtains fingerprint data using a fingerprint scanner.

[0225] Input: User's fingerprint.

[0226] Output: Fingerprint data.

[0227] The user places their finger on a fingerprint scanner connected to their smartphone. The scanner reads the fingerprint data and stores it digitally. This data is also used in subsequent authentication processes.

[0228] Step 3:

[0229] The device sends facial image and fingerprint data to the server.

[0230] Input: facial photo file, fingerprint data.

[0231] Output: Data transfer to server complete.

[0232] The device (smartphone) uses an internet connection to send the acquired facial image and fingerprint data to the server. This data is transmitted securely using encrypted communication.

[0233] Step 4:

[0234] The server generates a digital twin and links it with facial image and fingerprint data.

[0235] Input: facial photo file, fingerprint data.

[0236] Output: Digital twin.

[0237] The server generates a digital twin corresponding to the user based on the received facial photograph and fingerprint data. This digital twin is associated with biometric information for identity verification.

[0238] Step 5:

[0239] The server stores event ticket information and entry / exit management information linked to a digital twin.

[0240] Input: Event ticket information or entry / exit management information.

[0241] Output: Information linked to the digital twin.

[0242] The server links pre-registered event ticket information and access control information to the generated digital twin. This allows the digital twin to hold access permission information for specific events or facilities.

[0243] Step 6:

[0244] When users enter or leave an office or facility, the installed terminals acquire facial images and fingerprint data.

[0245] Input: User's facial photo, fingerprint data.

[0246] Output: Acquired biometric information.

[0247] The user stands in front of the access control device, and a camera and fingerprint scanner are used to capture a facial image and fingerprint. This data is collected in real time.

[0248] Step 7:

[0249] The device sends the acquired data to the server and requests identity verification.

[0250] Input: Acquired facial photograph and fingerprint data.

[0251] Output: Sending a request for identity verification.

[0252] The device sends the collected facial image and fingerprint data to the server and requests that it be matched with the information in the digital twin.

[0253] Step 8:

[0254] The server compares the acquired biometric information with the digital twin information.

[0255] Input: Acquired facial photograph, fingerprint data, and biometric information from the digital twin.

[0256] Output: Identity verification result (success or failure).

[0257] The server compares the received biometric information with a digital twin. If they match, the identity verification is considered successful.

[0258] Step 9:

[0259] The server sends the identity verification result to the device.

[0260] Input: Identity verification result.

[0261] Output: Notification of results to the terminal.

[0262] The server returns the verification result to the terminal, notifying it whether entry or exit is permitted or denied.

[0263] Step 10:

[0264] The device will allow or deny entry / exit based on the identity verification results.

[0265] Input: Identity verification result.

[0266] Output: Display of permission or denial of entry / exit.

[0267] The terminal displays the results received from the server, and if identity verification is successful, it allows entry or exit. If there is a mismatch, it displays a warning and denies entry or exit.

[0268] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0269] Modes for carrying out the invention

[0270] This invention relates to a system that generates a digital twin corresponding to a purchaser and links event ticket information to that digital twin. Furthermore, by combining it with an emotion engine, the system can recognize the user's emotions and provide services accordingly.

[0271] System Configuration

[0272] This system primarily consists of a server, user terminals, an emotion engine, and event venue entry terminals. The server is responsible for generating and managing digital twins, linking event ticket information, and storing and verifying identity information. User terminals are devices used to input personal and identity information when purchasing tickets and to communicate with the server. The emotion engine analyzes the user's facial image and voice data to recognize their emotional state. Event venue entry terminals are devices used to verify the user's identity upon entry.

[0273] Ticket purchase processing

[0274] When a user wishes to purchase an event ticket, the device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into the device. The collected personal information is sent from the device to a server, which generates a digital twin based on the received information. The generated digital twin is linked to the information of the purchased event ticket.

[0275] Acquisition and storage of emotional information

[0276] The user's facial image and voice data are analyzed by an emotion engine to recognize the user's emotional state. The acquired emotional information is sent to a server, associated with a digital twin, and stored in a database. This information is also used as a reference when the user uses the system again.

[0277] Entry process

[0278] On the event day, the user arrives at the event venue and wishes to enter. The terminal (authentication device) installed at the entrance requests the user for face authentication or fingerprint authentication. When the user faces the authentication device or performs a fingerprint scan, the captured face photo and fingerprint data are sent from the terminal to the server. The server compares the received biometric information with the digital twin information and grants entry if they match.

[0279] Response Based on Emotion

[0280] From the face photo and voice data obtained at the time of entry, the emotion engine re-analyzes the user's emotional state. For example, if the user is nervous, the system can convey this information to the guiding staff and prompt them to take appropriate actions.

[0281] Specific Example

[0282] As a specific example, the process of purchasing an event ticket, the entry process on the day, and the management of emotion information are explained.

[0283] Example of Ticket Purchase

[0284] When the user purchases an event ticket using a smartphone, the terminal (smartphone app) prompts the user to take a face photo and perform a fingerprint scan. The terminal sends this information to the server, and the server generates a digital twin and saves the associated ticket information. Finally, the server sends a purchase completion notification to the user's email address. At the time of purchase, the user's face photo and voice data are analyzed by the emotion engine, and the emotion information is added to the digital twin.

[0285] Example of Entry

[0286] On the event day, when the user goes to the entrance of the venue, the authentication terminal performs face authentication or fingerprint authentication. The acquired face photo and fingerprint data are sent to the server and compared with the digital twin. When the server succeeds in the comparison, an entry permit is issued, and the terminal displays a message permitting the user to enter. At the same time, the emotion engine can analyze the user's emotional state and give necessary instructions to the event staff.

[0287] In this way, this system can prevent the resale of tickets, ensure that the purchaser himself participates in the event, and provide services according to the user's emotional state.

[0288] The following describes the processing flow.

[0289] Processing at the time of ticket purchase

[0290] Step 1:

[0291] The user hopes to purchase a ticket for the event and opens the purchase screen on the terminal.

[0292] Step 2:

[0293] The terminal displays an interface prompting the user to input name, email address, face photo, and fingerprint data.

[0294] Step 3:

[0295] The user inputs personal information (name, email address), takes a face photo, and performs a fingerprint scan.

[0296] Step 4:

[0297] The terminal sends the collected personal information, face photo, and fingerprint data to the server. This data is encrypted using the SSL / TLS protocol.

[0298] Step 5:

[0299] The server analyzes the personal information received and generates a digital twin of the user.

[0300] Step 6:

[0301] The server associates event ticket information (such as event name, date and time, seat information, etc.) with the generated digital twin and saves it in the database.

[0302] Step 7:

[0303] The server sends a ticket purchase completion notice to the user's email address.

[0304] Acquisition and storage of emotional information

[0305] Step 1:

[0306] Operate the terminal to display an interface for the user to provide a face photo and voice data.

[0307] Step 2:

[0308] The terminal collects the user's face photo and voice data and sends the data to the emotion engine.

[0309] Step 3:

[0310] The emotion engine analyzes the face photo and voice data to determine the user's emotional state.

[0311] Step 4:

[0312] The emotion engine sends the recognized emotional information to the server.

[0313] Step 5:

[0314] The server associates the received emotional information with the digital twin and saves it in the database.

[0315] Processing at the time of entry

[0316] Step 1:

[0317] The user arrives at the event venue and heads towards the authentication terminal located at the entrance.

[0318] Step 2:

[0319] The device displays a message requesting the user to authenticate using facial recognition or fingerprint recognition.

[0320] Step 3:

[0321] The user faces the authentication terminal or performs a fingerprint scan.

[0322] Step 4:

[0323] The device sends facial images and fingerprint data it has acquired to the server. This data is encrypted using the SSL / TLS protocol.

[0324] Step 5:

[0325] The server compares the received biometric information with the information in the digital twin.

[0326] Step 6:

[0327] The server checks the matching results and issues an entry permit if there is a match. If the matching is unsuccessful, it decides to deny entry.

[0328] Step 7:

[0329] The server sends the verification result to the terminal. This includes a permission message if successful, and a rejection message if unsuccessful.

[0330] Step 8:

[0331] The device displays the verification result to the user. If an permission message is displayed, the user is allowed to enter. If a denial message is displayed, the user is prompted to try authenticating again or to go through another procedure.

[0332] Emotion-based responses

[0333] Step 1:

[0334] The device then collects the user's facial image and voice data again and sends them to the emotion engine.

[0335] Step 2:

[0336] The emotion engine analyzes the collected data to determine the user's emotional state.

[0337] Step 3:

[0338] The emotion engine sends the analysis results to the server.

[0339] Step 4:

[0340] Based on the emotional information it receives, the server sends necessary instructions to the event guidance staff.

[0341] Step 5:

[0342] The guidance staff will respond according to the user's emotional state.

[0343] This will prevent ticket resale, ensure that legitimate purchasers can attend the event, and create a system that provides services tailored to the user's emotional state.

[0344] (Example 2)

[0345] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0346] In modern event ticketing systems, resale and fraudulent use are major problems. Furthermore, services that take into account the emotional state of users when attending events are not adequately provided. In particular, utilizing user emotional information is expected to enable more personalized services, but this is difficult to achieve with current systems.

[0347] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0348] In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring biometric information as identity verification information, means for storing the acquired biometric information in association with the digital twin, means for recognizing the emotional state via a user terminal, means for storing the recognized emotional information in association with the digital twin, input means for acquiring the purchaser's biometric information upon entry to the event venue, means for comparing the acquired biometric information with the identity verification information of the digital twin, means for granting entry if identity verification is successful, and means for changing the service based on the emotional state acquired upon entry. This prevents fraudulent use of tickets and enables the provision of personalized services according to the user's emotional state.

[0349] - "Means for generating a digital twin corresponding to a purchaser" refers to a means of creating a virtual entity (digital twin) corresponding to a user in digital space based on the user's personal information and biometric information.

[0350] "Method for linking event ticket information to a generated digital twin" refers to a method for associating and storing information about event tickets purchased by a user with the generated digital twin.

[0351] "Means for acquiring biometric information" refers to means for acquiring biometric characteristics such as a user's facial photograph or fingerprint data.

[0352] "Means for associating and storing acquired biometric information with a digital twin" refers to means for associating acquired user biometric information with a generated digital twin and storing it in a database or other storage device.

[0353] "Means for recognizing emotional states via a user terminal" refers to methods for analyzing a user's facial image and voice data to recognize their emotional state. This includes an emotion recognition engine.

[0354] "Means for associating and storing recognized emotional information with a digital twin" refers to means for associating analyzed user emotional information with a digital twin and storing it in a database or other storage device.

[0355] "Means for comparing acquired biometric information with digital twin identity verification information" refers to methods for comparing and verifying biometric information acquired at event venues, etc., with digital twin identity verification information that has been stored in advance.

[0356] "Means of granting entry upon successful identity verification" refers to the means of granting entry to a user upon successful verification. Specifically, this includes actions such as issuing an electronic permit or opening the venue gates.

[0357] "Means of modifying services based on emotional state acquired upon entry" refers to means of personalizing the services provided based on the emotional state of the user recognized upon entry. For example, this could include sending notifications to staff to help alleviate the user's anxiety.

[0358] This invention is a system that generates a digital twin corresponding to a purchaser and links event ticket information to that digital twin. In addition, by combining it with an emotion engine, it can recognize the user's emotional state and provide services accordingly. The main components of this system include a server, a user terminal, an emotion engine, and an entry terminal at the event venue.

[0359] Components and their operation

[0360] The server generates and manages digital twins, links event ticket information, and stores and verifies identity verification information. The hardware used is a cloud service (e.g., Amazon Web Services EC2 instance). The software used includes a database management system (e.g., AWS® RDS), an email sending service (e.g., AWS SES), an emotion recognition API (e.g., Microsoft® Azure® Emotion API), and a biometric authentication API (e.g., Google® Cloud Platform Biometric API).

[0361] The user's device is used to input personal information and biometric data (facial image and fingerprint data) required when purchasing tickets, and to communicate with the server. This includes smartphones, tablets, and personal computers.

[0362] The emotion engine analyzes the user's facial image and voice data to recognize their emotional state. The analysis results are sent to a server and stored in association with a digital twin.

[0363] Event venue entry terminals are devices used to verify the user's identity upon entry. These include facial recognition cameras and fingerprint scanners.

[0364] Ticket purchase processing

[0365] When a user wishes to purchase an event ticket, their device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into their device. The collected personal information is sent from the device to a server, which generates a digital twin based on the received information. The generated digital twin is linked to the information of the purchased event ticket. An emotion engine analyzes the facial photograph and voice data to recognize the user's emotional state and stores that information in association with the digital twin.

[0366] As a concrete example, when a user purchases concert tickets using their smartphone, the smartphone (device) instructs the user to take a photo of their face and perform a fingerprint scan. This information is sent from the device to a server, which generates a digital twin and stores the purchased ticket information in association with emotional information.

[0367] Entry process

[0368] On the day of the event, a user arrives at the venue and requests entry. An entry terminal (authentication device) installed at the entrance requests facial recognition or fingerprint authentication. When the user faces the authentication device or performs a fingerprint scan, the acquired biometric information is transmitted from the terminal to a server. The server compares the received information with the digital twin data and grants entry if they match. Simultaneously, an emotion engine analyzes the user's emotional state and notifies the event staff of the necessary information.

[0369] As a concrete example, when a user arrives at an event venue and undergoes facial recognition, the authentication terminal takes a picture of the user's face and sends it to the server. The server compares the received facial picture with a digital twin, and if they match, it displays an entry permission message on the terminal.

[0370] Examples of prompts for generative AI models

[0371] The following are examples of prompts for the generative AI model when implementing this system:

[0372] "Please explain the process for analyzing user emotional data at the time of concert ticket purchase and linking that emotional data to the corresponding digital twin."

[0373] "Please describe the system design for verifying user identity using biometric authentication information during the entry process on the day of the event, analyzing their emotional state, and notifying staff accordingly."

[0374] As described above, this system prevents the fraudulent use of tickets, ensures that the purchaser themselves can participate in the event, and provides personalized services tailored to the user's emotional state.

[0375] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0376] The processing flow of this system's program

[0377] Ticket purchase processing steps

[0378] Step 1:

[0379] The user accesses the event ticket purchase site using a device such as a smartphone.

[0380] Input: User actions (launching a web browser app or dedicated mobile app)

[0381] Output: Display of the purchase screen

[0382] Specific actions: The user enters the URL of the purchase site and accesses it, or launches the mobile app.

[0383] Step 2:

[0384] The device displays a ticket purchase screen and prompts the user to enter the necessary personal information (name, email address, photo, fingerprint data).

[0385] Input: System-side purchase screen display command

[0386] Output: Present an input form to the user.

[0387] Specific operation: The system generates HTML and application screens and displays forms for personal information input.

[0388] Step 3:

[0389] The user enters the required information into the provided form, takes a photo of their face, and performs a fingerprint scan.

[0390] Input: User's personal information (name, email address), facial photograph, fingerprints

[0391] Output: The input data is saved to the terminal and temporarily prepared.

[0392] Specific operation: The user takes a photo of their face using the camera and places their finger on the fingerprint sensor to obtain fingerprint data.

[0393] Step 4:

[0394] The device sends the entered personal information and biometric data to the server.

[0395] Input: User's personal information, facial photograph, fingerprint data

[0396] Output: Encrypted transmitted data

[0397] Specific operation: The device encrypts and sends data to the server using the HTTPS protocol.

[0398] Step 5:

[0399] The server generates a digital twin corresponding to the user based on the received data.

[0400] Input: Encrypted personal information and biometric information

[0401] Output: Generated data for the digital twin

[0402] Specific operation: The server executes the digital twin generation algorithm and creates a new entry in the database.

[0403] Step 6:

[0404] The server links and stores the purchased event ticket information to the generated digital twin.

[0405] Input: Digital twin data and purchased ticket information

[0406] Output: Database entries where ticket information is stored in the digital twin.

[0407] Specific operation: The server accesses the database and adds and saves ticket information to the digital twin entry.

[0408] Step 7:

[0409] The server sends a purchase completion notification to the user's email address.

[0410] Input: User's email address and purchase completion information

[0411] Output: Purchase completion notification email

[0412] Specific operation: The server generates and sends a purchase completion notification email using the email sending API.

[0413] Step 8:

[0414] The emotion engine analyzes received facial photos and voice data to recognize the user's emotional state.

[0415] Input: User's facial photo and voice data

[0416] Output: User's emotional state

[0417] Specific operation: An emotion recognition algorithm is executed, and as a result, an emotional state (e.g., joy, surprise, etc.) is generated.

[0418] Step 9:

[0419] The server stores the analyzed emotional information, associating it with a digital twin.

[0420] Input: Analyzed emotional information and digital twin

[0421] Output: Digital twin data with associated emotional information

[0422] Specific operation: The server accesses the database and adds and saves emotional information to the digital twin entry.

[0423] Entry processing steps

[0424] Step 1:

[0425] The user arrives at the venue entrance on the day of the event.

[0426] Input: User's actions (arrived at the venue entrance to participate in the event)

[0427] Output: Start of admission procedure

[0428] Specific action: The user travels to the event venue and arrives at the designated entrance.

[0429] Step 2:

[0430] The entry terminal prompts the user to perform facial recognition or fingerprint authentication.

[0431] Input: System-side authentication start command

[0432] Output: Display of authentication screen

[0433] Specific action: The device's display lights up and prepares for facial recognition or fingerprint recognition.

[0434] Step 3:

[0435] The user either takes a photo of their face or performs a fingerprint scan.

[0436] Input: User's facial photo or fingerprint

[0437] Output: Acquired biometric information

[0438] Specific actions: The user faces the camera or places their finger on the fingerprint sensor.

[0439] Step 4:

[0440] The device sends the acquired biometric information to the server.

[0441] Input: Biometric information (facial photo, fingerprint data)

[0442] Output: Encrypted transmitted data

[0443] Specific operation: The device encrypts and sends data to the server using the HTTPS protocol.

[0444] Step 5:

[0445] The server compares the received biometric information with the digital twin's identity verification information.

[0446] Input: Received biometric information, digital twin identity verification information

[0447] Output: Matching result (success / failure)

[0448] Specific operation: The server executes a biometric matching algorithm to check if there is a match.

[0449] Step 6:

[0450] If the server finds that the verification is successful, it will send an access permission message to the terminal.

[0451] Input: Data for successful matching

[0452] Output: Entry permission message

[0453] Specific operation: Based on the authentication result, the server generates an entry permission message as the next step and sends it to the terminal.

[0454] Step 7:

[0455] The device displays a message granting the user permission to enter.

[0456] Input: Entry permission message

[0457] Output: Screen display of entry permission

[0458] Specific actions: The terminal displays an entry permission message on its screen and performs actions such as opening the gate.

[0459] Step 8:

[0460] The emotion engine re-analyzes the user's latest facial photos and voice data and notifies event staff as needed.

[0461] Input: Latest facial photo and voice data

[0462] Output: Analysis results and notifications

[0463] Specific operation: The emotion engine re-analyzes the emotional state and notifies the event staff of the results via the server.

[0464] Step 9:

[0465] Event staff will receive notifications and take appropriate action for users.

[0466] Input: Notification from emotion engine

[0467] Output: Appropriate response (e.g., guidance and assistance to the user)

[0468] Specific actions: Event staff will check notifications from the emotion engine and respond to users.

[0469] By following these steps, the system can prevent the misuse of tickets and provide personalized services tailored to the user's emotional state.

[0470] (Application Example 2)

[0471] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0472] Conventional admission management systems focused on preventing ticket resale and verifying identity, but they did not address individual customer needs or provide services based on emotional states. Furthermore, they did not utilize customer purchase history in physical stores to provide services. This invention solves these problems, enabling the provision of services tailored to individual customer needs and considering emotional states at event venues and physical stores.

[0473] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring facial photographs and fingerprint data as identity verification information, means for associating and storing the acquired facial photographs and fingerprint data with the digital twin, input means for acquiring the purchaser's facial photograph or fingerprint data upon entry to the event venue, means for comparing the acquired facial photograph or fingerprint data with the identity verification information of the digital twin, means for granting entry if identity verification is successful, means for acquiring information and emotional state from the digital twin and displaying visual information when a customer enters a physical store, means for performing emotion analysis based on facial photographs and displaying the results, and means for acquiring the customer's past purchase history and preferences and suggesting recommended products. This makes it possible to provide services tailored to the individual needs of customers and services based on their emotional state at event venues and physical stores.

[0474] A "digital twin" is a virtual entity that stores and manages the personal information, purchase history, and emotional state of a real person in digital format.

[0475] An "emotion engine" is a software or hardware system that analyzes input facial images and voice data to recognize and evaluate a person's emotional state.

[0476] "Identity verification upon entry" refers to the process of verifying an individual's identity by comparing their actual face and fingerprints with pre-registered facial photographs and fingerprint data when entering a facility such as an event venue.

[0477] A "physical store" is a physical shop where goods are sold or services are provided in person.

[0478] "Purchase history" refers to a record of information about products and services that a customer has purchased in the past.

[0479] "Emotional state" refers to an individual's psychological and emotional condition, including specific emotional states such as joy, confusion, and tension.

[0480] A "recommended product" is a product that is selected and suggested based on the customer's past purchase history and preferences.

[0481] "Facial recognition" is a technology that analyzes captured facial images to identify who the person is.

[0482] "Smart glasses" are glasses-type wearable devices that incorporate a display, camera, and various sensors, enabling them to display, acquire, and process information.

[0483] "Visual information" refers to digital information in a form that can be visually confirmed, such as a customer's facial photograph, purchase history, and emotional state.

[0484] This invention is a system that generates a digital twin corresponding to a purchaser, links it with event ticket information, and acquires the customer's emotional state using an emotion engine, thereby enhancing service provision at physical stores and event venues. This system acquires and processes various data using applications installed on smart glasses, smartphones, robots, etc.

[0485] Hardware and software to use

[0486] server:

[0487] AWS Aurora: Used for database management.

[0488] TENSORFLOW®: A machine learning library used for implementing emotion analysis engines.

[0489] nVIDIA Jetson TX2: Hardware used for facial recognition and emotion analysis processing.

[0490] Terminal:

[0491] Smart glasses: Acquire customer facial photos and emotional states and display visual information.

[0492] Smartphone: Used for purchasing tickets and entering and submitting identity verification information.

[0493] Event venue authentication device: Used to obtain facial images and fingerprint data upon entry.

[0494] Specific processing of the system

[0495] In trading company operations, the system operates as follows:

[0496] Digital Twin Generation

[0497] 1. The server receives the user's facial photograph and fingerprint data and generates a digital twin of the person based on this data. The digital twin is linked to event ticket information and stored in a database along with personal information.

[0498] Providing services at physical stores

[0499] 1. When a user enters a physical store, smart glasses or a robot's camera captures a photo of their face and sends it to a server.

[0500] 2. The server analyzes the acquired facial photograph and retrieves the corresponding digital twin information and past purchase history.

[0501] 3. The device (smart glasses or robot) displays the acquired digital twin information and emotional state, and takes appropriate action based on the current emotional state.

[0502] Emotional analysis and product recommendations

[0503] 1. The emotion engine analyzes the user's facial expression data to determine their current emotional state.

[0504] 2. Based on the assessment results, the server references the customer's preferences and past purchase history to generate a list of recommended products.

[0505] 3. The terminal displays recommended products, and staff members make product suggestions based on that information.

[0506] Specific example

[0507] For example, when a customer enters an event venue, the process works as follows:

[0508] 1. The user arrives at the entrance of the event venue and faces the authentication device.

[0509] 2. The authentication device acquires a facial image and sends it to the server.

[0510] 3. The server verifies the digital twin and grants access if the identity verification is successful.

[0511] 4. Simultaneously, the emotion engine analyzes the user's facial image and notifies the staff of their emotional state.

[0512] Example of a prompt

[0513] Example prompt: "Please provide specific scenarios for analyzing a customer's emotional state and responding accordingly. For example, please describe in detail how to handle a situation where the customer is confused."

[0514] Thus, by utilizing digital twins and an emotion engine, the present invention enables the provision of more personalized and advanced services at event venues and physical stores.

[0515] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0516] Step 1:

[0517] The user launches a ticket purchase application on their smartphone and enters personal information such as their name, email address, facial photo, and fingerprint data.

[0518] Input: Name, email address, photo, fingerprint data

[0519] Data processing and calculation: Converting personal information into an appropriate format for storage in a database.

[0520] Output: Personal information in a saved format

[0521] Step 2:

[0522] The device sends the personal information it has acquired to the server.

[0523] Input: Personal information entered on a smartphone

[0524] Data processing and calculation: Encryption of personal information, network transmission

[0525] Output: Encrypted personal information

[0526] Step 3:

[0527] The server generates a digital twin based on the personal information it receives and links it to the event ticket information.

[0528] Input: Encrypted personal information

[0529] Data processing and computation: Decryption of information, generation of digital twins, linking of ticket information.

[0530] Output: Ticket information linked to the generated digital twin

[0531] Step 4:

[0532] The server sends a purchase completion notification to the user's email address.

[0533] Input: Completed purchase information

[0534] Data processing and calculation: Email notification generation

[0535] Output: Purchase completion notification email

[0536] Step 5:

[0537] On the day of the event, users will face an authentication device at the venue entrance to have their face photographed.

[0538] Input: Current photo

[0539] Data Processing and Calculation: Face Detection and Feature Point Extraction using Face Recognition Algorithms

[0540] Output: Feature point data from facial photographs

[0541] Step 6:

[0542] The device sends the acquired facial image to a server for comparison with a digital twin.

[0543] Input: Feature point data from a facial photograph

[0544] Data processing and calculation: Data encryption, identity verification using matching algorithms.

[0545] Output: Identity Verification Result

[0546] Step 7:

[0547] If the server successfully verifies the user's identity, it will send an access permission message to the device.

[0548] Input: Identity verification result

[0549] Data processing and calculation: Processing of authentication results, message generation.

[0550] Output: Entry permission message

[0551] Step 8:

[0552] When a user enters a physical store, the smart glasses capture a photo of their face and send it to a server.

[0553] Input: Current photo

[0554] Data processing and calculation: Face detection using face recognition algorithms, generation of visual information

[0555] Output: Visual information

[0556] Step 9:

[0557] The server analyzes the acquired facial photographs to obtain corresponding digital twin information and purchase history.

[0558] Input: Facial photograph, digital twin database

[0559] Data processing and calculation: Feature point analysis of facial images, purchase history extraction.

[0560] Output: Digital twin information, purchase history

[0561] Step 10:

[0562] The emotion engine analyzes the user's facial expression data to determine their current emotional state.

[0563] Input: Facial expression data

[0564] Data processing and calculation: Emotion determination using facial expression analysis algorithms

[0565] Output: Emotional state data

[0566] Step 11:

[0567] The server generates a list of recommended products based on the emotion assessment results and displays them on the smart glasses.

[0568] Input: Emotional state data, purchase history

[0569] Data processing and calculation: Execution of recommended product algorithms, list generation.

[0570] Output: Recommended product list

[0571] Step 12:

[0572] The terminal allows staff to suggest products to customers based on recommended items.

[0573] Input: Recommended product list

[0574] Data processing and calculation: Generation and display of proposed information

[0575] Output: Product suggestion message

[0576] This process allows customers to enjoy personalized service at physical stores and event venues.

[0577] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0578] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0579] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0580] [Second Embodiment]

[0581] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0582] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0583] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0584] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0585] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0586] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0587] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0588] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0589] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0590] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0591] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0592] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0593] Modes for carrying out the invention

[0594] This invention is a system that prevents ticket resale by generating a digital twin corresponding to the purchaser and linking event ticket information to that digital twin. A specific embodiment of this system is shown below.

[0595] System Configuration

[0596] This system primarily consists of a server, user terminals, and event venue entry terminals. The server is responsible for generating and managing digital twins, linking event ticket information, and storing and verifying identity information. User terminals are devices used to input personal and identity information when purchasing tickets and to communicate with the server. Event venue entry terminals are devices used to verify the user's identity upon entry.

[0597] Ticket purchase processing

[0598] When a user wishes to purchase an event ticket, the device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into the device. The collected personal information is sent from the device to a server, which generates a digital twin based on the received information. The generated digital twin is linked to the information of the purchased event ticket.

[0599] Entry process

[0600] On the day of the event, a user arrives at the event venue and wishes to enter. A terminal (authentication device) installed at the entrance requests facial recognition or fingerprint authentication from the user. When the user faces the authentication device or performs a fingerprint scan, the acquired facial image or fingerprint data is sent from the terminal to the server. The server compares the received biometric information with the digital twin information, and if they match, entry is permitted. The server sends an entry permission message to the terminal, and the terminal displays the entry permission to the user. If they do not match, the server sends an entry denial message, and the terminal displays this to the user.

[0601] Specific example

[0602] As a concrete example, I will explain the process of purchasing event tickets and the entry process on the day of the event.

[0603] Example of purchasing tickets

[0604] When a user purchases event tickets using their smartphone, the device (smartphone app) prompts the user to take a photo of their face and perform a fingerprint scan. The device sends this information to a server, which generates a digital twin and stores the linked ticket information. Finally, the server sends a purchase completion notification to the user's email address.

[0605] Example of entry

[0606] On the day of the event, when a user arrives at the venue entrance, an authentication terminal will perform facial recognition or fingerprint authentication. The acquired facial image or fingerprint data is sent to a server and compared with a digital twin. If the server successfully matches the data, entry permission is issued, and the terminal displays a message granting entry to the user. If the matching fails, the terminal displays an entry denial message.

[0607] In this way, this system can effectively prevent ticket resale and ensure that legitimate purchasers can attend the event.

[0608] The following describes the processing flow.

[0609] Ticket purchase processing

[0610] Step 1:

[0611] A user wishes to purchase event tickets and opens the purchase screen on their device.

[0612] Step 2:

[0613] The device displays an interface prompting the user to enter their name, email address, facial photo, and fingerprint data.

[0614] Step 3:

[0615] The user provides personal information (name, email address), takes a photo of their face, and performs a fingerprint scan.

[0616] Step 4:

[0617] The device sends collected personal information, facial images, and fingerprint data to the server. This data is encrypted using the SSL / TLS protocol.

[0618] Step 5:

[0619] The server analyzes the personal information it receives and generates a digital twin of the user.

[0620] Step 6:

[0621] The server links event ticket information (event name, date and time, seat information, etc.) to the generated digital twin and stores it in a database.

[0622] Step 7:

[0623] The server sends a ticket purchase completion notification to the user's email address.

[0624] Entry process

[0625] Step 1:

[0626] The user arrives at the event venue and heads towards the authentication terminal located at the entrance.

[0627] Step 2:

[0628] The device displays a message requesting the user to authenticate using facial recognition or fingerprint recognition.

[0629] Step 3:

[0630] The user faces the authentication terminal or performs a fingerprint scan.

[0631] Step 4:

[0632] The device sends facial images and fingerprint data it has acquired to the server. This data is encrypted using the SSL / TLS protocol.

[0633] Step 5:

[0634] The server compares the received biometric information with the information in the digital twin.

[0635] Step 6:

[0636] The server checks the matching results and issues an entry permit if there is a match. If the matching is unsuccessful, it decides to deny entry.

[0637] Step 7:

[0638] The server sends the verification result to the terminal. This includes a permission message if successful, and a rejection message if unsuccessful.

[0639] Step 8:

[0640] The device displays the verification result to the user. If an permission message is displayed, the user is allowed to enter. If a denial message is displayed, the user is prompted to try authenticating again or to go through another procedure.

[0641] The above processing steps create a system that prevents ticket resale and ensures that legitimate purchasers can attend the event.

[0642] (Example 1)

[0643] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0644] Currently, illegal ticket reselling is a problem at many events. This makes it difficult for legitimate buyers to obtain tickets at the regular price. Furthermore, reselling can make identity verification difficult at the entrance, potentially weakening security. To address this, a system is needed that ensures that the original purchaser can enter the event venue while also preventing ticket reselling.

[0645] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0646] In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring image data and biometric data as identity verification information, means for associating and storing the acquired image data and biometric data with the digital twin, input means for acquiring the purchaser's image data or biometric data upon entry to the event venue, means for comparing the acquired image data or biometric data with the identity verification information of the digital twin, means for granting entry if identity verification is successful, and means for generating and storing the digital twin and associated ticket information using an external computing service. This prevents fraudulent ticket resale and enables legitimate purchasers to enter the event with security ensured.

[0647] A "purchaser" is a person who has purchased an event ticket and possesses certain rights.

[0648] A "digital twin" is a virtual digital profile generated based on the purchaser's personal information and biometric data.

[0649] "Event ticket information" refers to data concerning the information and rights necessary to participate in a specific event.

[0650] "Identity verification information" refers to information used to identify the purchaser, and specifically includes image data and biometric data.

[0651] "Image data" refers to data that digitally represents visual information, such as a photograph of the buyer's face.

[0652] "Biometric data" refers to data that includes unique physical characteristics used for purchaser fingerprints and facial recognition.

[0653] "Input means" refers to a device or equipment used to input the purchaser's image data or biometric data at the event venue.

[0654] "Means of matching" refers to methods for comparing acquired image data or biometric data with information from a digital twin to confirm whether they match.

[0655] "Means of granting entry" refers to the means by which a purchaser can enter the event venue if their identity verification is successful.

[0656] "External computing services" refer to external cloud services or computing resources used to generate and store digital twins and associated ticket information.

[0657] This invention is a system that prevents ticket resale and ensures that legitimate purchasers can participate in events. This system primarily includes a server, user terminals, and entry terminals for event facilities. Specific embodiments are described below.

[0658] System Configuration

[0659] This system consists of three main hardware components: a server, user terminals (smartphones, PCs, etc.), and event facility entry terminals (facial recognition cameras, fingerprint scanners, etc.).

[0660] Server: Plays a central role in generating and managing digital twins, linking event ticket information, and storing and verifying identity information. The software used will utilize external computing services (e.g., cloud services) and database systems (e.g., relational database systems (RDBMS)).

[0661] User's device: This includes smartphones and PCs, which are devices used to enter personal information and identity verification information when purchasing tickets and to communicate with the server. The software used includes dedicated applications and web browsers.

[0662] Event venue entry terminals: These are devices used for facial recognition and fingerprint authentication, thereby managing entry to event venues. The hardware used includes facial recognition cameras and fingerprint scanners.

[0663] Ticket purchase processing

[0664] When a user wishes to purchase event tickets, the device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into the device. Based on this information, the device sends the necessary data to the server. The server generates a digital twin based on the transmitted personal information and links the purchased event ticket information to the generated digital twin. In this process, the server uses an external computing service to generate the digital twin and stores the saved information in a database system.

[0665] Specific examples of ticket purchases

[0666] In the specific ticket purchase process, the user purchases event tickets using their smartphone. The smartphone app prompts the user to take a photo of their face and perform a fingerprint scan. The device sends this information to a server, which generates a digital twin and stores the linked ticket information using an external computing service. Finally, the server sends a purchase completion notification to the user's email address.

[0667] Example of a prompt

[0668] To purchase event tickets, please enter the following information: name, email address, photo, and fingerprint data. After entering the information, please press the "Submit" button.

[0669] Entry process

[0670] On the day of the event, a user arrives at the event venue and requests entry. An authentication device installed at the entry terminal requests facial recognition or fingerprint authentication from the user. When the user faces the authentication device or performs a fingerprint scan, the acquired facial image or fingerprint data is sent from the entry terminal to the server. The server compares the received biometric information with the information in the digital twin. If they match, an entry permission message is sent to the terminal and displayed to the user. If the matching fails, an entry denial message is sent and displayed to the user on the terminal.

[0671] Specific examples of entry

[0672] On the day of the event, the user goes to the venue entrance. The entry terminal (authentication device) performs facial recognition or fingerprint authentication. The facial image or fingerprint data is sent to the server and compared with a pre-generated digital twin. If the server successfully matches the user, an entry permission message is issued, and the terminal displays a message granting the user entry. If the matching fails, an entry denial message is displayed.

[0673] Example of a prompt

[0674] "Please use facial recognition or fingerprint authentication to enter. Authentication is in progress, so please wait."

[0675] Thus, this system prevents ticket resale and ensures that legitimate purchasers can safely participate in events while maintaining security.

[0676] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0677] Step 1:

[0678] When a user wishes to purchase event tickets, the device displays a ticket purchase screen. Here, the user enters personal information such as their name, email address, facial photograph, and fingerprint data. This information is retrieved and used as input for the next step.

[0679] Input: Name, email address, facial photo, fingerprint data

[0680] Output: Prepared personal data

[0681] Specific actions:

[0682] Device: The UI for purchasing tickets will be displayed in a web browser or a dedicated app.

[0683] User: Enters their name and email address into a form, takes a photo of their face with the camera, and obtains fingerprint data with a fingerprint scanner.

[0684] Step 2:

[0685] The device sends the acquired personal information to the server. The server receives this information and generates a digital twin.

[0686] Input: Personal information data

[0687] Output: Digital twin generation request

[0688] Specific actions:

[0689] Terminal: Sends personal information to the server via AJAX requests or form submissions.

[0690] Server: Establishes an API endpoint to process received personal information and receives data.

[0691] Step 3:

[0692] The server generates a digital twin based on the personal information it receives. The server uses a relational database system to generate and store the digital twin for the purchaser. Simultaneously, event ticket information is also associated with this digital twin.

[0693] Input: Personal information data

[0694] Output: Digital twin and associated ticket information

[0695] Specific actions:

[0696] Server: Calls APIs of external computing services to generate digital twins from personal information.

[0697] Server: Stores the generated digital twin in an RDBMS and associates it with event ticket information.

[0698] Step 4:

[0699] The server generates a purchase completion notification and sends it to the user's email address. The email includes details of the purchased event and confirmation information such as a QR code.

[0700] Input: Digital twin and ticket information

[0701] Output: Purchase confirmation email

[0702] Specific actions:

[0703] Server: Uses an email sending service to send purchase completion notifications to users.

[0704] Server: The email content will include event details, purchase confirmation information, QR code, etc.

[0705] Step 5:

[0706] On the day of the event, users arrive at the venue entrance and authenticate at an entry terminal. The authentication device will request facial recognition or fingerprint authentication.

[0707] Input: None

[0708] Output: Facial image or fingerprint data (real-time)

[0709] Specific actions:

[0710] User: Face the authentication terminal or perform a fingerprint scan.

[0711] Device: A biometric authentication device is used to obtain facial images and fingerprint data.

[0712] Step 6:

[0713] The device sends the acquired facial image or fingerprint data to the server. The server receives this data and compares it with the information in the digital twin. If they match, entry is permitted; otherwise, it is denied.

[0714] Input: Real-time biometric data

[0715] Output: Entry permitted message or entry denied message

[0716] Specific actions:

[0717] Terminal: Sends acquired biometric data to the server.

[0718] Server: Matches biometric data with digital twin information and generates results.

[0719] Server: Sends the authentication result to the entry terminal.

[0720] Step 7:

[0721] The terminal receives a response from the server and displays a message to the user indicating whether entry is permitted or denied.

[0722] Input: Admission permission message or admission denial message

[0723] Output: User feedback

[0724] Specific actions:

[0725] Terminal: Receives responses from the server and displays the corresponding message.

[0726] User: Act according to the displayed message (either enter or be denied).

[0727] (Application Example 1)

[0728] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0729] Conventional access control systems carry the risk of loss or leakage of card keys or PINs, highlighting the need for improved security and convenience. Furthermore, while digital twin-based identity verification systems have been used to prevent resale of event tickets, there was a desire for their application to a wider range of uses. Therefore, there is a need for technology that enables highly accurate and secure identity verification and prevents unauthorized access in office and facility access control systems.

[0730] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0731] In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring facial photographs and fingerprint data as identity verification information, means for associating and storing the acquired facial photographs and fingerprint data with the digital twin, input means for acquiring a user's facial photograph or fingerprint data when entering or leaving an office or facility, means for comparing the acquired facial photograph or fingerprint data with the identity verification information of the digital twin, and means for granting entry or exit if identity verification is successful. This enables highly accurate and secure identity verification using digital twins in office and facility access control, and prevents unauthorized access.

[0732] A "digital twin" is a digital avatar corresponding to a specific individual, a virtual entity that possesses the individual's biometric information and related data.

[0733] "Event ticket information" refers to data provided to purchasers as permission to participate in a specific event, and includes information such as the ticket number, event date and time, and seat number.

[0734] "Personal identification information" refers to biometric information used to identify a specific person, such as a person's facial photograph or fingerprint data.

[0735] An "event venue" refers to the physical location or facility where a specific event takes place.

[0736] "Input means" refers to devices or interfaces used to acquire or input data or information.

[0737] An "access control system" is a system used to monitor and manage the entry and exit of people in a specific facility or office.

[0738] "Verification means" refers to technologies and processes for comparing acquired biometric information with information associated with a digital twin to confirm whether they match.

[0739] "Encrypted communication" is a technology used to securely send and receive data, and it is a means of preventing unauthorized access by third parties by transforming data using a specific algorithm.

[0740] This invention is a system that generates a digital twin for use as an access control system for offices and facilities, and performs highly accurate and secure identity verification by using facial photographs and fingerprint data as identity verification information. Specifically, it includes a server, a user terminal, and a device for access control (hereinafter referred to as the terminal).

[0741] System Configuration

[0742] This system uses the following hardware and software:

[0743] Hardware: Terminal devices including smartphones, fingerprint scanners, and cameras.

[0744] Software: OpenCV, requests, fingerprint_scanner library

[0745] Server Functions

[0746] The server includes the following measures:

[0747] 1. Means for generating a digital twin: A digital twin is generated based on the user's personal information.

[0748] 2. Method for linking event ticket information: Link event tickets and entry / exit management information to the generated digital twin.

[0749] 3. Means of storing and verifying identity information: Facial photographs and fingerprint data are stored in association with a digital twin and verified.

[0750] Device functions

[0751] The user's device includes the following features:

[0752] 1. Method of obtaining facial photographs and fingerprint data: Identity verification information is obtained using a smartphone camera and fingerprint reader.

[0753] 2. Data transmission method: The acquired data is sent to the server.

[0754] Access control device functions

[0755] The terminal (access control device) is installed at the entrance of the office or facility and has the following functions:

[0756] 1. Input method: Acquire facial photographs and fingerprint data.

[0757] 2. Verification method: The acquired data is sent to a server and compared with the digital twin.

[0758] 3. Entry / Exit Permission Method: Entry and exit will be permitted only after successful identity verification.

[0759] Example of operation

[0760] Program processing

[0761] The basic processing flow of this system is as follows:

[0762] 1. The user takes a photo of their face with their smartphone camera.

[0763] 2. Obtain fingerprint data using a fingerprint scanner.

[0764] 3. The acquired data is sent to the server to generate and store a digital twin.

[0765] 4. When a user enters or leaves an office or facility, the installed terminal acquires a facial photograph and fingerprint data and sends it to the server.

[0766] 5. The server verifies the biometric information and returns the result to the terminal.

[0767] 6. If they match, entry and exit will be permitted; otherwise, a warning will be displayed.

[0768] Examples

[0769] Employee A enters the office in the morning and follows the following prompts:

[0770] 1. "Please take a photo of your face with your smartphone camera."

[0771] 2. "Please scan your fingerprints."

[0772] 3. "We are checking the authentication results. Please wait a moment."

[0773] The server compares the received information with the digital twin and grants access. During this process, encrypted communication is used throughout the entire system to ensure data security.

[0774] In this way, the invention provides highly accurate identity verification using digital twins, enabling secure access control for offices and facilities.

[0775] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0776] Step 1:

[0777] The user takes a photo of their face using their smartphone.

[0778] Input: User's face image.

[0779] Output: Face image file (e.g., face_image.jpg).

[0780] The user activates their smartphone camera and points it at their face to take a picture. The captured photo is saved as a file on the smartphone. This photo must be of high quality as it will be used in a later authentication process.

[0781] Step 2:

[0782] The user obtains fingerprint data using a fingerprint scanner.

[0783] Input: User's fingerprint.

[0784] Output: Fingerprint data.

[0785] The user places their finger on a fingerprint scanner connected to their smartphone. The scanner reads the fingerprint data and stores it digitally. This data is also used in subsequent authentication processes.

[0786] Step 3:

[0787] The device sends facial image and fingerprint data to the server.

[0788] Input: facial photo file, fingerprint data.

[0789] Output: Data transfer to server complete.

[0790] The device (smartphone) uses an internet connection to send the acquired facial image and fingerprint data to the server. This data is transmitted securely using encrypted communication.

[0791] Step 4:

[0792] The server generates a digital twin and links it with facial image and fingerprint data.

[0793] Input: facial photo file, fingerprint data.

[0794] Output: Digital twin.

[0795] The server generates a digital twin corresponding to the user based on the received facial photograph and fingerprint data. This digital twin is associated with biometric information for identity verification.

[0796] Step 5:

[0797] The server stores event ticket information and entry / exit management information linked to a digital twin.

[0798] Input: Event ticket information or entry / exit management information.

[0799] Output: Information linked to the digital twin.

[0800] The server links pre-registered event ticket information and access control information to the generated digital twin. This allows the digital twin to hold access permission information for specific events or facilities.

[0801] Step 6:

[0802] When users enter or leave an office or facility, the installed terminals acquire facial images and fingerprint data.

[0803] Input: User's facial photo, fingerprint data.

[0804] Output: Acquired biometric information.

[0805] The user stands in front of the access control device, and a camera and fingerprint scanner are used to capture a facial image and fingerprint. This data is collected in real time.

[0806] Step 7:

[0807] The device sends the acquired data to the server and requests identity verification.

[0808] Input: Acquired facial photograph and fingerprint data.

[0809] Output: Sending a request for identity verification.

[0810] The device sends the collected facial image and fingerprint data to the server and requests that it be matched with the information in the digital twin.

[0811] Step 8:

[0812] The server compares the acquired biometric information with the digital twin information.

[0813] Input: Acquired facial photograph, fingerprint data, and biometric information from the digital twin.

[0814] Output: Identity verification result (success or failure).

[0815] The server compares the received biometric information with a digital twin. If they match, the identity verification is considered successful.

[0816] Step 9:

[0817] The server sends the identity verification result to the device.

[0818] Input: Identity verification result.

[0819] Output: Notification of results to the terminal.

[0820] The server returns the verification result to the terminal, notifying it whether entry or exit is permitted or denied.

[0821] Step 10:

[0822] The device will allow or deny entry / exit based on the identity verification results.

[0823] Input: Identity verification result.

[0824] Output: Display of permission or denial of entry / exit.

[0825] The terminal displays the results received from the server, and if identity verification is successful, it allows entry or exit. If there is a mismatch, it displays a warning and denies entry or exit.

[0826] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0827] Modes for carrying out the invention

[0828] This invention relates to a system that generates a digital twin corresponding to a purchaser and links event ticket information to that digital twin. Furthermore, by combining it with an emotion engine, the system can recognize the user's emotions and provide services accordingly.

[0829] System Configuration

[0830] This system primarily consists of a server, user terminals, an emotion engine, and event venue entry terminals. The server is responsible for generating and managing digital twins, linking event ticket information, and storing and verifying identity information. User terminals are devices used to input personal and identity information when purchasing tickets and to communicate with the server. The emotion engine analyzes the user's facial image and voice data to recognize their emotional state. Event venue entry terminals are devices used to verify the user's identity upon entry.

[0831] Ticket purchase processing

[0832] When a user wishes to purchase an event ticket, the device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into the device. The collected personal information is sent from the device to a server, which generates a digital twin based on the received information. The generated digital twin is linked to the information of the purchased event ticket.

[0833] Acquisition and storage of emotional information

[0834] The user's facial image and voice data are analyzed by an emotion engine to recognize the user's emotional state. The acquired emotional information is sent to a server, associated with a digital twin, and stored in a database. This information is also used as a reference when the user uses the system again.

[0835] Entry process

[0836] On the day of the event, a user arrives at the event venue and requests entry. A terminal (authentication device) installed at the entrance requests facial recognition or fingerprint authentication from the user. When the user faces the authentication device or performs a fingerprint scan, the acquired facial image or fingerprint data is sent from the terminal to a server. The server compares the received biometric information with the digital twin information, and if they match, entry is permitted.

[0837] Emotion-based responses

[0838] Based on facial images and voice data acquired upon entry, the emotion engine re-analyzes the user's emotional state. For example, if the user is nervous, the system can relay this information to the guidance staff and prompt them to take appropriate action.

[0839] Specific example

[0840] As concrete examples, we will explain the process of purchasing event tickets, the entry process on the day of the event, and the management of emotional information.

[0841] Example of purchasing tickets

[0842] When a user purchases an event ticket using their smartphone, the device (smartphone app) prompts the user to take a photo of their face and perform a fingerprint scan. The device sends this information to a server, which generates a digital twin and stores the linked ticket information. Finally, the server sends a purchase completion notification to the user's email address. During the purchase, the user's facial image and voice data are analyzed by an emotion engine, and emotion information is added to the digital twin.

[0843] Example of entry

[0844] On the day of the event, when a user arrives at the venue entrance, an authentication terminal performs facial recognition or fingerprint authentication. The acquired facial image or fingerprint data is sent to a server and compared with a digital twin. If the server successfully matches the data, entry permission is issued, and the terminal displays a message granting entry to the user. Simultaneously, an emotion engine analyzes the user's emotional state and can provide necessary instructions to the event staff.

[0845] In this way, this system prevents ticket resale, ensures that the purchaser is indeed able to participate in the event, and provides services tailored to the user's emotional state.

[0846] The following describes the processing flow.

[0847] Ticket purchase processing

[0848] Step 1:

[0849] A user wishes to purchase event tickets and opens the purchase screen on their device.

[0850] Step 2:

[0851] The device displays an interface prompting the user to enter their name, email address, facial photo, and fingerprint data.

[0852] Step 3:

[0853] The user provides personal information (name, email address), takes a photo of their face, and performs a fingerprint scan.

[0854] Step 4:

[0855] The device sends collected personal information, facial images, and fingerprint data to the server. This data is encrypted using the SSL / TLS protocol.

[0856] Step 5:

[0857] The server analyzes the personal information it receives and generates a digital twin of the user.

[0858] Step 6:

[0859] The server links event ticket information (event name, date and time, seat information, etc.) to the generated digital twin and stores it in a database.

[0860] Step 7:

[0861] The server sends a ticket purchase completion notification to the user's email address.

[0862] Acquisition and storage of emotional information

[0863] Step 1:

[0864] The device is operated to display an interface for the user to provide a facial photograph and voice data.

[0865] Step 2:

[0866] The device collects the user's facial image and voice data, and sends that data to the emotion engine.

[0867] Step 3:

[0868] The emotion engine analyzes facial images and voice data to determine the user's emotional state.

[0869] Step 4:

[0870] The emotion engine sends the recognized emotion information to the server.

[0871] Step 5:

[0872] The server associates the received emotional information with a digital twin and stores it in a database.

[0873] Entry process

[0874] Step 1:

[0875] The user arrives at the event venue and heads towards the authentication terminal located at the entrance.

[0876] Step 2:

[0877] The device displays a message requesting the user to authenticate using facial recognition or fingerprint recognition.

[0878] Step 3:

[0879] The user faces the authentication terminal or performs a fingerprint scan.

[0880] Step 4:

[0881] The device sends facial images and fingerprint data it has acquired to the server. This data is encrypted using the SSL / TLS protocol.

[0882] Step 5:

[0883] The server compares the received biometric information with the information in the digital twin.

[0884] Step 6:

[0885] The server checks the matching results and issues an entry permit if there is a match. If the matching is unsuccessful, it decides to deny entry.

[0886] Step 7:

[0887] The server sends the verification result to the terminal. This includes a permission message if successful, and a rejection message if unsuccessful.

[0888] Step 8:

[0889] The device displays the verification result to the user. If an permission message is displayed, the user is allowed to enter. If a denial message is displayed, the user is prompted to try authenticating again or to go through another procedure.

[0890] Emotion-based responses

[0891] Step 1:

[0892] The device then collects the user's facial image and voice data again and sends them to the emotion engine.

[0893] Step 2:

[0894] The emotion engine analyzes the collected data to determine the user's emotional state.

[0895] Step 3:

[0896] The emotion engine sends the analysis results to the server.

[0897] Step 4:

[0898] Based on the emotional information it receives, the server sends necessary instructions to the event guidance staff.

[0899] Step 5:

[0900] The guidance staff will respond according to the user's emotional state.

[0901] This will prevent ticket resale, ensure that legitimate purchasers can attend the event, and create a system that provides services tailored to the user's emotional state.

[0902] (Example 2)

[0903] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0904] In modern event ticketing systems, resale and fraudulent use are major problems. Furthermore, services that take into account the emotional state of users when attending events are not adequately provided. In particular, utilizing user emotional information is expected to enable more personalized services, but this is difficult to achieve with current systems.

[0905] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0906] In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring biometric information as identity verification information, means for storing the acquired biometric information in association with the digital twin, means for recognizing the emotional state via a user terminal, means for storing the recognized emotional information in association with the digital twin, input means for acquiring the purchaser's biometric information upon entry to the event venue, means for comparing the acquired biometric information with the identity verification information of the digital twin, means for granting entry if identity verification is successful, and means for changing the service based on the emotional state acquired upon entry. This prevents fraudulent use of tickets and enables the provision of personalized services according to the user's emotional state.

[0907] - "Means for generating a digital twin corresponding to a purchaser" refers to a means of creating a virtual entity (digital twin) corresponding to a user in digital space based on the user's personal information and biometric information.

[0908] "Method for linking event ticket information to a generated digital twin" refers to a method for associating and storing information about event tickets purchased by a user with the generated digital twin.

[0909] "Means for acquiring biometric information" refers to means for acquiring biometric characteristics such as a user's facial photograph or fingerprint data.

[0910] "Means for associating and storing acquired biometric information with a digital twin" refers to means for associating acquired user biometric information with a generated digital twin and storing it in a database or other storage device.

[0911] "Means for recognizing emotional states via a user terminal" refers to methods for analyzing a user's facial image and voice data to recognize their emotional state. This includes an emotion recognition engine.

[0912] "Means for associating and storing recognized emotional information with a digital twin" refers to means for associating analyzed user emotional information with a digital twin and storing it in a database or other storage device.

[0913] "Means for comparing acquired biometric information with digital twin identity verification information" refers to methods for comparing and verifying biometric information acquired at event venues, etc., with digital twin identity verification information that has been stored in advance.

[0914] "Means of granting entry upon successful identity verification" refers to the means of granting entry to a user upon successful verification. Specifically, this includes actions such as issuing an electronic permit or opening the venue gates.

[0915] "Means of modifying services based on emotional state acquired upon entry" refers to means of personalizing the services provided based on the emotional state of the user recognized upon entry. For example, this could include sending notifications to staff to help alleviate the user's anxiety.

[0916] This invention is a system that generates a digital twin corresponding to a purchaser and links event ticket information to that digital twin. In addition, by combining it with an emotion engine, it can recognize the user's emotional state and provide services accordingly. The main components of this system include a server, a user terminal, an emotion engine, and an entry terminal at the event venue.

[0917] Components and their operation

[0918] The server will generate and manage digital twins, link event ticket information, and store and verify identity information. The hardware used will be cloud services (e.g., Amazon Web Services EC2 instances). The software used will include a database management system (e.g., AWS RDS), an email sending service (e.g., AWS SES), an emotion recognition API (e.g., Microsoft Azure's Emotion API), and a biometric authentication API (e.g., Google Cloud Platform's Biometric API).

[0919] The user's device is used to input personal information and biometric data (facial image and fingerprint data) required when purchasing tickets, and to communicate with the server. This includes smartphones, tablets, and personal computers.

[0920] The emotion engine analyzes the user's facial image and voice data to recognize their emotional state. The analysis results are sent to a server and stored in association with a digital twin.

[0921] Event venue entry terminals are devices used to verify the user's identity upon entry. These include facial recognition cameras and fingerprint scanners.

[0922] Ticket purchase processing

[0923] When a user wishes to purchase an event ticket, their device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into their device. The collected personal information is sent from the device to a server, which generates a digital twin based on the received information. The generated digital twin is linked to the information of the purchased event ticket. An emotion engine analyzes the facial photograph and voice data to recognize the user's emotional state and stores that information in association with the digital twin.

[0924] As a concrete example, when a user purchases concert tickets using their smartphone, the smartphone (device) instructs the user to take a photo of their face and perform a fingerprint scan. This information is sent from the device to a server, which generates a digital twin and stores the purchased ticket information in association with emotional information.

[0925] Entry process

[0926] On the day of the event, a user arrives at the venue and requests entry. An entry terminal (authentication device) installed at the entrance requests facial recognition or fingerprint authentication. When the user faces the authentication device or performs a fingerprint scan, the acquired biometric information is transmitted from the terminal to a server. The server compares the received information with the digital twin data and grants entry if they match. Simultaneously, an emotion engine analyzes the user's emotional state and notifies the event staff of the necessary information.

[0927] As a concrete example, when a user arrives at an event venue and undergoes facial recognition, the authentication terminal takes a picture of the user's face and sends it to the server. The server compares the received facial picture with a digital twin, and if they match, it displays an entry permission message on the terminal.

[0928] Examples of prompts for generative AI models

[0929] The following are examples of prompts for the generative AI model when implementing this system:

[0930] "Please explain the process for analyzing user emotional data at the time of concert ticket purchase and linking that emotional data to the corresponding digital twin."

[0931] "Please describe the system design for verifying user identity using biometric authentication information during the entry process on the day of the event, analyzing their emotional state, and notifying staff accordingly."

[0932] As described above, this system prevents the fraudulent use of tickets, ensures that the purchaser themselves can participate in the event, and provides personalized services tailored to the user's emotional state.

[0933] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0934] The processing flow of this system's program

[0935] Ticket purchase processing steps

[0936] Step 1:

[0937] The user accesses the event ticket purchase site using a device such as a smartphone.

[0938] Input: User actions (launching a web browser app or dedicated mobile app)

[0939] Output: Display of the purchase screen

[0940] Specific actions: The user enters the URL of the purchase site and accesses it, or launches the mobile app.

[0941] Step 2:

[0942] The device displays a ticket purchase screen and prompts the user to enter the necessary personal information (name, email address, photo, fingerprint data).

[0943] Input: System-side purchase screen display command

[0944] Output: Present an input form to the user.

[0945] Specific operation: The system generates HTML and application screens and displays forms for personal information input.

[0946] Step 3:

[0947] The user enters the required information into the provided form, takes a photo of their face, and performs a fingerprint scan.

[0948] Input: User's personal information (name, email address), facial photograph, fingerprints

[0949] Output: The input data is saved to the terminal and temporarily prepared.

[0950] Specific operation: The user takes a photo of their face using the camera and places their finger on the fingerprint sensor to obtain fingerprint data.

[0951] Step 4:

[0952] The device sends the entered personal information and biometric data to the server.

[0953] Input: User's personal information, facial photograph, fingerprint data

[0954] Output: Encrypted transmitted data

[0955] Specific operation: The device encrypts and sends data to the server using the HTTPS protocol.

[0956] Step 5:

[0957] The server generates a digital twin corresponding to the user based on the received data.

[0958] Input: Encrypted personal information and biometric information

[0959] Output: Generated data for the digital twin

[0960] Specific operation: The server executes the digital twin generation algorithm and creates a new entry in the database.

[0961] Step 6:

[0962] The server links and stores the purchased event ticket information to the generated digital twin.

[0963] Input: Digital twin data and purchased ticket information

[0964] Output: Database entries where ticket information is stored in the digital twin.

[0965] Specific operation: The server accesses the database and adds and saves ticket information to the digital twin entry.

[0966] Step 7:

[0967] The server sends a purchase completion notification to the user's email address.

[0968] Input: User's email address and purchase completion information

[0969] Output: Purchase completion notification email

[0970] Specific operation: The server generates and sends a purchase completion notification email using the email sending API.

[0971] Step 8:

[0972] The emotion engine analyzes received facial photos and voice data to recognize the user's emotional state.

[0973] Input: User's facial photo and voice data

[0974] Output: User's emotional state

[0975] Specific operation: An emotion recognition algorithm is executed, and as a result, an emotional state (e.g., joy, surprise, etc.) is generated.

[0976] Step 9:

[0977] The server stores the analyzed emotional information, associating it with a digital twin.

[0978] Input: Analyzed emotional information and digital twin

[0979] Output: Digital twin data with associated emotional information

[0980] Specific operation: The server accesses the database and adds and saves emotional information to the digital twin entry.

[0981] Entry processing steps

[0982] Step 1:

[0983] The user arrives at the venue entrance on the day of the event.

[0984] Input: User's actions (arrived at the venue entrance to participate in the event)

[0985] Output: Start of admission procedure

[0986] Specific action: The user travels to the event venue and arrives at the designated entrance.

[0987] Step 2:

[0988] The entry terminal prompts the user to perform facial recognition or fingerprint authentication.

[0989] Input: System-side authentication start command

[0990] Output: Display of authentication screen

[0991] Specific action: The device's display lights up and prepares for facial recognition or fingerprint recognition.

[0992] Step 3:

[0993] The user either takes a photo of their face or performs a fingerprint scan.

[0994] Input: User's facial photo or fingerprint

[0995] Output: Acquired biometric information

[0996] Specific actions: The user faces the camera or places their finger on the fingerprint sensor.

[0997] Step 4:

[0998] The device sends the acquired biometric information to the server.

[0999] Input: Biometric information (facial photo, fingerprint data)

[1000] Output: Encrypted transmitted data

[1001] Specific operation: The device encrypts and sends data to the server using the HTTPS protocol.

[1002] Step 5:

[1003] The server compares the received biometric information with the digital twin's identity verification information.

[1004] Input: Received biometric information, digital twin identity verification information

[1005] Output: Matching result (success / failure)

[1006] Specific operation: The server executes a biometric matching algorithm to check if there is a match.

[1007] Step 6:

[1008] If the server finds that the verification is successful, it will send an access permission message to the terminal.

[1009] Input: Data for successful matching

[1010] Output: Entry permission message

[1011] Specific operation: Based on the authentication result, the server generates an entry permission message as the next step and sends it to the terminal.

[1012] Step 7:

[1013] The device displays a message granting the user permission to enter.

[1014] Input: Entry permission message

[1015] Output: Screen display of entry permission

[1016] Specific actions: The terminal displays an entry permission message on its screen and performs actions such as opening the gate.

[1017] Step 8:

[1018] The emotion engine re-analyzes the user's latest facial photos and voice data and notifies event staff as needed.

[1019] Input: Latest facial photo and voice data

[1020] Output: Analysis results and notifications

[1021] Specific operation: The emotion engine re-analyzes the emotional state and notifies the event staff of the results via the server.

[1022] Step 9:

[1023] Event staff will receive notifications and take appropriate action for users.

[1024] Input: Notification from emotion engine

[1025] Output: Appropriate response (e.g., guidance and assistance to the user)

[1026] Specific actions: Event staff will check notifications from the emotion engine and respond to users.

[1027] By following these steps, the system can prevent the misuse of tickets and provide personalized services tailored to the user's emotional state.

[1028] (Application Example 2)

[1029] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1030] Conventional admission management systems focused on preventing ticket resale and verifying identity, but they did not address individual customer needs or provide services based on emotional states. Furthermore, they did not utilize customer purchase history in physical stores to provide services. This invention solves these problems, enabling the provision of services tailored to individual customer needs and considering emotional states at event venues and physical stores.

[1031] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring facial photographs and fingerprint data as identity verification information, means for associating and storing the acquired facial photographs and fingerprint data with the digital twin, input means for acquiring the purchaser's facial photograph or fingerprint data upon entry to the event venue, means for comparing the acquired facial photograph or fingerprint data with the identity verification information of the digital twin, means for granting entry if identity verification is successful, means for acquiring information and emotional state from the digital twin and displaying visual information when a customer enters a physical store, means for performing emotion analysis based on facial photographs and displaying the results, and means for acquiring the customer's past purchase history and preferences and suggesting recommended products. This makes it possible to provide services tailored to the individual needs of customers and services based on their emotional state at event venues and physical stores.

[1032] A "digital twin" is a virtual entity that stores and manages the personal information, purchase history, and emotional state of a real person in digital format.

[1033] An "emotion engine" is a software or hardware system that analyzes input facial images and voice data to recognize and evaluate a person's emotional state.

[1034] "Identity verification upon entry" refers to the process of verifying an individual's identity by comparing their actual face and fingerprints with pre-registered facial photographs and fingerprint data when entering a facility such as an event venue.

[1035] A "physical store" is a physical shop where goods are sold or services are provided in person.

[1036] "Purchase history" refers to a record of information about products and services that a customer has purchased in the past.

[1037] "Emotional state" refers to an individual's psychological and emotional condition, including specific emotional states such as joy, confusion, and tension.

[1038] A "recommended product" is a product that is selected and suggested based on the customer's past purchase history and preferences.

[1039] "Facial recognition" is a technology that analyzes captured facial images to identify who the person is.

[1040] "Smart glasses" are glasses-type wearable devices that incorporate a display, camera, and various sensors, enabling them to display, acquire, and process information.

[1041] "Visual information" refers to digital information in a form that can be visually confirmed, such as a customer's facial photograph, purchase history, and emotional state.

[1042] This invention is a system that generates a digital twin corresponding to a purchaser, links it with event ticket information, and acquires the customer's emotional state using an emotion engine, thereby enhancing service provision at physical stores and event venues. This system acquires and processes various data using applications installed on smart glasses, smartphones, robots, etc.

[1043] Hardware and software to use

[1044] server:

[1045] AWS Aurora: Used for database management.

[1046] TensorFlow: A machine learning library used to implement sentiment analysis engines.

[1047] nVIDIA Jetson TX2: Hardware used for facial recognition and emotion analysis processing.

[1048] Terminal:

[1049] Smart glasses: Acquire customer facial photos and emotional states and display visual information.

[1050] Smartphone: Used for purchasing tickets and entering and submitting identity verification information.

[1051] Event venue authentication device: Used to obtain facial images and fingerprint data upon entry.

[1052] Specific processing of the system

[1053] In trading company operations, the system operates as follows:

[1054] Digital Twin Generation

[1055] 1. The server receives the user's facial photograph and fingerprint data and generates a digital twin of the person based on this data. The digital twin is linked to event ticket information and stored in a database along with personal information.

[1056] Providing services at physical stores

[1057] 1. When a user enters a physical store, smart glasses or a robot's camera captures a photo of their face and sends it to a server.

[1058] 2. The server analyzes the acquired facial photograph and retrieves the corresponding digital twin information and past purchase history.

[1059] 3. The device (smart glasses or robot) displays the acquired digital twin information and emotional state, and takes appropriate action based on the current emotional state.

[1060] Emotional analysis and product recommendations

[1061] 1. The emotion engine analyzes the user's facial expression data to determine their current emotional state.

[1062] 2. Based on the assessment results, the server references the customer's preferences and past purchase history to generate a list of recommended products.

[1063] 3. The terminal displays recommended products, and staff members make product suggestions based on that information.

[1064] Specific example

[1065] For example, when a customer enters an event venue, the process works as follows:

[1066] 1. The user arrives at the entrance of the event venue and faces the authentication device.

[1067] 2. The authentication device acquires a facial image and sends it to the server.

[1068] 3. The server verifies the digital twin and grants access if the identity verification is successful.

[1069] 4. Simultaneously, the emotion engine analyzes the user's facial image and notifies the staff of their emotional state.

[1070] Example of a prompt

[1071] Example prompt: "Please provide specific scenarios for analyzing a customer's emotional state and responding accordingly. For example, please describe in detail how to handle a situation where the customer is confused."

[1072] Thus, by utilizing digital twins and an emotion engine, the present invention enables the provision of more personalized and advanced services at event venues and physical stores.

[1073] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1074] Step 1:

[1075] The user launches a ticket purchase application on their smartphone and enters personal information such as their name, email address, facial photo, and fingerprint data.

[1076] Input: Name, email address, photo, fingerprint data

[1077] Data processing and calculation: Converting personal information into an appropriate format for storage in a database.

[1078] Output: Personal information in a saved format

[1079] Step 2:

[1080] The device sends the personal information it has acquired to the server.

[1081] Input: Personal information entered on a smartphone

[1082] Data processing and calculation: Encryption of personal information, network transmission

[1083] Output: Encrypted personal information

[1084] Step 3:

[1085] The server generates a digital twin based on the personal information it receives and links it to the event ticket information.

[1086] Input: Encrypted personal information

[1087] Data processing and computation: Decryption of information, generation of digital twins, linking of ticket information.

[1088] Output: Ticket information linked to the generated digital twin

[1089] Step 4:

[1090] The server sends a purchase completion notification to the user's email address.

[1091] Input: Completed purchase information

[1092] Data processing and calculation: Email notification generation

[1093] Output: Purchase completion notification email

[1094] Step 5:

[1095] On the day of the event, users will face an authentication device at the venue entrance to have their face photographed.

[1096] Input: Current photo

[1097] Data Processing and Calculation: Face Detection and Feature Point Extraction using Face Recognition Algorithms

[1098] Output: Feature point data from facial photographs

[1099] Step 6:

[1100] The device sends the acquired facial image to a server for comparison with a digital twin.

[1101] Input: Feature point data from a facial photograph

[1102] Data processing and calculation: Data encryption, identity verification using matching algorithms.

[1103] Output: Identity Verification Result

[1104] Step 7:

[1105] If the server successfully verifies the user's identity, it will send an access permission message to the device.

[1106] Input: Identity verification result

[1107] Data processing and calculation: Processing of authentication results, message generation.

[1108] Output: Entry permission message

[1109] Step 8:

[1110] When a user enters a physical store, the smart glasses capture a photo of their face and send it to a server.

[1111] Input: Current photo

[1112] Data processing and calculation: Face detection using face recognition algorithms, generation of visual information

[1113] Output: Visual information

[1114] Step 9:

[1115] The server analyzes the acquired facial photographs to obtain corresponding digital twin information and purchase history.

[1116] Input: Facial photograph, digital twin database

[1117] Data processing and calculation: Feature point analysis of facial images, purchase history extraction.

[1118] Output: Digital twin information, purchase history

[1119] Step 10:

[1120] The emotion engine analyzes the user's facial expression data to determine their current emotional state.

[1121] Input: Facial expression data

[1122] Data processing and calculation: Emotion determination using facial expression analysis algorithms

[1123] Output: Emotional state data

[1124] Step 11:

[1125] The server generates a list of recommended products based on the emotion assessment results and displays them on the smart glasses.

[1126] Input: Emotional state data, purchase history

[1127] Data processing and calculation: Execution of recommended product algorithms, list generation.

[1128] Output: Recommended product list

[1129] Step 12:

[1130] The terminal allows staff to suggest products to customers based on recommended items.

[1131] Input: Recommended product list

[1132] Data processing and calculation: Generation and display of proposed information

[1133] Output: Product suggestion message

[1134] This process allows customers to enjoy personalized service at physical stores and event venues.

[1135] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1136] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1137] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1138] [Third Embodiment]

[1139] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1140] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1141] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1142] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1143] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1144] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1145] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1146] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1147] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1148] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1149] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1150] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1151] Modes for carrying out the invention

[1152] This invention is a system that prevents ticket resale by generating a digital twin corresponding to the purchaser and linking event ticket information to that digital twin. A specific embodiment of this system is shown below.

[1153] System Configuration

[1154] This system primarily consists of a server, user terminals, and event venue entry terminals. The server is responsible for generating and managing digital twins, linking event ticket information, and storing and verifying identity information. User terminals are devices used to input personal and identity information when purchasing tickets and to communicate with the server. Event venue entry terminals are devices used to verify the user's identity upon entry.

[1155] Ticket purchase processing

[1156] When a user wishes to purchase an event ticket, the device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into the device. The collected personal information is sent from the device to a server, which generates a digital twin based on the received information. The generated digital twin is linked to the information of the purchased event ticket.

[1157] Entry process

[1158] On the day of the event, a user arrives at the event venue and wishes to enter. A terminal (authentication device) installed at the entrance requests facial recognition or fingerprint authentication from the user. When the user faces the authentication device or performs a fingerprint scan, the acquired facial image or fingerprint data is sent from the terminal to the server. The server compares the received biometric information with the digital twin information, and if they match, entry is permitted. The server sends an entry permission message to the terminal, and the terminal displays the entry permission to the user. If they do not match, the server sends an entry denial message, and the terminal displays this to the user.

[1159] Specific example

[1160] As a concrete example, I will explain the process of purchasing event tickets and the entry process on the day of the event.

[1161] Example of purchasing tickets

[1162] When a user purchases event tickets using their smartphone, the device (smartphone app) prompts the user to take a photo of their face and perform a fingerprint scan. The device sends this information to a server, which generates a digital twin and stores the linked ticket information. Finally, the server sends a purchase completion notification to the user's email address.

[1163] Example of entry

[1164] On the day of the event, when a user arrives at the venue entrance, an authentication terminal will perform facial recognition or fingerprint authentication. The acquired facial image or fingerprint data is sent to a server and compared with a digital twin. If the server successfully matches the data, entry permission is issued, and the terminal displays a message granting entry to the user. If the matching fails, the terminal displays an entry denial message.

[1165] In this way, this system can effectively prevent ticket resale and ensure that legitimate purchasers can attend the event.

[1166] The following describes the processing flow.

[1167] Ticket purchase processing

[1168] Step 1:

[1169] A user wishes to purchase event tickets and opens the purchase screen on their device.

[1170] Step 2:

[1171] The device displays an interface prompting the user to enter their name, email address, facial photo, and fingerprint data.

[1172] Step 3:

[1173] The user provides personal information (name, email address), takes a photo of their face, and performs a fingerprint scan.

[1174] Step 4:

[1175] The device sends collected personal information, facial images, and fingerprint data to the server. This data is encrypted using the SSL / TLS protocol.

[1176] Step 5:

[1177] The server analyzes the personal information it receives and generates a digital twin of the user.

[1178] Step 6:

[1179] The server links event ticket information (event name, date and time, seat information, etc.) to the generated digital twin and stores it in a database.

[1180] Step 7:

[1181] The server sends a ticket purchase completion notification to the user's email address.

[1182] Entry process

[1183] Step 1:

[1184] The user arrives at the event venue and heads towards the authentication terminal located at the entrance.

[1185] Step 2:

[1186] The device displays a message requesting the user to authenticate using facial recognition or fingerprint recognition.

[1187] Step 3:

[1188] The user faces the authentication terminal or performs a fingerprint scan.

[1189] Step 4:

[1190] The device sends facial images and fingerprint data it has acquired to the server. This data is encrypted using the SSL / TLS protocol.

[1191] Step 5:

[1192] The server compares the received biometric information with the information in the digital twin.

[1193] Step 6:

[1194] The server checks the matching results and issues an entry permit if there is a match. If the matching is unsuccessful, it decides to deny entry.

[1195] Step 7:

[1196] The server sends the verification result to the terminal. This includes a permission message if successful, and a rejection message if unsuccessful.

[1197] Step 8:

[1198] The device displays the verification result to the user. If an permission message is displayed, the user is allowed to enter. If a denial message is displayed, the user is prompted to try authenticating again or to go through another procedure.

[1199] The above processing steps create a system that prevents ticket resale and ensures that legitimate purchasers can attend the event.

[1200] (Example 1)

[1201] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1202] Currently, illegal ticket reselling is a problem at many events. This makes it difficult for legitimate buyers to obtain tickets at the regular price. Furthermore, reselling can make identity verification difficult at the entrance, potentially weakening security. To address this, a system is needed that ensures that the original purchaser can enter the event venue while also preventing ticket reselling.

[1203] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1204] In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring image data and biometric data as identity verification information, means for associating and storing the acquired image data and biometric data with the digital twin, input means for acquiring the purchaser's image data or biometric data upon entry to the event venue, means for comparing the acquired image data or biometric data with the identity verification information of the digital twin, means for granting entry if identity verification is successful, and means for generating and storing the digital twin and associated ticket information using an external computing service. This prevents fraudulent ticket resale and enables legitimate purchasers to enter the event with security ensured.

[1205] A "purchaser" is a person who has purchased an event ticket and possesses certain rights.

[1206] A "digital twin" is a virtual digital profile generated based on the purchaser's personal information and biometric data.

[1207] "Event ticket information" refers to data concerning the information and rights necessary to participate in a specific event.

[1208] "Identity verification information" refers to information used to identify the purchaser, and specifically includes image data and biometric data.

[1209] "Image data" refers to data that digitally represents visual information, such as a photograph of the buyer's face.

[1210] "Biometric data" refers to data that includes unique physical characteristics used for purchaser fingerprints and facial recognition.

[1211] "Input means" refers to a device or equipment used to input the purchaser's image data or biometric data at the event venue.

[1212] "Means of matching" refers to methods for comparing acquired image data or biometric data with information from a digital twin to confirm whether they match.

[1213] "Means of granting entry" refers to the means by which a purchaser can enter the event venue if their identity verification is successful.

[1214] "External computing services" refer to external cloud services or computing resources used to generate and store digital twins and associated ticket information.

[1215] This invention is a system that prevents ticket resale and ensures that legitimate purchasers can participate in events. This system primarily includes a server, user terminals, and entry terminals for event facilities. Specific embodiments are described below.

[1216] System Configuration

[1217] This system consists of three main hardware components: a server, user terminals (smartphones, PCs, etc.), and event facility entry terminals (facial recognition cameras, fingerprint scanners, etc.).

[1218] Server: Plays a central role in generating and managing digital twins, linking event ticket information, and storing and verifying identity information. The software used will utilize external computing services (e.g., cloud services) and database systems (e.g., relational database systems (RDBMS)).

[1219] User's device: This includes smartphones and PCs, which are devices used to enter personal information and identity verification information when purchasing tickets and to communicate with the server. The software used includes dedicated applications and web browsers.

[1220] Event venue entry terminals: These are devices used for facial recognition and fingerprint authentication, thereby managing entry to event venues. The hardware used includes facial recognition cameras and fingerprint scanners.

[1221] Ticket purchase processing

[1222] When a user wishes to purchase event tickets, the device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into the device. Based on this information, the device sends the necessary data to the server. The server generates a digital twin based on the transmitted personal information and links the purchased event ticket information to the generated digital twin. In this process, the server uses an external computing service to generate the digital twin and stores the saved information in a database system.

[1223] Specific examples of ticket purchases

[1224] In the specific ticket purchase process, the user purchases event tickets using their smartphone. The smartphone app prompts the user to take a photo of their face and perform a fingerprint scan. The device sends this information to a server, which generates a digital twin and stores the linked ticket information using an external computing service. Finally, the server sends a purchase completion notification to the user's email address.

[1225] Example of a prompt

[1226] To purchase event tickets, please enter the following information: name, email address, photo, and fingerprint data. After entering the information, please press the "Submit" button.

[1227] Entry process

[1228] On the day of the event, a user arrives at the event venue and requests entry. An authentication device installed at the entry terminal requests facial recognition or fingerprint authentication from the user. When the user faces the authentication device or performs a fingerprint scan, the acquired facial image or fingerprint data is sent from the entry terminal to the server. The server compares the received biometric information with the information in the digital twin. If they match, an entry permission message is sent to the terminal and displayed to the user. If the matching fails, an entry denial message is sent and displayed to the user on the terminal.

[1229] Specific examples of entry

[1230] On the day of the event, the user goes to the venue entrance. The entry terminal (authentication device) performs facial recognition or fingerprint authentication. The facial image or fingerprint data is sent to the server and compared with a pre-generated digital twin. If the server successfully matches the user, an entry permission message is issued, and the terminal displays a message granting the user entry. If the matching fails, an entry denial message is displayed.

[1231] Example of a prompt

[1232] "Please use facial recognition or fingerprint authentication to enter. Authentication is in progress, so please wait."

[1233] Thus, this system prevents ticket resale and ensures that legitimate purchasers can safely participate in events while maintaining security.

[1234] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1235] Step 1:

[1236] When a user wishes to purchase event tickets, the device displays a ticket purchase screen. Here, the user enters personal information such as their name, email address, facial photograph, and fingerprint data. This information is retrieved and used as input for the next step.

[1237] Input: Name, email address, facial photo, fingerprint data

[1238] Output: Prepared personal data

[1239] Specific actions:

[1240] Device: The UI for purchasing tickets will be displayed in a web browser or a dedicated app.

[1241] User: Enters their name and email address into a form, takes a photo of their face with the camera, and obtains fingerprint data with a fingerprint scanner.

[1242] Step 2:

[1243] The device sends the acquired personal information to the server. The server receives this information and generates a digital twin.

[1244] Input: Personal information data

[1245] Output: Digital twin generation request

[1246] Specific actions:

[1247] Terminal: Sends personal information to the server via AJAX requests or form submissions.

[1248] Server: Establishes an API endpoint to process received personal information and receives data.

[1249] Step 3:

[1250] The server generates a digital twin based on the personal information it receives. The server uses a relational database system to generate and store the digital twin for the purchaser. Simultaneously, event ticket information is also associated with this digital twin.

[1251] Input: Personal information data

[1252] Output: Digital twin and associated ticket information

[1253] Specific actions:

[1254] Server: Calls APIs of external computing services to generate digital twins from personal information.

[1255] Server: Stores the generated digital twin in an RDBMS and associates it with event ticket information.

[1256] Step 4:

[1257] The server generates a purchase completion notification and sends it to the user's email address. The email includes details of the purchased event and confirmation information such as a QR code.

[1258] Input: Digital twin and ticket information

[1259] Output: Purchase confirmation email

[1260] Specific actions:

[1261] Server: Uses an email sending service to send purchase completion notifications to users.

[1262] Server: The email content will include event details, purchase confirmation information, QR code, etc.

[1263] Step 5:

[1264] On the day of the event, users arrive at the venue entrance and authenticate at an entry terminal. The authentication device will request facial recognition or fingerprint authentication.

[1265] Input: None

[1266] Output: Facial image or fingerprint data (real-time)

[1267] Specific actions:

[1268] User: Face the authentication terminal or perform a fingerprint scan.

[1269] Device: A biometric authentication device is used to obtain facial images and fingerprint data.

[1270] Step 6:

[1271] The device sends the acquired facial image or fingerprint data to the server. The server receives this data and compares it with the information in the digital twin. If they match, entry is permitted; otherwise, it is denied.

[1272] Input: Real-time biometric data

[1273] Output: Entry permitted message or entry denied message

[1274] Specific actions:

[1275] Terminal: Sends acquired biometric data to the server.

[1276] Server: Matches biometric data with digital twin information and generates results.

[1277] Server: Sends the authentication result to the entry terminal.

[1278] Step 7:

[1279] The terminal receives a response from the server and displays a message to the user indicating whether entry is permitted or denied.

[1280] Input: Admission permission message or admission denial message

[1281] Output: User feedback

[1282] Specific actions:

[1283] Terminal: Receives responses from the server and displays the corresponding message.

[1284] User: Act according to the displayed message (either enter or be denied).

[1285] (Application Example 1)

[1286] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1287] Conventional access control systems carry the risk of loss or leakage of card keys or PINs, highlighting the need for improved security and convenience. Furthermore, while digital twin-based identity verification systems have been used to prevent resale of event tickets, there was a desire for their application to a wider range of uses. Therefore, there is a need for technology that enables highly accurate and secure identity verification and prevents unauthorized access in office and facility access control systems.

[1288] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1289] In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring facial photographs and fingerprint data as identity verification information, means for associating and storing the acquired facial photographs and fingerprint data with the digital twin, input means for acquiring a user's facial photograph or fingerprint data when entering or leaving an office or facility, means for comparing the acquired facial photograph or fingerprint data with the identity verification information of the digital twin, and means for granting entry or exit if identity verification is successful. This enables highly accurate and secure identity verification using digital twins in office and facility access control, and prevents unauthorized access.

[1290] A "digital twin" is a digital avatar corresponding to a specific individual, a virtual entity that possesses the individual's biometric information and related data.

[1291] "Event ticket information" refers to data provided to purchasers as permission to participate in a specific event, and includes information such as the ticket number, event date and time, and seat number.

[1292] "Personal identification information" refers to biometric information used to identify a specific person, such as a person's facial photograph or fingerprint data.

[1293] An "event venue" refers to the physical location or facility where a specific event takes place.

[1294] "Input means" refers to devices or interfaces used to acquire or input data or information.

[1295] An "access control system" is a system used to monitor and manage the entry and exit of people in a specific facility or office.

[1296] "Verification means" refers to technologies and processes for comparing acquired biometric information with information associated with a digital twin to confirm whether they match.

[1297] "Encrypted communication" is a technology used to securely send and receive data, and it is a means of preventing unauthorized access by third parties by transforming data using a specific algorithm.

[1298] This invention is a system that generates a digital twin for use as an access control system for offices and facilities, and performs highly accurate and secure identity verification by using facial photographs and fingerprint data as identity verification information. Specifically, it includes a server, a user terminal, and a device for access control (hereinafter referred to as the terminal).

[1299] System Configuration

[1300] This system uses the following hardware and software:

[1301] Hardware: Terminal devices including smartphones, fingerprint scanners, and cameras.

[1302] Software: OpenCV, requests, fingerprint_scanner library

[1303] Server Functions

[1304] The server includes the following measures:

[1305] 1. Means for generating a digital twin: A digital twin is generated based on the user's personal information.

[1306] 2. Method for linking event ticket information: Link event tickets and entry / exit management information to the generated digital twin.

[1307] 3. Means of storing and verifying identity information: Facial photographs and fingerprint data are stored in association with a digital twin and verified.

[1308] Device functions

[1309] The user's device includes the following features:

[1310] 1. Method of obtaining facial photographs and fingerprint data: Identity verification information is obtained using a smartphone camera and fingerprint reader.

[1311] 2. Data transmission method: The acquired data is sent to the server.

[1312] Access control device functions

[1313] The terminal (access control device) is installed at the entrance of the office or facility and has the following functions:

[1314] 1. Input method: Acquire facial photographs and fingerprint data.

[1315] 2. Verification method: The acquired data is sent to a server and compared with the digital twin.

[1316] 3. Entry / Exit Permission Method: Entry and exit will be permitted only after successful identity verification.

[1317] Example of operation

[1318] Program processing

[1319] The basic processing flow of this system is as follows:

[1320] 1. The user takes a photo of their face with their smartphone camera.

[1321] 2. Obtain fingerprint data using a fingerprint scanner.

[1322] 3. The acquired data is sent to the server to generate and store a digital twin.

[1323] 4. When a user enters or leaves an office or facility, the installed terminal acquires a facial photograph and fingerprint data and sends it to the server.

[1324] 5. The server verifies the biometric information and returns the result to the terminal.

[1325] 6. If they match, entry and exit will be permitted; otherwise, a warning will be displayed.

[1326] Examples

[1327] Employee A enters the office in the morning and follows the following prompts:

[1328] 1. "Please take a photo of your face with your smartphone camera."

[1329] 2. "Please scan your fingerprints."

[1330] 3. "We are checking the authentication results. Please wait a moment."

[1331] The server compares the received information with the digital twin and grants access. During this process, encrypted communication is used throughout the entire system to ensure data security.

[1332] In this way, the invention provides highly accurate identity verification using digital twins, enabling secure access control for offices and facilities.

[1333] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1334] Step 1:

[1335] The user takes a photo of their face using their smartphone.

[1336] Input: User's face image.

[1337] Output: Face image file (e.g., face_image.jpg).

[1338] The user activates their smartphone camera and points it at their face to take a picture. The captured photo is saved as a file on the smartphone. This photo must be of high quality as it will be used in a later authentication process.

[1339] Step 2:

[1340] The user obtains fingerprint data using a fingerprint scanner.

[1341] Input: User's fingerprint.

[1342] Output: Fingerprint data.

[1343] The user places their finger on a fingerprint scanner connected to their smartphone. The scanner reads the fingerprint data and stores it digitally. This data is also used in subsequent authentication processes.

[1344] Step 3:

[1345] The device sends facial image and fingerprint data to the server.

[1346] Input: facial photo file, fingerprint data.

[1347] Output: Data transfer to server complete.

[1348] The device (smartphone) uses an internet connection to send the acquired facial image and fingerprint data to the server. This data is transmitted securely using encrypted communication.

[1349] Step 4:

[1350] The server generates a digital twin and links it with facial image and fingerprint data.

[1351] Input: facial photo file, fingerprint data.

[1352] Output: Digital twin.

[1353] The server generates a digital twin corresponding to the user based on the received facial photograph and fingerprint data. This digital twin is associated with biometric information for identity verification.

[1354] Step 5:

[1355] The server stores event ticket information and entry / exit management information linked to a digital twin.

[1356] Input: Event ticket information or entry / exit management information.

[1357] Output: Information linked to the digital twin.

[1358] The server links pre-registered event ticket information and access control information to the generated digital twin. This allows the digital twin to hold access permission information for specific events or facilities.

[1359] Step 6:

[1360] When users enter or leave an office or facility, the installed terminals acquire facial images and fingerprint data.

[1361] Input: User's facial photo, fingerprint data.

[1362] Output: Acquired biometric information.

[1363] The user stands in front of the access control device, and a camera and fingerprint scanner are used to capture a facial image and fingerprint. This data is collected in real time.

[1364] Step 7:

[1365] The device sends the acquired data to the server and requests identity verification.

[1366] Input: Acquired facial photograph and fingerprint data.

[1367] Output: Sending a request for identity verification.

[1368] The device sends the collected facial image and fingerprint data to the server and requests that it be matched with the information in the digital twin.

[1369] Step 8:

[1370] The server compares the acquired biometric information with the digital twin information.

[1371] Input: Acquired facial photograph, fingerprint data, and biometric information from the digital twin.

[1372] Output: Identity verification result (success or failure).

[1373] The server compares the received biometric information with a digital twin. If they match, the identity verification is considered successful.

[1374] Step 9:

[1375] The server sends the identity verification result to the device.

[1376] Input: Identity verification result.

[1377] Output: Notification of results to the terminal.

[1378] The server returns the verification result to the terminal, notifying it whether entry or exit is permitted or denied.

[1379] Step 10:

[1380] The device will allow or deny entry / exit based on the identity verification results.

[1381] Input: Identity verification result.

[1382] Output: Display of permission or denial of entry / exit.

[1383] The terminal displays the results received from the server, and if identity verification is successful, it allows entry or exit. If there is a mismatch, it displays a warning and denies entry or exit.

[1384] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1385] Modes for carrying out the invention

[1386] This invention relates to a system that generates a digital twin corresponding to a purchaser and links event ticket information to that digital twin. Furthermore, by combining it with an emotion engine, the system can recognize the user's emotions and provide services accordingly.

[1387] System Configuration

[1388] This system primarily consists of a server, user terminals, an emotion engine, and event venue entry terminals. The server is responsible for generating and managing digital twins, linking event ticket information, and storing and verifying identity information. User terminals are devices used to input personal and identity information when purchasing tickets and to communicate with the server. The emotion engine analyzes the user's facial image and voice data to recognize their emotional state. Event venue entry terminals are devices used to verify the user's identity upon entry.

[1389] Ticket purchase processing

[1390] When a user wishes to purchase an event ticket, the device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into the device. The collected personal information is sent from the device to a server, which generates a digital twin based on the received information. The generated digital twin is linked to the information of the purchased event ticket.

[1391] Acquisition and storage of emotional information

[1392] The user's facial image and voice data are analyzed by an emotion engine to recognize the user's emotional state. The acquired emotional information is sent to a server, associated with a digital twin, and stored in a database. This information is also used as a reference when the user uses the system again.

[1393] Entry process

[1394] On the day of the event, a user arrives at the event venue and requests entry. A terminal (authentication device) installed at the entrance requests facial recognition or fingerprint authentication from the user. When the user faces the authentication device or performs a fingerprint scan, the acquired facial image or fingerprint data is sent from the terminal to a server. The server compares the received biometric information with the digital twin information, and if they match, entry is permitted.

[1395] Emotion-based responses

[1396] Based on facial images and voice data acquired upon entry, the emotion engine re-analyzes the user's emotional state. For example, if the user is nervous, the system can relay this information to the guidance staff and prompt them to take appropriate action.

[1397] Specific example

[1398] As concrete examples, we will explain the process of purchasing event tickets, the entry process on the day of the event, and the management of emotional information.

[1399] Example of purchasing tickets

[1400] When a user purchases an event ticket using their smartphone, the device (smartphone app) prompts the user to take a photo of their face and perform a fingerprint scan. The device sends this information to a server, which generates a digital twin and stores the linked ticket information. Finally, the server sends a purchase completion notification to the user's email address. During the purchase, the user's facial image and voice data are analyzed by an emotion engine, and emotion information is added to the digital twin.

[1401] Example of entry

[1402] On the day of the event, when a user arrives at the venue entrance, an authentication terminal performs facial recognition or fingerprint authentication. The acquired facial image or fingerprint data is sent to a server and compared with a digital twin. If the server successfully matches the data, entry permission is issued, and the terminal displays a message granting entry to the user. Simultaneously, an emotion engine analyzes the user's emotional state and can provide necessary instructions to the event staff.

[1403] In this way, this system prevents ticket resale, ensures that the purchaser is indeed able to participate in the event, and provides services tailored to the user's emotional state.

[1404] The following describes the processing flow.

[1405] Ticket purchase processing

[1406] Step 1:

[1407] A user wishes to purchase event tickets and opens the purchase screen on their device.

[1408] Step 2:

[1409] The device displays an interface prompting the user to enter their name, email address, facial photo, and fingerprint data.

[1410] Step 3:

[1411] The user provides personal information (name, email address), takes a photo of their face, and performs a fingerprint scan.

[1412] Step 4:

[1413] The device sends collected personal information, facial images, and fingerprint data to the server. This data is encrypted using the SSL / TLS protocol.

[1414] Step 5:

[1415] The server analyzes the personal information it receives and generates a digital twin of the user.

[1416] Step 6:

[1417] The server links event ticket information (event name, date and time, seat information, etc.) to the generated digital twin and stores it in a database.

[1418] Step 7:

[1419] The server sends a ticket purchase completion notification to the user's email address.

[1420] Acquisition and storage of emotional information

[1421] Step 1:

[1422] The device is operated to display an interface for the user to provide a facial photograph and voice data.

[1423] Step 2:

[1424] The device collects the user's facial image and voice data, and sends that data to the emotion engine.

[1425] Step 3:

[1426] The emotion engine analyzes facial images and voice data to determine the user's emotional state.

[1427] Step 4:

[1428] The emotion engine sends the recognized emotion information to the server.

[1429] Step 5:

[1430] The server associates the received emotional information with a digital twin and stores it in a database.

[1431] Entry process

[1432] Step 1:

[1433] The user arrives at the event venue and heads towards the authentication terminal located at the entrance.

[1434] Step 2:

[1435] The device displays a message requesting the user to authenticate using facial recognition or fingerprint recognition.

[1436] Step 3:

[1437] The user faces the authentication terminal or performs a fingerprint scan.

[1438] Step 4:

[1439] The device sends facial images and fingerprint data it has acquired to the server. This data is encrypted using the SSL / TLS protocol.

[1440] Step 5:

[1441] The server compares the received biometric information with the information in the digital twin.

[1442] Step 6:

[1443] The server checks the matching results and issues an entry permit if there is a match. If the matching is unsuccessful, it decides to deny entry.

[1444] Step 7:

[1445] The server sends the verification result to the terminal. This includes a permission message if successful, and a rejection message if unsuccessful.

[1446] Step 8:

[1447] The device displays the verification result to the user. If an permission message is displayed, the user is allowed to enter. If a denial message is displayed, the user is prompted to try authenticating again or to go through another procedure.

[1448] Emotion-based responses

[1449] Step 1:

[1450] The device then collects the user's facial image and voice data again and sends them to the emotion engine.

[1451] Step 2:

[1452] The emotion engine analyzes the collected data to determine the user's emotional state.

[1453] Step 3:

[1454] The emotion engine sends the analysis results to the server.

[1455] Step 4:

[1456] Based on the emotional information it receives, the server sends necessary instructions to the event guidance staff.

[1457] Step 5:

[1458] The guidance staff will respond according to the user's emotional state.

[1459] This will prevent ticket resale, ensure that legitimate purchasers can attend the event, and create a system that provides services tailored to the user's emotional state.

[1460] (Example 2)

[1461] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1462] In modern event ticketing systems, resale and fraudulent use are major problems. Furthermore, services that take into account the emotional state of users when attending events are not adequately provided. In particular, utilizing user emotional information is expected to enable more personalized services, but this is difficult to achieve with current systems.

[1463] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1464] In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring biometric information as identity verification information, means for storing the acquired biometric information in association with the digital twin, means for recognizing the emotional state via a user terminal, means for storing the recognized emotional information in association with the digital twin, input means for acquiring the purchaser's biometric information upon entry to the event venue, means for comparing the acquired biometric information with the identity verification information of the digital twin, means for granting entry if identity verification is successful, and means for changing the service based on the emotional state acquired upon entry. This prevents fraudulent use of tickets and enables the provision of personalized services according to the user's emotional state.

[1465] - "Means for generating a digital twin corresponding to a purchaser" refers to a means of creating a virtual entity (digital twin) corresponding to a user in digital space based on the user's personal information and biometric information.

[1466] "Method for linking event ticket information to a generated digital twin" refers to a method for associating and storing information about event tickets purchased by a user with the generated digital twin.

[1467] "Means for acquiring biometric information" refers to means for acquiring biometric characteristics such as a user's facial photograph or fingerprint data.

[1468] "Means for associating and storing acquired biometric information with a digital twin" refers to means for associating acquired user biometric information with a generated digital twin and storing it in a database or other storage device.

[1469] "Means for recognizing emotional states via a user terminal" refers to methods for analyzing a user's facial image and voice data to recognize their emotional state. This includes an emotion recognition engine.

[1470] "Means for associating and storing recognized emotional information with a digital twin" refers to means for associating analyzed user emotional information with a digital twin and storing it in a database or other storage device.

[1471] "Means for comparing acquired biometric information with digital twin identity verification information" refers to methods for comparing and verifying biometric information acquired at event venues, etc., with digital twin identity verification information that has been stored in advance.

[1472] "Means of granting entry upon successful identity verification" refers to the means of granting entry to a user upon successful verification. Specifically, this includes actions such as issuing an electronic permit or opening the venue gates.

[1473] "Means of modifying services based on emotional state acquired upon entry" refers to means of personalizing the services provided based on the emotional state of the user recognized upon entry. For example, this could include sending notifications to staff to help alleviate the user's anxiety.

[1474] This invention is a system that generates a digital twin corresponding to a purchaser and links event ticket information to that digital twin. In addition, by combining it with an emotion engine, it can recognize the user's emotional state and provide services accordingly. The main components of this system include a server, a user terminal, an emotion engine, and an entry terminal at the event venue.

[1475] Components and their operation

[1476] The server will generate and manage digital twins, link event ticket information, and store and verify identity information. The hardware used will be cloud services (e.g., Amazon Web Services EC2 instances). The software used will include a database management system (e.g., AWS RDS), an email sending service (e.g., AWS SES), an emotion recognition API (e.g., Microsoft Azure's Emotion API), and a biometric authentication API (e.g., Google Cloud Platform's Biometric API).

[1477] The user's device is used to input personal information and biometric data (facial image and fingerprint data) required when purchasing tickets, and to communicate with the server. This includes smartphones, tablets, and personal computers.

[1478] The emotion engine analyzes the user's facial image and voice data to recognize their emotional state. The analysis results are sent to a server and stored in association with a digital twin.

[1479] Event venue entry terminals are devices used to verify the user's identity upon entry. These include facial recognition cameras and fingerprint scanners.

[1480] Ticket purchase processing

[1481] When a user wishes to purchase an event ticket, their device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into their device. The collected personal information is sent from the device to a server, which generates a digital twin based on the received information. The generated digital twin is linked to the information of the purchased event ticket. An emotion engine analyzes the facial photograph and voice data to recognize the user's emotional state and stores that information in association with the digital twin.

[1482] As a concrete example, when a user purchases concert tickets using their smartphone, the smartphone (device) instructs the user to take a photo of their face and perform a fingerprint scan. This information is sent from the device to a server, which generates a digital twin and stores the purchased ticket information in association with emotional information.

[1483] Entry process

[1484] On the day of the event, a user arrives at the venue and requests entry. An entry terminal (authentication device) installed at the entrance requests facial recognition or fingerprint authentication. When the user faces the authentication device or performs a fingerprint scan, the acquired biometric information is transmitted from the terminal to a server. The server compares the received information with the digital twin data and grants entry if they match. Simultaneously, an emotion engine analyzes the user's emotional state and notifies the event staff of the necessary information.

[1485] As a concrete example, when a user arrives at an event venue and undergoes facial recognition, the authentication terminal takes a picture of the user's face and sends it to the server. The server compares the received facial picture with a digital twin, and if they match, it displays an entry permission message on the terminal.

[1486] Examples of prompts for generative AI models

[1487] The following are examples of prompts for the generative AI model when implementing this system:

[1488] "Please explain the process for analyzing user emotional data at the time of concert ticket purchase and linking that emotional data to the corresponding digital twin."

[1489] "Please describe the system design for verifying user identity using biometric authentication information during the entry process on the day of the event, analyzing their emotional state, and notifying staff accordingly."

[1490] As described above, this system prevents the fraudulent use of tickets, ensures that the purchaser themselves can participate in the event, and provides personalized services tailored to the user's emotional state.

[1491] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1492] The processing flow of this system's program

[1493] Ticket purchase processing steps

[1494] Step 1:

[1495] The user accesses the event ticket purchase site using a device such as a smartphone.

[1496] Input: User actions (launching a web browser app or dedicated mobile app)

[1497] Output: Display of the purchase screen

[1498] Specific actions: The user enters the URL of the purchase site and accesses it, or launches the mobile app.

[1499] Step 2:

[1500] The device displays a ticket purchase screen and prompts the user to enter the necessary personal information (name, email address, photo, fingerprint data).

[1501] Input: System-side purchase screen display command

[1502] Output: Present an input form to the user.

[1503] Specific operation: The system generates HTML and application screens and displays forms for personal information input.

[1504] Step 3:

[1505] The user enters the required information into the provided form, takes a photo of their face, and performs a fingerprint scan.

[1506] Input: User's personal information (name, email address), facial photograph, fingerprints

[1507] Output: The input data is saved to the terminal and temporarily prepared.

[1508] Specific operation: The user takes a photo of their face using the camera and places their finger on the fingerprint sensor to obtain fingerprint data.

[1509] Step 4:

[1510] The device sends the entered personal information and biometric data to the server.

[1511] Input: User's personal information, facial photograph, fingerprint data

[1512] Output: Encrypted transmitted data

[1513] Specific operation: The device encrypts and sends data to the server using the HTTPS protocol.

[1514] Step 5:

[1515] The server generates a digital twin corresponding to the user based on the received data.

[1516] Input: Encrypted personal information and biometric information

[1517] Output: Generated data for the digital twin

[1518] Specific operation: The server executes the digital twin generation algorithm and creates a new entry in the database.

[1519] Step 6:

[1520] The server links and stores the purchased event ticket information to the generated digital twin.

[1521] Input: Digital twin data and purchased ticket information

[1522] Output: Database entries where ticket information is stored in the digital twin.

[1523] Specific operation: The server accesses the database and adds and saves ticket information to the digital twin entry.

[1524] Step 7:

[1525] The server sends a purchase completion notification to the user's email address.

[1526] Input: User's email address and purchase completion information

[1527] Output: Purchase completion notification email

[1528] Specific operation: The server generates and sends a purchase completion notification email using the email sending API.

[1529] Step 8:

[1530] The emotion engine analyzes received facial photos and voice data to recognize the user's emotional state.

[1531] Input: User's facial photo and voice data

[1532] Output: User's emotional state

[1533] Specific operation: An emotion recognition algorithm is executed, and as a result, an emotional state (e.g., joy, surprise, etc.) is generated.

[1534] Step 9:

[1535] The server stores the analyzed emotional information, associating it with a digital twin.

[1536] Input: Analyzed emotional information and digital twin

[1537] Output: Digital twin data with associated emotional information

[1538] Specific operation: The server accesses the database and adds and saves emotional information to the digital twin entry.

[1539] Entry processing steps

[1540] Step 1:

[1541] The user arrives at the venue entrance on the day of the event.

[1542] Input: User's actions (arrived at the venue entrance to participate in the event)

[1543] Output: Start of admission procedure

[1544] Specific action: The user travels to the event venue and arrives at the designated entrance.

[1545] Step 2:

[1546] The entry terminal prompts the user to perform facial recognition or fingerprint authentication.

[1547] Input: System-side authentication start command

[1548] Output: Display of authentication screen

[1549] Specific action: The device's display lights up and prepares for facial recognition or fingerprint recognition.

[1550] Step 3:

[1551] The user either takes a photo of their face or performs a fingerprint scan.

[1552] Input: User's facial photo or fingerprint

[1553] Output: Acquired biometric information

[1554] Specific actions: The user faces the camera or places their finger on the fingerprint sensor.

[1555] Step 4:

[1556] The device sends the acquired biometric information to the server.

[1557] Input: Biometric information (facial photo, fingerprint data)

[1558] Output: Encrypted transmitted data

[1559] Specific operation: The device encrypts and sends data to the server using the HTTPS protocol.

[1560] Step 5:

[1561] The server compares the received biometric information with the digital twin's identity verification information.

[1562] Input: Received biometric information, digital twin identity verification information

[1563] Output: Matching result (success / failure)

[1564] Specific operation: The server executes a biometric matching algorithm to check if there is a match.

[1565] Step 6:

[1566] If the server finds that the verification is successful, it will send an access permission message to the terminal.

[1567] Input: Data for successful matching

[1568] Output: Entry permission message

[1569] Specific operation: Based on the authentication result, the server generates an entry permission message as the next step and sends it to the terminal.

[1570] Step 7:

[1571] The device displays a message granting the user permission to enter.

[1572] Input: Entry permission message

[1573] Output: Screen display of entry permission

[1574] Specific actions: The terminal displays an entry permission message on its screen and performs actions such as opening the gate.

[1575] Step 8:

[1576] The emotion engine re-analyzes the user's latest facial photos and voice data and notifies event staff as needed.

[1577] Input: Latest facial photo and voice data

[1578] Output: Analysis results and notifications

[1579] Specific operation: The emotion engine re-analyzes the emotional state and notifies the event staff of the results via the server.

[1580] Step 9:

[1581] Event staff will receive notifications and take appropriate action for users.

[1582] Input: Notification from emotion engine

[1583] Output: Appropriate response (e.g., guidance and assistance to the user)

[1584] Specific actions: Event staff will check notifications from the emotion engine and respond to users.

[1585] By following these steps, the system can prevent the misuse of tickets and provide personalized services tailored to the user's emotional state.

[1586] (Application Example 2)

[1587] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1588] Conventional admission management systems focused on preventing ticket resale and verifying identity, but they did not address individual customer needs or provide services based on emotional states. Furthermore, they did not utilize customer purchase history in physical stores to provide services. This invention solves these problems, enabling the provision of services tailored to individual customer needs and considering emotional states at event venues and physical stores.

[1589] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring facial photographs and fingerprint data as identity verification information, means for associating and storing the acquired facial photographs and fingerprint data with the digital twin, input means for acquiring the purchaser's facial photograph or fingerprint data upon entry to the event venue, means for comparing the acquired facial photograph or fingerprint data with the identity verification information of the digital twin, means for granting entry if identity verification is successful, means for acquiring information and emotional state from the digital twin and displaying visual information when a customer enters a physical store, means for performing emotion analysis based on facial photographs and displaying the results, and means for acquiring the customer's past purchase history and preferences and suggesting recommended products. This makes it possible to provide services tailored to the individual needs of customers and services based on their emotional state at event venues and physical stores.

[1590] A "digital twin" is a virtual entity that stores and manages the personal information, purchase history, and emotional state of a real person in digital format.

[1591] An "emotion engine" is a software or hardware system that analyzes input facial images and voice data to recognize and evaluate a person's emotional state.

[1592] "Identity verification upon entry" refers to the process of verifying an individual's identity by comparing their actual face and fingerprints with pre-registered facial photographs and fingerprint data when entering a facility such as an event venue.

[1593] A "physical store" is a physical shop where goods are sold or services are provided in person.

[1594] "Purchase history" refers to a record of information about products and services that a customer has purchased in the past.

[1595] "Emotional state" refers to an individual's psychological and emotional condition, including specific emotional states such as joy, confusion, and tension.

[1596] A "recommended product" is a product that is selected and suggested based on the customer's past purchase history and preferences.

[1597] "Facial recognition" is a technology that analyzes captured facial images to identify who the person is.

[1598] "Smart glasses" are glasses-type wearable devices that incorporate a display, camera, and various sensors, enabling them to display, acquire, and process information.

[1599] "Visual information" refers to digital information in a form that can be visually confirmed, such as a customer's facial photograph, purchase history, and emotional state.

[1600] This invention is a system that generates a digital twin corresponding to a purchaser, links it with event ticket information, and acquires the customer's emotional state using an emotion engine, thereby enhancing service provision at physical stores and event venues. This system acquires and processes various data using applications installed on smart glasses, smartphones, robots, etc.

[1601] Hardware and software to use

[1602] server:

[1603] AWS Aurora: Used for database management.

[1604] TensorFlow: A machine learning library used to implement sentiment analysis engines.

[1605] nVIDIA Jetson TX2: Hardware used for facial recognition and emotion analysis processing.

[1606] Terminal:

[1607] Smart glasses: Acquire customer facial photos and emotional states and display visual information.

[1608] Smartphone: Used for purchasing tickets and entering and submitting identity verification information.

[1609] Event venue authentication device: Used to obtain facial images and fingerprint data upon entry.

[1610] Specific processing of the system

[1611] In trading company operations, the system operates as follows:

[1612] Digital Twin Generation

[1613] 1. The server receives the user's facial photograph and fingerprint data and generates a digital twin of the person based on this data. The digital twin is linked to event ticket information and stored in a database along with personal information.

[1614] Providing services at physical stores

[1615] 1. When a user enters a physical store, smart glasses or a robot's camera captures a photo of their face and sends it to a server.

[1616] 2. The server analyzes the acquired facial photograph and retrieves the corresponding digital twin information and past purchase history.

[1617] 3. The device (smart glasses or robot) displays the acquired digital twin information and emotional state, and takes appropriate action based on the current emotional state.

[1618] Emotional analysis and product recommendations

[1619] 1. The emotion engine analyzes the user's facial expression data to determine their current emotional state.

[1620] 2. Based on the assessment results, the server references the customer's preferences and past purchase history to generate a list of recommended products.

[1621] 3. The terminal displays recommended products, and staff members make product suggestions based on that information.

[1622] Specific example

[1623] For example, when a customer enters an event venue, the process works as follows:

[1624] 1. The user arrives at the entrance of the event venue and faces the authentication device.

[1625] 2. The authentication device acquires a facial image and sends it to the server.

[1626] 3. The server verifies the digital twin and grants access if the identity verification is successful.

[1627] 4. Simultaneously, the emotion engine analyzes the user's facial image and notifies the staff of their emotional state.

[1628] Example of a prompt

[1629] Example prompt: "Please provide specific scenarios for analyzing a customer's emotional state and responding accordingly. For example, please describe in detail how to handle a situation where the customer is confused."

[1630] Thus, by utilizing digital twins and an emotion engine, the present invention enables the provision of more personalized and advanced services at event venues and physical stores.

[1631] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1632] Step 1:

[1633] The user launches a ticket purchase application on their smartphone and enters personal information such as their name, email address, facial photo, and fingerprint data.

[1634] Input: Name, email address, photo, fingerprint data

[1635] Data processing and calculation: Converting personal information into an appropriate format for storage in a database.

[1636] Output: Personal information in a saved format

[1637] Step 2:

[1638] The device sends the personal information it has acquired to the server.

[1639] Input: Personal information entered on a smartphone

[1640] Data processing and calculation: Encryption of personal information, network transmission

[1641] Output: Encrypted personal information

[1642] Step 3:

[1643] The server generates a digital twin based on the personal information it receives and links it to the event ticket information.

[1644] Input: Encrypted personal information

[1645] Data processing and computation: Decryption of information, generation of digital twins, linking of ticket information.

[1646] Output: Ticket information linked to the generated digital twin

[1647] Step 4:

[1648] The server sends a purchase completion notification to the user's email address.

[1649] Input: Completed purchase information

[1650] Data processing and calculation: Email notification generation

[1651] Output: Purchase completion notification email

[1652] Step 5:

[1653] On the day of the event, users will face an authentication device at the venue entrance to have their face photographed.

[1654] Input: Current photo

[1655] Data Processing and Calculation: Face Detection and Feature Point Extraction using Face Recognition Algorithms

[1656] Output: Feature point data from facial photographs

[1657] Step 6:

[1658] The device sends the acquired facial image to a server for comparison with a digital twin.

[1659] Input: Feature point data from a facial photograph

[1660] Data processing and calculation: Data encryption, identity verification using matching algorithms.

[1661] Output: Identity Verification Result

[1662] Step 7:

[1663] If the server successfully verifies the user's identity, it will send an access permission message to the device.

[1664] Input: Identity verification result

[1665] Data processing and calculation: Processing of authentication results, message generation.

[1666] Output: Entry permission message

[1667] Step 8:

[1668] When a user enters a physical store, the smart glasses capture a photo of their face and send it to a server.

[1669] Input: Current photo

[1670] Data processing and calculation: Face detection using face recognition algorithms, generation of visual information

[1671] Output: Visual information

[1672] Step 9:

[1673] The server analyzes the acquired facial photographs to obtain corresponding digital twin information and purchase history.

[1674] Input: Facial photograph, digital twin database

[1675] Data processing and calculation: Feature point analysis of facial images, purchase history extraction.

[1676] Output: Digital twin information, purchase history

[1677] Step 10:

[1678] The emotion engine analyzes the user's facial expression data to determine their current emotional state.

[1679] Input: Facial expression data

[1680] Data processing and calculation: Emotion determination using facial expression analysis algorithms

[1681] Output: Emotional state data

[1682] Step 11:

[1683] The server generates a list of recommended products based on the emotion assessment results and displays them on the smart glasses.

[1684] Input: Emotional state data, purchase history

[1685] Data processing and calculation: Execution of recommended product algorithms, list generation.

[1686] Output: Recommended product list

[1687] Step 12:

[1688] The terminal allows staff to suggest products to customers based on recommended items.

[1689] Input: Recommended product list

[1690] Data processing and calculation: Generation and display of proposed information

[1691] Output: Product suggestion message

[1692] This process allows customers to enjoy personalized service at physical stores and event venues.

[1693] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1694] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1695] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1696] [Fourth Embodiment]

[1697] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1698] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1699] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1700] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1701] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1702] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1703] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1704] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1705] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1706] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1707] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1708] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1709] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1710] Modes for carrying out the invention

[1711] This invention is a system that prevents ticket resale by generating a digital twin corresponding to the purchaser and linking event ticket information to that digital twin. A specific embodiment of this system is shown below.

[1712] System Configuration

[1713] This system primarily consists of a server, user terminals, and event venue entry terminals. The server is responsible for generating and managing digital twins, linking event ticket information, and storing and verifying identity information. User terminals are devices used to input personal and identity information when purchasing tickets and to communicate with the server. Event venue entry terminals are devices used to verify the user's identity upon entry.

[1714] Ticket purchase processing

[1715] When a user wishes to purchase an event ticket, the device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into the device. The collected personal information is sent from the device to a server, which generates a digital twin based on the received information. The generated digital twin is linked to the information of the purchased event ticket.

[1716] Entry process

[1717] On the day of the event, a user arrives at the event venue and wishes to enter. A terminal (authentication device) installed at the entrance requests facial recognition or fingerprint authentication from the user. When the user faces the authentication device or performs a fingerprint scan, the acquired facial image or fingerprint data is sent from the terminal to the server. The server compares the received biometric information with the digital twin information, and if they match, entry is permitted. The server sends an entry permission message to the terminal, and the terminal displays the entry permission to the user. If they do not match, the server sends an entry denial message, and the terminal displays this to the user.

[1718] Specific example

[1719] As a concrete example, I will explain the process of purchasing event tickets and the entry process on the day of the event.

[1720] Example of purchasing tickets

[1721] When a user purchases event tickets using their smartphone, the device (smartphone app) prompts the user to take a photo of their face and perform a fingerprint scan. The device sends this information to a server, which generates a digital twin and stores the linked ticket information. Finally, the server sends a purchase completion notification to the user's email address.

[1722] Example of entry

[1723] On the day of the event, when a user arrives at the venue entrance, an authentication terminal will perform facial recognition or fingerprint authentication. The acquired facial image or fingerprint data is sent to a server and compared with a digital twin. If the server successfully matches the data, entry permission is issued, and the terminal displays a message granting entry to the user. If the matching fails, the terminal displays an entry denial message.

[1724] In this way, this system can effectively prevent ticket resale and ensure that legitimate purchasers can attend the event.

[1725] The following describes the processing flow.

[1726] Ticket purchase processing

[1727] Step 1:

[1728] A user wishes to purchase event tickets and opens the purchase screen on their device.

[1729] Step 2:

[1730] The device displays an interface prompting the user to enter their name, email address, facial photo, and fingerprint data.

[1731] Step 3:

[1732] The user provides personal information (name, email address), takes a photo of their face, and performs a fingerprint scan.

[1733] Step 4:

[1734] The device sends collected personal information, facial images, and fingerprint data to the server. This data is encrypted using the SSL / TLS protocol.

[1735] Step 5:

[1736] The server analyzes the personal information it receives and generates a digital twin of the user.

[1737] Step 6:

[1738] The server links event ticket information (event name, date and time, seat information, etc.) to the generated digital twin and stores it in a database.

[1739] Step 7:

[1740] The server sends a ticket purchase completion notification to the user's email address.

[1741] Entry process

[1742] Step 1:

[1743] The user arrives at the event venue and heads towards the authentication terminal located at the entrance.

[1744] Step 2:

[1745] The device displays a message requesting the user to authenticate using facial recognition or fingerprint recognition.

[1746] Step 3:

[1747] The user faces the authentication terminal or performs a fingerprint scan.

[1748] Step 4:

[1749] The device sends facial images and fingerprint data it has acquired to the server. This data is encrypted using the SSL / TLS protocol.

[1750] Step 5:

[1751] The server compares the received biometric information with the information in the digital twin.

[1752] Step 6:

[1753] The server checks the matching results and issues an entry permit if there is a match. If the matching is unsuccessful, it decides to deny entry.

[1754] Step 7:

[1755] The server sends the verification result to the terminal. This includes a permission message if successful, and a rejection message if unsuccessful.

[1756] Step 8:

[1757] The device displays the verification result to the user. If an permission message is displayed, the user is allowed to enter. If a denial message is displayed, the user is prompted to try authenticating again or to go through another procedure.

[1758] The above processing steps create a system that prevents ticket resale and ensures that legitimate purchasers can attend the event.

[1759] (Example 1)

[1760] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1761] Currently, illegal ticket reselling is a problem at many events. This makes it difficult for legitimate buyers to obtain tickets at the regular price. Furthermore, reselling can make identity verification difficult at the entrance, potentially weakening security. To address this, a system is needed that ensures that the original purchaser can enter the event venue while also preventing ticket reselling.

[1762] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1763] In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring image data and biometric data as identity verification information, means for associating and storing the acquired image data and biometric data with the digital twin, input means for acquiring the purchaser's image data or biometric data upon entry to the event venue, means for comparing the acquired image data or biometric data with the identity verification information of the digital twin, means for granting entry if identity verification is successful, and means for generating and storing the digital twin and associated ticket information using an external computing service. This prevents fraudulent ticket resale and enables legitimate purchasers to enter the event with security ensured.

[1764] A "purchaser" is a person who has purchased an event ticket and possesses certain rights.

[1765] A "digital twin" is a virtual digital profile generated based on the purchaser's personal information and biometric data.

[1766] "Event ticket information" refers to data concerning the information and rights necessary to participate in a specific event.

[1767] "Identity verification information" refers to information used to identify the purchaser, and specifically includes image data and biometric data.

[1768] "Image data" refers to data that digitally represents visual information, such as a photograph of the buyer's face.

[1769] "Biometric data" refers to data that includes unique physical characteristics used for purchaser fingerprints and facial recognition.

[1770] "Input means" refers to a device or equipment used to input the purchaser's image data or biometric data at the event venue.

[1771] "Means of matching" refers to methods for comparing acquired image data or biometric data with information from a digital twin to confirm whether they match.

[1772] "Means of granting entry" refers to the means by which a purchaser can enter the event venue if their identity verification is successful.

[1773] "External computing services" refer to external cloud services or computing resources used to generate and store digital twins and associated ticket information.

[1774] This invention is a system that prevents ticket resale and ensures that legitimate purchasers can participate in events. This system primarily includes a server, user terminals, and entry terminals for event facilities. Specific embodiments are described below.

[1775] System Configuration

[1776] This system consists of three main hardware components: a server, user terminals (smartphones, PCs, etc.), and event facility entry terminals (facial recognition cameras, fingerprint scanners, etc.).

[1777] Server: Plays a central role in generating and managing digital twins, linking event ticket information, and storing and verifying identity information. The software used will utilize external computing services (e.g., cloud services) and database systems (e.g., relational database systems (RDBMS)).

[1778] User's device: This includes smartphones and PCs, which are devices used to enter personal information and identity verification information when purchasing tickets and to communicate with the server. The software used includes dedicated applications and web browsers.

[1779] Event venue entry terminals: These are devices used for facial recognition and fingerprint authentication, thereby managing entry to event venues. The hardware used includes facial recognition cameras and fingerprint scanners.

[1780] Ticket purchase processing

[1781] When a user wishes to purchase event tickets, the device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into the device. Based on this information, the device sends the necessary data to the server. The server generates a digital twin based on the transmitted personal information and links the purchased event ticket information to the generated digital twin. In this process, the server uses an external computing service to generate the digital twin and stores the saved information in a database system.

[1782] Specific examples of ticket purchases

[1783] In the specific ticket purchase process, the user purchases event tickets using their smartphone. The smartphone app prompts the user to take a photo of their face and perform a fingerprint scan. The device sends this information to a server, which generates a digital twin and stores the linked ticket information using an external computing service. Finally, the server sends a purchase completion notification to the user's email address.

[1784] Example of a prompt

[1785] To purchase event tickets, please enter the following information: name, email address, photo, and fingerprint data. After entering the information, please press the "Submit" button.

[1786] Entry process

[1787] On the day of the event, a user arrives at the event venue and requests entry. An authentication device installed at the entry terminal requests facial recognition or fingerprint authentication from the user. When the user faces the authentication device or performs a fingerprint scan, the acquired facial image or fingerprint data is sent from the entry terminal to the server. The server compares the received biometric information with the information in the digital twin. If they match, an entry permission message is sent to the terminal and displayed to the user. If the matching fails, an entry denial message is sent and displayed to the user on the terminal.

[1788] Specific examples of entry

[1789] On the day of the event, the user goes to the venue entrance. The entry terminal (authentication device) performs facial recognition or fingerprint authentication. The facial image or fingerprint data is sent to the server and compared with a pre-generated digital twin. If the server successfully matches the user, an entry permission message is issued, and the terminal displays a message granting the user entry. If the matching fails, an entry denial message is displayed.

[1790] Example of a prompt

[1791] "Please use facial recognition or fingerprint authentication to enter. Authentication is in progress, so please wait."

[1792] Thus, this system prevents ticket resale and ensures that legitimate purchasers can safely participate in events while maintaining security.

[1793] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1794] Step 1:

[1795] When a user wishes to purchase event tickets, the device displays a ticket purchase screen. Here, the user enters personal information such as their name, email address, facial photograph, and fingerprint data. This information is retrieved and used as input for the next step.

[1796] Input: Name, email address, facial photo, fingerprint data

[1797] Output: Prepared personal data

[1798] Specific actions:

[1799] Device: The UI for purchasing tickets will be displayed in a web browser or a dedicated app.

[1800] User: Enters their name and email address into a form, takes a photo of their face with the camera, and obtains fingerprint data with a fingerprint scanner.

[1801] Step 2:

[1802] The device sends the acquired personal information to the server. The server receives this information and generates a digital twin.

[1803] Input: Personal information data

[1804] Output: Digital twin generation request

[1805] Specific actions:

[1806] Terminal: Sends personal information to the server via AJAX requests or form submissions.

[1807] Server: Establishes an API endpoint to process received personal information and receives data.

[1808] Step 3:

[1809] The server generates a digital twin based on the personal information it receives. The server uses a relational database system to generate and store the digital twin for the purchaser. Simultaneously, event ticket information is also associated with this digital twin.

[1810] Input: Personal information data

[1811] Output: Digital twin and associated ticket information

[1812] Specific actions:

[1813] Server: Calls APIs of external computing services to generate digital twins from personal information.

[1814] Server: Stores the generated digital twin in an RDBMS and associates it with event ticket information.

[1815] Step 4:

[1816] The server generates a purchase completion notification and sends it to the user's email address. The email includes details of the purchased event and confirmation information such as a QR code.

[1817] Input: Digital twin and ticket information

[1818] Output: Purchase confirmation email

[1819] Specific actions:

[1820] Server: Uses an email sending service to send purchase completion notifications to users.

[1821] Server: The email content will include event details, purchase confirmation information, QR code, etc.

[1822] Step 5:

[1823] On the day of the event, users arrive at the venue entrance and authenticate at an entry terminal. The authentication device will request facial recognition or fingerprint authentication.

[1824] Input: None

[1825] Output: Facial image or fingerprint data (real-time)

[1826] Specific actions:

[1827] User: Face the authentication terminal or perform a fingerprint scan.

[1828] Device: A biometric authentication device is used to obtain facial images and fingerprint data.

[1829] Step 6:

[1830] The device sends the acquired facial image or fingerprint data to the server. The server receives this data and compares it with the information in the digital twin. If they match, entry is permitted; otherwise, it is denied.

[1831] Input: Real-time biometric data

[1832] Output: Entry permitted message or entry denied message

[1833] Specific actions:

[1834] Terminal: Sends acquired biometric data to the server.

[1835] Server: Matches biometric data with digital twin information and generates results.

[1836] Server: Sends the authentication result to the entry terminal.

[1837] Step 7:

[1838] The terminal receives a response from the server and displays a message to the user indicating whether entry is permitted or denied.

[1839] Input: Admission permission message or admission denial message

[1840] Output: User feedback

[1841] Specific actions:

[1842] Terminal: Receives responses from the server and displays the corresponding message.

[1843] User: Act according to the displayed message (either enter or be denied).

[1844] (Application Example 1)

[1845] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1846] Conventional access control systems carry the risk of loss or leakage of card keys or PINs, highlighting the need for improved security and convenience. Furthermore, while digital twin-based identity verification systems have been used to prevent resale of event tickets, there was a desire for their application to a wider range of uses. Therefore, there is a need for technology that enables highly accurate and secure identity verification and prevents unauthorized access in office and facility access control systems.

[1847] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1848] In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring facial photographs and fingerprint data as identity verification information, means for associating and storing the acquired facial photographs and fingerprint data with the digital twin, input means for acquiring a user's facial photograph or fingerprint data when entering or leaving an office or facility, means for comparing the acquired facial photograph or fingerprint data with the identity verification information of the digital twin, and means for granting entry or exit if identity verification is successful. This enables highly accurate and secure identity verification using digital twins in office and facility access control, and prevents unauthorized access.

[1849] A "digital twin" is a digital avatar corresponding to a specific individual, a virtual entity that possesses the individual's biometric information and related data.

[1850] "Event ticket information" refers to data provided to purchasers as permission to participate in a specific event, and includes information such as the ticket number, event date and time, and seat number.

[1851] "Personal identification information" refers to biometric information used to identify a specific person, such as a person's facial photograph or fingerprint data.

[1852] An "event venue" refers to the physical location or facility where a specific event takes place.

[1853] "Input means" refers to devices or interfaces used to acquire or input data or information.

[1854] An "access control system" is a system used to monitor and manage the entry and exit of people in a specific facility or office.

[1855] "Verification means" refers to technologies and processes for comparing acquired biometric information with information associated with a digital twin to confirm whether they match.

[1856] "Encrypted communication" is a technology used to securely send and receive data, and it is a means of preventing unauthorized access by third parties by transforming data using a specific algorithm.

[1857] This invention is a system that generates a digital twin for use as an access control system for offices and facilities, and performs highly accurate and secure identity verification by using facial photographs and fingerprint data as identity verification information. Specifically, it includes a server, a user terminal, and a device for access control (hereinafter referred to as the terminal).

[1858] System Configuration

[1859] This system uses the following hardware and software:

[1860] Hardware: Terminal devices including smartphones, fingerprint scanners, and cameras.

[1861] Software: OpenCV, requests, fingerprint_scanner library

[1862] Server Functions

[1863] The server includes the following measures:

[1864] 1. Means for generating a digital twin: A digital twin is generated based on the user's personal information.

[1865] 2. Method for linking event ticket information: Link event tickets and entry / exit management information to the generated digital twin.

[1866] 3. Means of storing and verifying identity information: Facial photographs and fingerprint data are stored in association with a digital twin and verified.

[1867] Device functions

[1868] The user's device includes the following features:

[1869] 1. Method of obtaining facial photographs and fingerprint data: Identity verification information is obtained using a smartphone camera and fingerprint reader.

[1870] 2. Data transmission method: The acquired data is sent to the server.

[1871] Access control device functions

[1872] The terminal (access control device) is installed at the entrance of the office or facility and has the following functions:

[1873] 1. Input method: Acquire facial photographs and fingerprint data.

[1874] 2. Verification method: The acquired data is sent to a server and compared with the digital twin.

[1875] 3. Entry / Exit Permission Method: Entry and exit will be permitted only after successful identity verification.

[1876] Example of operation

[1877] Program processing

[1878] The basic processing flow of this system is as follows:

[1879] 1. The user takes a photo of their face with their smartphone camera.

[1880] 2. Obtain fingerprint data using a fingerprint scanner.

[1881] 3. The acquired data is sent to the server to generate and store a digital twin.

[1882] 4. When a user enters or leaves an office or facility, the installed terminal acquires a facial photograph and fingerprint data and sends it to the server.

[1883] 5. The server verifies the biometric information and returns the result to the terminal.

[1884] 6. If they match, entry and exit will be permitted; otherwise, a warning will be displayed.

[1885] Examples

[1886] Employee A enters the office in the morning and follows the following prompts:

[1887] 1. "Please take a photo of your face with your smartphone camera."

[1888] 2. "Please scan your fingerprints."

[1889] 3. "We are checking the authentication results. Please wait a moment."

[1890] The server compares the received information with the digital twin and grants access. During this process, encrypted communication is used throughout the entire system to ensure data security.

[1891] In this way, the invention provides highly accurate identity verification using digital twins, enabling secure access control for offices and facilities.

[1892] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1893] Step 1:

[1894] The user takes a photo of their face using their smartphone.

[1895] Input: User's face image.

[1896] Output: Face image file (e.g., face_image.jpg).

[1897] The user activates their smartphone camera and points it at their face to take a picture. The captured photo is saved as a file on the smartphone. This photo must be of high quality as it will be used in a later authentication process.

[1898] Step 2:

[1899] The user obtains fingerprint data using a fingerprint scanner.

[1900] Input: User's fingerprint.

[1901] Output: Fingerprint data.

[1902] The user places their finger on a fingerprint scanner connected to their smartphone. The scanner reads the fingerprint data and stores it digitally. This data is also used in subsequent authentication processes.

[1903] Step 3:

[1904] The device sends facial image and fingerprint data to the server.

[1905] Input: facial photo file, fingerprint data.

[1906] Output: Data transfer to server complete.

[1907] The device (smartphone) uses an internet connection to send the acquired facial image and fingerprint data to the server. This data is transmitted securely using encrypted communication.

[1908] Step 4:

[1909] The server generates a digital twin and links it with facial image and fingerprint data.

[1910] Input: facial photo file, fingerprint data.

[1911] Output: Digital twin.

[1912] The server generates a digital twin corresponding to the user based on the received facial photograph and fingerprint data. This digital twin is associated with biometric information for identity verification.

[1913] Step 5:

[1914] The server stores event ticket information and entry / exit management information linked to a digital twin.

[1915] Input: Event ticket information or entry / exit management information.

[1916] Output: Information linked to the digital twin.

[1917] The server links pre-registered event ticket information and access control information to the generated digital twin. This allows the digital twin to hold access permission information for specific events or facilities.

[1918] Step 6:

[1919] When users enter or leave an office or facility, the installed terminals acquire facial images and fingerprint data.

[1920] Input: User's facial photo, fingerprint data.

[1921] Output: Acquired biometric information.

[1922] The user stands in front of the access control device, and a camera and fingerprint scanner are used to capture a facial image and fingerprint. This data is collected in real time.

[1923] Step 7:

[1924] The device sends the acquired data to the server and requests identity verification.

[1925] Input: Acquired facial photograph and fingerprint data.

[1926] Output: Sending a request for identity verification.

[1927] The device sends the collected facial image and fingerprint data to the server and requests that it be matched with the information in the digital twin.

[1928] Step 8:

[1929] The server compares the acquired biometric information with the digital twin information.

[1930] Input: Acquired facial photograph, fingerprint data, and biometric information from the digital twin.

[1931] Output: Identity verification result (success or failure).

[1932] The server compares the received biometric information with a digital twin. If they match, the identity verification is considered successful.

[1933] Step 9:

[1934] The server sends the identity verification result to the device.

[1935] Input: Identity verification result.

[1936] Output: Notification of results to the terminal.

[1937] The server returns the verification result to the terminal, notifying it whether entry or exit is permitted or denied.

[1938] Step 10:

[1939] The device will allow or deny entry / exit based on the identity verification results.

[1940] Input: Identity verification result.

[1941] Output: Display of permission or denial of entry / exit.

[1942] The terminal displays the results received from the server, and if identity verification is successful, it allows entry or exit. If there is a mismatch, it displays a warning and denies entry or exit.

[1943] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1944] Modes for carrying out the invention

[1945] This invention relates to a system that generates a digital twin corresponding to a purchaser and links event ticket information to that digital twin. Furthermore, by combining it with an emotion engine, the system can recognize the user's emotions and provide services accordingly.

[1946] System Configuration

[1947] This system primarily consists of a server, user terminals, an emotion engine, and event venue entry terminals. The server is responsible for generating and managing digital twins, linking event ticket information, and storing and verifying identity information. User terminals are devices used to input personal and identity information when purchasing tickets and to communicate with the server. The emotion engine analyzes the user's facial image and voice data to recognize their emotional state. Event venue entry terminals are devices used to verify the user's identity upon entry.

[1948] Ticket purchase processing

[1949] When a user wishes to purchase an event ticket, the device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into the device. The collected personal information is sent from the device to a server, which generates a digital twin based on the received information. The generated digital twin is linked to the information of the purchased event ticket.

[1950] Acquisition and storage of emotional information

[1951] The user's facial image and voice data are analyzed by an emotion engine to recognize the user's emotional state. The acquired emotional information is sent to a server, associated with a digital twin, and stored in a database. This information is also used as a reference when the user uses the system again.

[1952] Entry process

[1953] On the day of the event, a user arrives at the event venue and requests entry. A terminal (authentication device) installed at the entrance requests facial recognition or fingerprint authentication from the user. When the user faces the authentication device or performs a fingerprint scan, the acquired facial image or fingerprint data is sent from the terminal to a server. The server compares the received biometric information with the digital twin information, and if they match, entry is permitted.

[1954] Emotion-based responses

[1955] Based on facial images and voice data acquired upon entry, the emotion engine re-analyzes the user's emotional state. For example, if the user is nervous, the system can relay this information to the guidance staff and prompt them to take appropriate action.

[1956] Specific example

[1957] As concrete examples, we will explain the process of purchasing event tickets, the entry process on the day of the event, and the management of emotional information.

[1958] Example of purchasing tickets

[1959] When a user purchases an event ticket using their smartphone, the device (smartphone app) prompts the user to take a photo of their face and perform a fingerprint scan. The device sends this information to a server, which generates a digital twin and stores the linked ticket information. Finally, the server sends a purchase completion notification to the user's email address. During the purchase, the user's facial image and voice data are analyzed by an emotion engine, and emotion information is added to the digital twin.

[1960] Example of entry

[1961] On the day of the event, when a user arrives at the venue entrance, an authentication terminal performs facial recognition or fingerprint authentication. The acquired facial image or fingerprint data is sent to a server and compared with a digital twin. If the server successfully matches the data, entry permission is issued, and the terminal displays a message granting entry to the user. Simultaneously, an emotion engine analyzes the user's emotional state and can provide necessary instructions to the event staff.

[1962] In this way, this system prevents ticket resale, ensures that the purchaser is indeed able to participate in the event, and provides services tailored to the user's emotional state.

[1963] The following describes the processing flow.

[1964] Ticket purchase processing

[1965] Step 1:

[1966] A user wishes to purchase event tickets and opens the purchase screen on their device.

[1967] Step 2:

[1968] The device displays an interface prompting the user to enter their name, email address, facial photo, and fingerprint data.

[1969] Step 3:

[1970] The user provides personal information (name, email address), takes a photo of their face, and performs a fingerprint scan.

[1971] Step 4:

[1972] The device sends collected personal information, facial images, and fingerprint data to the server. This data is encrypted using the SSL / TLS protocol.

[1973] Step 5:

[1974] The server analyzes the personal information it receives and generates a digital twin of the user.

[1975] Step 6:

[1976] The server links event ticket information (event name, date and time, seat information, etc.) to the generated digital twin and stores it in a database.

[1977] Step 7:

[1978] The server sends a ticket purchase completion notification to the user's email address.

[1979] Acquisition and storage of emotional information

[1980] Step 1:

[1981] The device is operated to display an interface for the user to provide a facial photograph and voice data.

[1982] Step 2:

[1983] The device collects the user's facial image and voice data, and sends that data to the emotion engine.

[1984] Step 3:

[1985] The emotion engine analyzes facial images and voice data to determine the user's emotional state.

[1986] Step 4:

[1987] The emotion engine sends the recognized emotion information to the server.

[1988] Step 5:

[1989] The server associates the received emotional information with a digital twin and stores it in a database.

[1990] Entry process

[1991] Step 1:

[1992] The user arrives at the event venue and heads towards the authentication terminal located at the entrance.

[1993] Step 2:

[1994] The device displays a message requesting the user to authenticate using facial recognition or fingerprint recognition.

[1995] Step 3:

[1996] The user faces the authentication terminal or performs a fingerprint scan.

[1997] Step 4:

[1998] The device sends facial images and fingerprint data it has acquired to the server. This data is encrypted using the SSL / TLS protocol.

[1999] Step 5:

[2000] The server compares the received biometric information with the information in the digital twin.

[2001] Step 6:

[2002] The server checks the matching results and issues an entry permit if there is a match. If the matching is unsuccessful, it decides to deny entry.

[2003] Step 7:

[2004] The server sends the verification result to the terminal. This includes a permission message if successful, and a rejection message if unsuccessful.

[2005] Step 8:

[2006] The device displays the verification result to the user. If an permission message is displayed, the user is allowed to enter. If a denial message is displayed, the user is prompted to try authenticating again or to go through another procedure.

[2007] Emotion-based responses

[2008] Step 1:

[2009] The device then collects the user's facial image and voice data again and sends them to the emotion engine.

[2010] Step 2:

[2011] The emotion engine analyzes the collected data to determine the user's emotional state.

[2012] Step 3:

[2013] The emotion engine sends the analysis results to the server.

[2014] Step 4:

[2015] Based on the emotional information it receives, the server sends necessary instructions to the event guidance staff.

[2016] Step 5:

[2017] The guidance staff will respond according to the user's emotional state.

[2018] This will prevent ticket resale, ensure that legitimate purchasers can attend the event, and create a system that provides services tailored to the user's emotional state.

[2019] (Example 2)

[2020] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2021] In modern event ticketing systems, resale and fraudulent use are major problems. Furthermore, services that take into account the emotional state of users when attending events are not adequately provided. In particular, utilizing user emotional information is expected to enable more personalized services, but this is difficult to achieve with current systems.

[2022] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[2023] In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring biometric information as identity verification information, means for storing the acquired biometric information in association with the digital twin, means for recognizing the emotional state via a user terminal, means for storing the recognized emotional information in association with the digital twin, input means for acquiring the purchaser's biometric information upon entry to the event venue, means for comparing the acquired biometric information with the identity verification information of the digital twin, means for granting entry if identity verification is successful, and means for changing the service based on the emotional state acquired upon entry. This prevents fraudulent use of tickets and enables the provision of personalized services according to the user's emotional state.

[2024] - "Means for generating a digital twin corresponding to a purchaser" refers to a means of creating a virtual entity (digital twin) corresponding to a user in digital space based on the user's personal information and biometric information.

[2025] "Method for linking event ticket information to a generated digital twin" refers to a method for associating and storing information about event tickets purchased by a user with the generated digital twin.

[2026] "Means for acquiring biometric information" refers to means for acquiring biometric characteristics such as a user's facial photograph or fingerprint data.

[2027] "Means for associating and storing acquired biometric information with a digital twin" refers to means for associating acquired user biometric information with a generated digital twin and storing it in a database or other storage device.

[2028] "Means for recognizing emotional states via a user terminal" refers to methods for analyzing a user's facial image and voice data to recognize their emotional state. This includes an emotion recognition engine.

[2029] "Means for associating and storing recognized emotional information with a digital twin" refers to means for associating analyzed user emotional information with a digital twin and storing it in a database or other storage device.

[2030] "Means for comparing acquired biometric information with digital twin identity verification information" refers to methods for comparing and verifying biometric information acquired at event venues, etc., with digital twin identity verification information that has been stored in advance.

[2031] "Means of granting entry upon successful identity verification" refers to the means of granting entry to a user upon successful verification. Specifically, this includes actions such as issuing an electronic permit or opening the venue gates.

[2032] "Means of modifying services based on emotional state acquired upon entry" refers to means of personalizing the services provided based on the emotional state of the user recognized upon entry. For example, this could include sending notifications to staff to help alleviate the user's anxiety.

[2033] This invention is a system that generates a digital twin corresponding to a purchaser and links event ticket information to that digital twin. In addition, by combining it with an emotion engine, it can recognize the user's emotional state and provide services accordingly. The main components of this system include a server, a user terminal, an emotion engine, and an entry terminal at the event venue.

[2034] Components and their operation

[2035] The server will generate and manage digital twins, link event ticket information, and store and verify identity information. The hardware used will be cloud services (e.g., Amazon Web Services EC2 instances). The software used will include a database management system (e.g., AWS RDS), an email sending service (e.g., AWS SES), an emotion recognition API (e.g., Microsoft Azure's Emotion API), and a biometric authentication API (e.g., Google Cloud Platform's Biometric API).

[2036] The user's device is used to input personal information and biometric data (facial image and fingerprint data) required when purchasing tickets, and to communicate with the server. This includes smartphones, tablets, and personal computers.

[2037] The emotion engine analyzes the user's facial image and voice data to recognize their emotional state. The analysis results are sent to a server and stored in association with a digital twin.

[2038] Event venue entry terminals are devices used to verify the user's identity upon entry. These include facial recognition cameras and fingerprint scanners.

[2039] Ticket purchase processing

[2040] When a user wishes to purchase an event ticket, their device displays a ticket purchase screen. The user enters personal information such as their name, email address, facial photograph, and fingerprint data into their device. The collected personal information is sent from the device to a server, which generates a digital twin based on the received information. The generated digital twin is linked to the information of the purchased event ticket. An emotion engine analyzes the facial photograph and voice data to recognize the user's emotional state and stores that information in association with the digital twin.

[2041] As a concrete example, when a user purchases concert tickets using their smartphone, the smartphone (device) instructs the user to take a photo of their face and perform a fingerprint scan. This information is sent from the device to a server, which generates a digital twin and stores the purchased ticket information in association with emotional information.

[2042] Entry process

[2043] On the day of the event, a user arrives at the venue and requests entry. An entry terminal (authentication device) installed at the entrance requests facial recognition or fingerprint authentication. When the user faces the authentication device or performs a fingerprint scan, the acquired biometric information is transmitted from the terminal to a server. The server compares the received information with the digital twin data and grants entry if they match. Simultaneously, an emotion engine analyzes the user's emotional state and notifies the event staff of the necessary information.

[2044] As a concrete example, when a user arrives at an event venue and undergoes facial recognition, the authentication terminal takes a picture of the user's face and sends it to the server. The server compares the received facial picture with a digital twin, and if they match, it displays an entry permission message on the terminal.

[2045] Examples of prompts for generative AI models

[2046] The following are examples of prompts for the generative AI model when implementing this system:

[2047] "Please explain the process for analyzing user emotional data at the time of concert ticket purchase and linking that emotional data to the corresponding digital twin."

[2048] "Please describe the system design for verifying user identity using biometric authentication information during the entry process on the day of the event, analyzing their emotional state, and notifying staff accordingly."

[2049] As described above, this system prevents the fraudulent use of tickets, ensures that the purchaser themselves can participate in the event, and provides personalized services tailored to the user's emotional state.

[2050] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2051] The processing flow of this system's program

[2052] Ticket purchase processing steps

[2053] Step 1:

[2054] The user accesses the event ticket purchase site using a device such as a smartphone.

[2055] Input: User actions (launching a web browser app or dedicated mobile app)

[2056] Output: Display of the purchase screen

[2057] Specific actions: The user enters the URL of the purchase site and accesses it, or launches the mobile app.

[2058] Step 2:

[2059] The device displays a ticket purchase screen and prompts the user to enter the necessary personal information (name, email address, photo, fingerprint data).

[2060] Input: System-side purchase screen display command

[2061] Output: Present an input form to the user.

[2062] Specific operation: The system generates HTML and application screens and displays forms for personal information input.

[2063] Step 3:

[2064] The user enters the required information into the provided form, takes a photo of their face, and performs a fingerprint scan.

[2065] Input: User's personal information (name, email address), facial photograph, fingerprints

[2066] Output: The input data is saved to the terminal and temporarily prepared.

[2067] Specific operation: The user takes a photo of their face using the camera and places their finger on the fingerprint sensor to obtain fingerprint data.

[2068] Step 4:

[2069] The device sends the entered personal information and biometric data to the server.

[2070] Input: User's personal information, facial photograph, fingerprint data

[2071] Output: Encrypted transmitted data

[2072] Specific operation: The device encrypts and sends data to the server using the HTTPS protocol.

[2073] Step 5:

[2074] The server generates a digital twin corresponding to the user based on the received data.

[2075] Input: Encrypted personal information and biometric information

[2076] Output: Generated data for the digital twin

[2077] Specific operation: The server executes the digital twin generation algorithm and creates a new entry in the database.

[2078] Step 6:

[2079] The server links and stores the purchased event ticket information to the generated digital twin.

[2080] Input: Digital twin data and purchased ticket information

[2081] Output: Database entries where ticket information is stored in the digital twin.

[2082] Specific operation: The server accesses the database and adds and saves ticket information to the digital twin entry.

[2083] Step 7:

[2084] The server sends a purchase completion notification to the user's email address.

[2085] Input: User's email address and purchase completion information

[2086] Output: Purchase completion notification email

[2087] Specific operation: The server generates and sends a purchase completion notification email using the email sending API.

[2088] Step 8:

[2089] The emotion engine analyzes received facial photos and voice data to recognize the user's emotional state.

[2090] Input: User's facial photo and voice data

[2091] Output: User's emotional state

[2092] Specific operation: An emotion recognition algorithm is executed, and as a result, an emotional state (e.g., joy, surprise, etc.) is generated.

[2093] Step 9:

[2094] The server stores the analyzed emotional information, associating it with a digital twin.

[2095] Input: Analyzed emotional information and digital twin

[2096] Output: Digital twin data with associated emotional information

[2097] Specific operation: The server accesses the database and adds and saves emotional information to the digital twin entry.

[2098] Entry processing steps

[2099] Step 1:

[2100] The user arrives at the venue entrance on the day of the event.

[2101] Input: User's actions (arrived at the venue entrance to participate in the event)

[2102] Output: Start of admission procedure

[2103] Specific action: The user travels to the event venue and arrives at the designated entrance.

[2104] Step 2:

[2105] The entry terminal prompts the user to perform facial recognition or fingerprint authentication.

[2106] Input: System-side authentication start command

[2107] Output: Display of authentication screen

[2108] Specific action: The device's display lights up and prepares for facial recognition or fingerprint recognition.

[2109] Step 3:

[2110] The user either takes a photo of their face or performs a fingerprint scan.

[2111] Input: User's facial photo or fingerprint

[2112] Output: Acquired biometric information

[2113] Specific actions: The user faces the camera or places their finger on the fingerprint sensor.

[2114] Step 4:

[2115] The device sends the acquired biometric information to the server.

[2116] Input: Biometric information (facial photo, fingerprint data)

[2117] Output: Encrypted transmitted data

[2118] Specific operation: The device encrypts and sends data to the server using the HTTPS protocol.

[2119] Step 5:

[2120] The server compares the received biometric information with the digital twin's identity verification information.

[2121] Input: Received biometric information, digital twin identity verification information

[2122] Output: Matching result (success / failure)

[2123] Specific operation: The server executes a biometric matching algorithm to check if there is a match.

[2124] Step 6:

[2125] If the server finds that the verification is successful, it will send an access permission message to the terminal.

[2126] Input: Data for successful matching

[2127] Output: Entry permission message

[2128] Specific operation: Based on the authentication result, the server generates an entry permission message as the next step and sends it to the terminal.

[2129] Step 7:

[2130] The device displays a message granting the user permission to enter.

[2131] Input: Entry permission message

[2132] Output: Screen display of entry permission

[2133] Specific actions: The terminal displays an entry permission message on its screen and performs actions such as opening the gate.

[2134] Step 8:

[2135] The emotion engine re-analyzes the user's latest facial photos and voice data and notifies event staff as needed.

[2136] Input: Latest facial photo and voice data

[2137] Output: Analysis results and notifications

[2138] Specific operation: The emotion engine re-analyzes the emotional state and notifies the event staff of the results via the server.

[2139] Step 9:

[2140] Event staff will receive notifications and take appropriate action for users.

[2141] Input: Notification from emotion engine

[2142] Output: Appropriate response (e.g., guidance and assistance to the user)

[2143] Specific actions: Event staff will check notifications from the emotion engine and respond to users.

[2144] By following these steps, the system can prevent the misuse of tickets and provide personalized services tailored to the user's emotional state.

[2145] (Application Example 2)

[2146] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2147] Conventional admission management systems focused on preventing ticket resale and verifying identity, but they did not address individual customer needs or provide services based on emotional states. Furthermore, they did not utilize customer purchase history in physical stores to provide services. This invention solves these problems, enabling the provision of services tailored to individual customer needs and considering emotional states at event venues and physical stores.

[2148] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for generating a digital twin corresponding to a purchaser, means for linking event ticket information to the generated digital twin, means for acquiring facial photographs and fingerprint data as identity verification information, means for associating and storing the acquired facial photographs and fingerprint data with the digital twin, input means for acquiring the purchaser's facial photograph or fingerprint data upon entry to the event venue, means for comparing the acquired facial photograph or fingerprint data with the identity verification information of the digital twin, means for granting entry if identity verification is successful, means for acquiring information and emotional state from the digital twin and displaying visual information when a customer enters a physical store, means for performing emotion analysis based on facial photographs and displaying the results, and means for acquiring the customer's past purchase history and preferences and suggesting recommended products. This makes it possible to provide services tailored to the individual needs of customers and services based on their emotional state at event venues and physical stores.

[2149] A "digital twin" is a virtual entity that stores and manages the personal information, purchase history, and emotional state of a real person in digital format.

[2150] An "emotion engine" is a software or hardware system that analyzes input facial images and voice data to recognize and evaluate a person's emotional state.

[2151] "Identity verification upon entry" refers to the process of verifying an individual's identity by comparing their actual face and fingerprints with pre-registered facial photographs and fingerprint data when entering a facility such as an event venue.

[2152] A "physical store" is a physical shop where goods are sold or services are provided in person.

[2153] "Purchase history" refers to a record of information about products and services that a customer has purchased in the past.

[2154] "Emotional state" refers to an individual's psychological and emotional condition, including specific emotional states such as joy, confusion, and tension.

[2155] A "recommended product" is a product that is selected and suggested based on the customer's past purchase history and preferences.

[2156] "Facial recognition" is a technology that analyzes captured facial images to identify who the person is.

[2157] "Smart glasses" are glasses-type wearable devices that incorporate a display, camera, and various sensors, enabling them to display, acquire, and process information.

[2158] "Visual information" refers to digital information in a form that can be visually confirmed, such as a customer's facial photograph, purchase history, and emotional state.

[2159] This invention is a system that generates a digital twin corresponding to a purchaser, links it with event ticket information, and acquires the customer's emotional state using an emotion engine, thereby enhancing service provision at physical stores and event venues. This system acquires and processes various data using applications installed on smart glasses, smartphones, robots, etc.

[2160] Hardware and software to use

[2161] server:

[2162] AWS Aurora: Used for database management.

[2163] TensorFlow: A machine learning library used to implement sentiment analysis engines.

[2164] nVIDIA Jetson TX2: Hardware used for facial recognition and emotion analysis processing.

[2165] Terminal:

[2166] Smart glasses: Acquire customer facial photos and emotional states and display visual information.

[2167] Smartphone: Used for purchasing tickets and entering and submitting identity verification information.

[2168] Event venue authentication device: Used to obtain facial images and fingerprint data upon entry.

[2169] Specific processing of the system

[2170] In trading company operations, the system operates as follows:

[2171] Digital Twin Generation

[2172] 1. The server receives the user's facial photograph and fingerprint data and generates a digital twin of the person based on this data. The digital twin is linked to event ticket information and stored in a database along with personal information.

[2173] Providing services at physical stores

[2174] 1. When a user enters a physical store, smart glasses or a robot's camera captures a photo of their face and sends it to a server.

[2175] 2. The server analyzes the acquired facial photograph and retrieves the corresponding digital twin information and past purchase history.

[2176] 3. The device (smart glasses or robot) displays the acquired digital twin information and emotional state, and takes appropriate action based on the current emotional state.

[2177] Emotional analysis and product recommendations

[2178] 1. The emotion engine analyzes the user's facial expression data to determine their current emotional state.

[2179] 2. Based on the assessment results, the server references the customer's preferences and past purchase history to generate a list of recommended products.

[2180] 3. The terminal displays recommended products, and staff members make product suggestions based on that information.

[2181] Specific example

[2182] For example, when a customer enters an event venue, the process works as follows:

[2183] 1. The user arrives at the entrance of the event venue and faces the authentication device.

[2184] 2. The authentication device acquires a facial image and sends it to the server.

[2185] 3. The server verifies the digital twin and grants access if the identity verification is successful.

[2186] 4. Simultaneously, the emotion engine analyzes the user's facial image and notifies the staff of their emotional state.

[2187] Example of a prompt

[2188] Example prompt: "Please provide specific scenarios for analyzing a customer's emotional state and responding accordingly. For example, please describe in detail how to handle a situation where the customer is confused."

[2189] Thus, by utilizing digital twins and an emotion engine, the present invention enables the provision of more personalized and advanced services at event venues and physical stores.

[2190] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2191] Step 1:

[2192] The user launches a ticket purchase application on their smartphone and enters personal information such as their name, email address, facial photo, and fingerprint data.

[2193] Input: Name, email address, photo, fingerprint data

[2194] Data processing and calculation: Converting personal information into an appropriate format for storage in a database.

[2195] Output: Personal information in a saved format

[2196] Step 2:

[2197] The device sends the personal information it has acquired to the server.

[2198] Input: Personal information entered on a smartphone

[2199] Data processing and calculation: Encryption of personal information, network transmission

[2200] Output: Encrypted personal information

[2201] Step 3:

[2202] The server generates a digital twin based on the personal information it receives and links it to the event ticket information.

[2203] Input: Encrypted personal information

[2204] Data processing and computation: Decryption of information, generation of digital twins, linking of ticket information.

[2205] Output: Ticket information linked to the generated digital twin

[2206] Step 4:

[2207] The server sends a purchase completion notification to the user's email address.

[2208] Input: Completed purchase information

[2209] Data processing and calculation: Email notification generation

[2210] Output: Purchase completion notification email

[2211] Step 5:

[2212] On the day of the event, users will face an authentication device at the venue entrance to have their face photographed.

[2213] Input: Current photo

[2214] Data Processing and Calculation: Face Detection and Feature Point Extraction using Face Recognition Algorithms

[2215] Output: Feature point data from facial photographs

[2216] Step 6:

[2217] The device sends the acquired facial image to a server for comparison with a digital twin.

[2218] Input: Feature point data from a facial photograph

[2219] Data processing and calculation: Data encryption, identity verification using matching algorithms.

[2220] Output: Identity Verification Result

[2221] Step 7:

[2222] If the server successfully verifies the user's identity, it will send an access permission message to the device.

[2223] Input: Identity verification result

[2224] Data processing and calculation: Processing of authentication results, message generation.

[2225] Output: Entry permission message

[2226] Step 8:

[2227] When a user enters a physical store, the smart glasses capture a photo of their face and send it to a server.

[2228] Input: Current photo

[2229] Data processing and calculation: Face detection using face recognition algorithms, generation of visual information

[2230] Output: Visual information

[2231] Step 9:

[2232] The server analyzes the acquired facial photographs to obtain corresponding digital twin information and purchase history.

[2233] Input: Facial photograph, digital twin database

[2234] Data processing and calculation: Feature point analysis of facial images, purchase history extraction.

[2235] Output: Digital twin information, purchase history

[2236] Step 10:

[2237] The emotion engine analyzes the user's facial expression data to determine their current emotional state.

[2238] Input: Facial expression data

[2239] Data processing and calculation: Emotion determination using facial expression analysis algorithms

[2240] Output: Emotional state data

[2241] Step 11:

[2242] The server generates a list of recommended products based on the emotion assessment results and displays them on the smart glasses.

[2243] Input: Emotional state data, purchase history

[2244] Data processing and calculation: Execution of recommended product algorithms, list generation.

[2245] Output: Recommended product list

[2246] Step 12:

[2247] The terminal allows staff to suggest products to customers based on recommended items.

[2248] Input: Recommended product list

[2249] Data processing and calculation: Generation and display of proposed information

[2250] Output: Product suggestion message

[2251] This process allows customers to enjoy personalized service at physical stores and event venues.

[2252] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2253] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2254] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[2255] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2256] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[2257] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[2258] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[2259] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[2260] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance"...

Claims

1. A means of generating a digital twin corresponding to the purchaser, A method for linking event ticket information to the generated digital twin, Methods for obtaining facial photographs and fingerprint data as identity verification information, A method for associating and storing acquired facial photographs and fingerprint data in a digital twin, An input method to obtain the purchaser's facial photograph or fingerprint data upon entry to the event venue, A means of matching acquired facial photograph or fingerprint data with the identity verification information of the digital twin, A system that includes means to grant entry only after successful identity verification.

2. The system according to claim 1, configured such that a digital twin is stored in a database containing event ticket information.

3. The system according to claim 1, configured such that the acquisition of facial photographs and fingerprint data and their matching with a digital twin are performed by encrypted communication.

Images