system

Abstract

We provide the system. [Solution] A means for capturing facial data, A means of sending facial data to a server, A means of matching against a facial recognition database stored on a server, Means of obtaining personal information, A means of transmitting acquired person information to a terminal, A means of displaying or outputting audio on the terminal, A system that includes means for performing the above operations only within a specific facility.

Description

Technical Field

[0004] , , , ,

[0005] , , , , ,

[0003] , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a modern business environment, it is required to quickly and accurately grasp information about the other party. However, in many business scenarios, people often cannot recall the name or position of the other party. This problem is further exacerbated by the time-consuming management and search of business cards. In addition, the risk of information leakage is also high, and it is necessary to display information in a secure environment. Therefore, a system that can easily obtain information about the other party and quickly display it is required.

Means for Solving the Problems

[0005] The present invention provides a system that includes means for capturing facial data, means for transmitting facial data to a server, means for comparing it with a facial recognition database stored on the server, means for acquiring person information, means for transmitting the acquired person information to a terminal, means for displaying or outputting audio on the terminal, and means for performing the above operations only within a specific facility. This makes it possible for a user to recognize the face of another person within a designated facility and quickly display or output information about that person acquired from a business card database or the like, thereby facilitating information management and communication in business settings. Furthermore, security is enhanced by limiting the use of the system to a specific facility in order to prevent information leakage.

[0006] "Facial data" refers to data that digitally records the characteristics of a person's face.

[0007] A "server" is a computer system that stores, manages, and processes data over a network.

[0008] A "facial recognition database" is a database that stores facial data and uses it for matching purposes.

[0009] "Personal information" refers to data obtained from sources such as business card databases, which includes information related to a specific person, such as their name and affiliation.

[0010] A "terminal" is a device that has functions such as capturing, transmitting, displaying, or outputting facial data.

[0011] "Specific facilities" refers to specific locations where the use of the system is restricted, such as school grounds or offices.

[0012] "Capture" refers to the process of acquiring facial data in digital format using a device such as a camera.

[0013] "Transmitting" refers to the operation of transmitting data from one device to another.

[0014] "Matching" refers to the operation of comparing the captured face data with an existing face authentication database.

[0015] "Display" refers to the operation of visually presenting the acquired person information on the screen of the terminal.

[0016] "Voice output" refers to the operation of presenting the acquired person information by voice through the speaker of the terminal.

[0017] "Business card database" is a database in which information such as a person's name and affiliation is stored.

Brief Description of the Drawings

[0018] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] This invention is a system that links facial recognition data and business card data to display or output information about a person using smart glasses within a specific facility.

[0040] Facial data capture and registration

[0041] The user stands in front of a sensor-equipped terminal at the facility entrance, and their face is captured by the camera. The terminal sends the captured facial data to a server. The server stores this facial data in a facial recognition database. The user also uploads business card data and affiliation lists to the terminal and sends them to the server. The server stores this information in a business card database.

[0042] Authentication within the facility

[0043] When a user enters the facility, the terminal works in conjunction with the facility's access control system to perform facial recognition. Once authentication is complete, the terminal authorizes the user to use smart glasses and performs the necessary settings.

[0044] Facial recognition and data linkage

[0045] When a user wears smart glasses and moves around the facility, the smart glasses (device) capture the faces of people around them using its camera. The captured face data is sent to a server in real time. The server compares the data with a facial recognition database and retrieves matching person information from a business card database.

[0046] Displaying information

[0047] The server sends the acquired person information to the device. The device (smart glasses) displays this person information on the lens or notifies the user via voice through the speaker.

[0048] Specific example

[0049] 1. User registration and facial data acquisition:

[0050] The user stands still at the entrance of the facility, facing the camera.

[0051] The device will notify you that "face data has been captured" and send the face data to the server.

[0052] The server displays a confirmation message stating, "Face data saving complete."

[0053] 2. Authentication within the facility:

[0054] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[0055] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[0056] 3. Facial recognition and data linkage:

[0057] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[0058] The captured facial data is sent to the server.

[0059] The server displays "Matching against facial recognition database" and retrieves matching person information.

[0060] 4. Display of information:

[0061] The server sends the retrieved person information (for example, "Taro Tanaka - Sales Department") to the terminal.

[0062] Smart glasses display this information on the lenses. For example, "Taro Tanaka - Sales Department" is visually shown. It can also provide voice notifications such as "Taro Tanaka, Sales Department."

[0063] This system allows users to quickly obtain and verify information about others within a specific facility. Furthermore, by limiting the system's use to specific facilities, the risk of information leakage can be minimized.

[0064] The following describes the processing flow.

[0065] Step 1:

[0066] The user stands in front of a sensor-equipped terminal at the facility entrance, facing the camera directly, and remains still for approximately 5 seconds.

[0067] Step 2:

[0068] The device captures the user's face. The captured facial data is saved to the device in real time.

[0069] Step 3:

[0070] The device sends the captured facial data to the server. Once the transmission is complete, the device notifies the user that "facial data has been sent to the server."

[0071] Step 4:

[0072] The server saves the received facial data to the facial recognition database. Once saving is complete, the server sends a message stating, "Facial data saving complete."

[0073] Step 5:

[0074] Users upload business card data and affiliation lists to their devices. This is done through a specified input format.

[0075] Step 6:

[0076] The device sends the uploaded business card data to the server. Once the transmission is complete, a notification will appear stating, "Business card data has been sent to the server."

[0077] Step 7:

[0078] The server saves the submitted business card data to the business card database. Once saving is complete, the server notifies the user that "Business card data has been saved."

[0079] Step 8:

[0080] The user stands in front of a terminal for facial recognition at the facility entrance. The terminal is linked to the facility's access control system and authenticates the user's face.

[0081] Step 9:

[0082] The device displays a notification saying, "Authentication successful. Please enter," once facial recognition is complete.

[0083] Step 10:

[0084] The user puts on smart glasses and begins moving around the facility. The smart glasses (device) continuously capture the faces of people in the surrounding area using its camera.

[0085] Step 11:

[0086] The device sends the captured facial data to the server in real time. The status "Sending facial data" is displayed.

[0087] Step 12:

[0088] The server compares the received face data with the face recognition database and finds a matching face. It then outputs the message "Matching..." as the processing status.

[0089] Step 13:

[0090] The server retrieves person information linked to matching facial data from the business card database. After retrieval, a notification is sent stating, "Person information has been retrieved."

[0091] Step 14:

[0092] The server sends the acquired person information to the terminal. The status "Sending person information" is displayed.

[0093] Step 15:

[0094] The device (smart glasses) displays the transmitted person information on the lens. It also provides voice notifications via the speaker, such as "This is person AA," if necessary.

[0095] (Example 1)

[0096] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] Conventional facial recognition systems require manual recognition and data acquisition, making it difficult to quickly and efficiently obtain information about individuals within a specific facility. Furthermore, the lack of restrictions on system usage increases the risk of data leaks. Additionally, insufficient real-time information sharing and the incomplete integration of current facial recognition data with business card data make the system impractical.

[0098] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0099] In this invention, the server includes means for capturing facial data, means for transmitting facial data to the server, means for comparing it with a facial recognition database stored on the server, means for obtaining person information from a linked database, means for transmitting the obtained person information to a terminal, means for displaying or outputting audio on the terminal, and means for performing the above operations using smart glasses within a specific facility. This makes it possible to quickly and efficiently obtain and verify the information of a person within a specific facility, minimizing the risk of information leakage. Furthermore, real-time information sharing is realized, the linkage between facial recognition data and business card data is enhanced, and practicality is improved.

[0100] "Facial data" refers to digital data that includes images of an individual's face and its characteristics, captured by a camera.

[0101] A "server" is a computer system that stores, manages, and processes data on a network.

[0102] A "facial recognition database" is a dedicated database used to store captured facial data and perform facial recognition based on that data.

[0103] A "terminal" refers to a hardware device operated by the user, and this includes smart glasses and computing devices.

[0104] A "business card database" is a database that stores business card information and related data, and uses this information to provide personal information.

[0105] "Smart glasses" are wearable devices worn on the head that are equipped with cameras and displays and have the function of displaying and capturing information.

[0106] "Within a specific facility" refers to designated areas or the interior of buildings where the use of the system is restricted.

[0107] This invention relates to a system that links facial recognition data with personal information and displays or outputs information about another person using smart glasses within a specific facility. This system allows users to quickly obtain and verify information about others within a designated facility. This minimizes the risk of information leakage and improves practicality.

[0108] Hardware to use

[0109] 1. Facial data capture device: A camera installed in smart glasses (e.g., Microsoft® HoloLens®, Google® Glass®).

[0110] 2. Terminal: Smart glasses or a computer device operated by the user.

[0111] 3. Server: A computer system for managing and processing databases (e.g., Linux® server, AWS® server).

[0112] Software to use

[0113] 1. Facial recognition algorithms: Facial recognition APIs such as OpenCV and Amazon Rekognition.

[0114] 2. Database management systems: such as MySQL (registered trademark) and PostgreSQL.

[0115] 3. Communication Protocols: Protocols for real-time communication such as HTTP, WebSocket, and MQTT.

[0116] Specific details of data processing and data calculations

[0117] 1. User registration and facial data acquisition:

[0118] The user stands still at the entrance of the facility, facing the camera.

[0119] The device captures facial data from the camera image and notifies the user that "facial data has been captured."

[0120] The device transmits facial data to the server in real time.

[0121] The server saves the facial data to a MySQL database and notifies the terminal that "Facial data saving is complete."

[0122] 2. Authentication within the facility:

[0123] As the user passes through the facility entrance, they turn their face towards the camera again as they walk.

[0124] The device activates the facial recognition system, sends facial data to the server, and performs authentication.

[0125] The server checks whether the face data matches the data stored in the database and displays "Authentication successful" on the screen.

[0126] Once authentication is successful, the device communicates with the smart glasses and automatically performs the necessary settings (for example, Wi-Fi connection and user profile synchronization).

[0127] 3. Facial recognition and data linkage:

[0128] As the user wears the smart glasses and moves around the facility, the smart glasses' camera captures the faces of people in the surrounding area.

[0129] The captured facial data is transmitted from the smart glasses to the server via a high-speed network.

[0130] The server performs real-time matching against the facial recognition database and retrieves matching person information from the business card database.

[0131] During the matching process, the server displays the current processing progress on the smart glasses.

[0132] 4. Display of information:

[0133] The server sends the acquired person information to the smart glasses using the WebSocket protocol.

[0134] Smart glasses visually display information such as "Taro Tanaka - Sales Department" on the lenses.

[0135] The smart glasses' speaker notifies the user via voice, "Taro Tanaka, Sales Department."

[0136] Users receive visual and audible notifications, allowing them to instantly understand information about people around them.

[0137] Specific example

[0138] 1. User registration and facial data acquisition:

[0139] The user stands still at the entrance of the facility, facing the camera.

[0140] The device will notify you that "face data has been captured" and send the face data to the server.

[0141] The server displays a confirmation message stating, "Face data saving complete."

[0142] 2. Authentication within the facility:

[0143] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[0144] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[0145] 3. Facial recognition and data linkage:

[0146] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[0147] The captured facial data is sent to the server.

[0148] The server displays "Matching against facial recognition database" and retrieves matching person information.

[0149] 4. Display of information:

[0150] The server sends the retrieved person information (for example, "Taro Tanaka - Sales Department") to the terminal.

[0151] Smart glasses display this information on the lenses. For example, "Taro Tanaka - Sales Department" is visually shown.

[0152] It is also possible to notify users via voice message saying, "Taro Tanaka, Sales Department."

[0153] Example of a prompt

[0154] "Please describe a system that captures a user's face at the facility entrance, registers it in a database, and uses smart glasses to display or output information about other users within the facility in real time."

[0155] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0156] Step 1:

[0157] Capture and transmit facial data

[0158] The user stands still at the entrance of the facility, facing the camera. The device captures the face and sends the data to the server. Specifically, the device detects the face from the camera image, performs image processing, and generates face data. This face data is sent to the server in real time via the network. The server stores the received face data in a face recognition database.

[0159] Input: User's facial image (input from camera)

[0160] Data processing: Generating face data through image processing.

[0161] Output: Captured facial data (sent from the device to the server)

[0162] Specific actions:

[0163] The user stands in the designated location, faces the camera, and remains still for a few seconds.

[0164] The device's camera captures facial images, and internal processing generates facial data.

[0165] The device sends the generated facial data to the server.

[0166] The server stores the facial data in a database.

[0167] Step 2:

[0168] Authentication within the facility

[0169] When a user enters a facility, the terminal performs facial recognition. The terminal sends facial data to a server for authentication. The server compares the facial data with the data stored in the database to confirm a match. If authentication is successful, the terminal authorizes the use of smart glasses and performs the necessary settings.

[0170] Input: Entered facial data (captured from the device's camera)

[0171] Data processing: Matching using facial recognition algorithms

[0172] Output: Authentication result (success / failure)

[0173] Specific actions:

[0174] The user turns their face towards the camera as they pass through the facility entrance.

[0175] The device captures the facial data again and sends it to the server.

[0176] The server compares the face data stored in the database and returns the authentication result to the terminal.

[0177] If authentication is successful, the device will display permission to use the smart glasses on the screen and automatically configure the settings.

[0178] Step 3:

[0179] Facial recognition and data linkage

[0180] When a user wears smart glasses and moves around the facility, the smart glasses' camera captures the faces of people in the surrounding area. The captured facial data is sent to a server in real time. The server compares it with a facial recognition database and retrieves matching person information from a business card database.

[0181] Input: Images of people's faces in the surrounding area (input from the smart glasses' camera)

[0182] Data processing: Facial recognition and database matching

[0183] Output: Matching person information

[0184] Specific actions:

[0185] Users wear smart glasses and walk around the facility.

[0186] The smart glasses' camera continuously captures the faces of people in the surrounding area.

[0187] The captured facial data is transmitted from the smart glasses to the server via a high-speed network.

[0188] The server performs real-time matching against the facial recognition database and retrieves matching person information from the business card database.

[0189] Step 4:

[0190] Displaying information

[0191] The server transmits the acquired person information to the device (smart glasses). The device (smart glasses) either displays this information on the lenses or notifies the user via audio through the speaker.

[0192] Input: Matching results (person information)

[0193] Data processing: Information transmission

[0194] Output: Visual display or audio notification

[0195] Specific actions:

[0196] The server sends the acquired person information to the smart glasses using the WebSocket protocol.

[0197] Smart glasses visually display information such as "Taro Tanaka - Sales Department" on the lenses.

[0198] The smart glasses' speaker notifies the user via voice, "Taro Tanaka, Sales Department."

[0199] Users receive visual and audible notifications, allowing them to instantly understand information about people around them.

[0200] (Application Example 1)

[0201] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0202] In conventional systems, when acquiring personal information through facial recognition within a specific facility, the information obtained was limited, making it difficult to instantly retrieve and display detailed customer purchase history or special requests in commercial facilities. Furthermore, centralized information management was difficult, and data collection from various databases was complex. Additionally, delays in providing individual services could occur, posing a risk of lowering customer satisfaction.

[0203] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0204] In this invention, the server includes means for capturing facial data, means for transmitting the facial data to the server, means for matching and retrieving person information stored in a facial recognition database, and means for transmitting customer information, including purchase history and special requests, to a terminal. This enables store employees in commercial facilities to instantly obtain detailed information about visiting customers using smart devices and provide personalized services.

[0205] "Facial data" refers to digital image data that captures a person's face as a key feature.

[0206] A "server" is a computer system that sends, receives, and stores data over a network.

[0207] A "facial recognition database" is a database system that stores captured facial data for later comparison.

[0208] "Personal information" refers to information related to an individual obtained by matching it with a facial recognition database.

[0209] A "terminal" is a device that receives personal information and displays or outputs it to the user as audio.

[0210] "Within a specific commercial facility" refers to a specific physical area where commercial activities take place.

[0211] A "smart device" is an electronic device with advanced functions, such as smart glasses or a smartphone.

[0212] "Purchase history" refers to a record of products and services that a customer has purchased in the past.

[0213] A "special request" refers to a customer's specific request regarding a particular product or service.

[0214] A "purchase history database" is a database system used to store and manage a customer's purchase history.

[0215] This invention relates to a system that recognizes customer facial data within a specific commercial facility, allowing store staff to instantly retrieve and display associated purchase history and special requests. To implement this invention, a server, smart devices, a facial recognition database, a purchase history database, and the like are used.

[0216] System Configuration

[0217] The server stores facial data and matches it against a facial recognition database. It also incorporates a purchase history database that stores and manages purchase history and special requests. This database is used to understand and provide detailed customer information immediately.

[0218] A smart device (e.g., smart glasses) is equipped with a camera that captures the customer's face and a display that shows the information. The smart device has the function of capturing facial data and transmitting that data to a server in real time. It also displays or notifies the customer of the information transmitted from the server via sound.

[0219] Face data capture and transmission

[0220] The user (in this case, a store employee) wears a smart device while working in a commercial facility. When a customer enters the store, the camera on the smart device captures facial data. This facial data is transmitted to a server via the network.

[0221] Facial recognition and data matching

[0222] The server compares the transmitted facial data with a facial recognition database to retrieve matching person information. Furthermore, it extracts the customer's purchase history and special request information from a purchase history database. The retrieved information is immediately sent to the smart device.

[0223] Display and notification of customer information

[0224] The smart device displays personal information transmitted from the server. For example, it might show information such as "Hanako Yamada, Purchase History: Item A, Item B, Special Request: Express Delivery." This allows store staff to provide personalized service quickly. Voice notifications are also available depending on the settings.

[0225] Specific example

[0226] When creating an example prompt for a system in a meeting facility where a store employee wearing smart glasses takes a picture of a newly entering customer's face with a camera and sends the data to a server. The server then compares it with a facial recognition database and displays the person's information on a screen, the input should be: "Prompt: Write Python code to capture customer face data and display that information."

[0227] This invention makes it possible to improve customer service within specific commercial facilities and increase customer satisfaction. This allows for the provision of efficient and prompt service, contributing to the promotion of commercial activities.

[0228] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0229] Step 1:

[0230] A user wears a smart device and waits in a commercial facility. The smart device's camera continuously captures images of people in the surrounding area. The input is the smart device's camera feed, and the output is captured facial data. This facial data is temporarily stored on the smart device.

[0231] Step 2:

[0232] A smart device sends captured facial data to a server over the network. The input is the captured facial data, and the output is a message indicating that the facial data was successfully sent to the server. This data transmission is performed using an HTTP POST request.

[0233] Step 3:

[0234] The server compares the received facial data with a facial recognition database. The input is the newly received facial data, and the output is the matching result. A facial recognition algorithm is used in the matching process. The matching result is either matching person information (if any) or an error message if there is no match.

[0235] Step 4:

[0236] If the server finds a match in the facial recognition database, it retrieves the relevant purchase history and special request information from the purchase history database. The input is the ID of the matched person, and the output is the purchase history and request information of that person. This data is retrieved using SQL queries or similar methods.

[0237] Step 5:

[0238] The server sends customer information (purchase history and special requests) to the smart device. The input is the customer's personal information, and the output is a message indicating successful data transmission to the smart device. This transmission is performed using an HTTP GET request.

[0239] Step 6:

[0240] A smart device receives customer information transmitted from a server and displays it on its screen or provides an audio notification. The input is customer information received from the server, and the output is the information displayed on the screen or an audio notification. This process uses both a display function and a text-to-speech (TTS) function.

[0241] These steps enable the present invention to quickly and efficiently acquire customer information within a specific commercial facility and to provide personalized service by store staff.

[0242] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0243] This invention is a system that links facial recognition data and business card data, and uses smart glasses and an emotion engine within a specific facility to display or output information and emotional state of the person concerned.

[0244] Facial data capture and registration

[0245] The user stands in front of a sensor-equipped terminal at the facility entrance, and their face is captured by the camera. The terminal sends the captured facial data to a server. The server stores this facial data in a facial recognition database. The user also uploads business card data and affiliation lists to the terminal and sends them to the server. The server stores this information in a business card database.

[0246] Authentication within the facility

[0247] When a user enters the facility, the terminal works in conjunction with the facility's access control system to perform facial recognition. Once authentication is complete, the terminal authorizes the user to use smart glasses and performs the necessary settings.

[0248] Facial recognition and data linkage

[0249] As a user moves around the facility wearing smart glasses, the smart glasses (device) continuously capture the faces of people around them using its camera. The captured face data is transmitted to a server in real time. The server compares the data with a facial recognition database and retrieves matching person information from a business card database.

[0250] Analysis of the Emotion Engine

[0251] The emotion engine analyzes facial data captured by the device to identify the user's emotional state. This emotional data is sent to a server and processed along with the person's information.

[0252] Display of Information and Emotion

[0253] The server sends the acquired person information and emotion data from the emotion engine to the device. The device (smart glasses) displays this information on the lenses. It can also provide voice notifications through the speaker.

[0254] Specific example

[0255] 1. User registration and facial data acquisition:

[0256] The user stands still at the entrance of the facility, facing the camera.

[0257] The device will notify you that "face data has been captured" and send the face data to the server.

[0258] The server displays a confirmation message stating, "Face data saving complete," and prompts the user to upload business card data.

[0259] The user uploads business card data, and the device notifies them that "business card data has been sent to the server."

[0260] 2. Authentication within the facility:

[0261] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[0262] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[0263] 3. Facial recognition and data linkage:

[0264] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[0265] The captured facial data is sent to the server.

[0266] The server displays "Matching against facial recognition database" and retrieves matching person information.

[0267] 4. Analysis of the Emotion Engine:

[0268] The emotion engine analyzes the facial data and sends a message to the server saying, "Emotion data has been analyzed."

[0269] The server processes the emotional data as additional information and integrates it with the person's information.

[0270] 5. Displaying information and emotions:

[0271] The server sends the acquired person information and emotional data to the terminal.

[0272] The smart glasses display this information on the lenses. In addition to "Taro Tanaka - Sales Department," information such as "Current emotion: Joy" is displayed. A voice notification also says, "Taro Tanaka, Sales Department. Your current emotion is joy."

[0273] This system allows users to quickly obtain information about others within a specific facility, as well as understand their emotional state. This enables more appropriate communication and is expected to improve work efficiency. Furthermore, by limiting the use of the system to specific facilities, the risk of information leakage can be minimized.

[0274] The following describes the processing flow.

[0275] Step 1:

[0276] The user stands in front of a sensor-equipped terminal at the facility entrance, facing the camera directly, and remains still for approximately 5 seconds.

[0277] Step 2:

[0278] The device captures the user's face. The captured facial data is saved in real time.

[0279] Step 3:

[0280] The device sends the captured facial data to the server. Once the transmission is complete, the device notifies the user that "facial data has been sent to the server."

[0281] Step 4:

[0282] The server stores the received face data in the face authentication database and sends a message saying, "The storage of face data has been completed."

[0283] Step 5:

[0284] The user uploads business card data and the affiliation list to the terminal. The upload is performed through a specified input format.

[0285] Step 6:

[0286] The terminal sends the uploaded business card data to the server. When the transmission is completed, a notification saying, "The business card data has been sent to the server" is displayed.

[0287] Step 7:

[0288] The server stores the sent business card data in the business card database and notifies, "The storage of business card data has been completed."

[0289] Step 8:

[0290] The user stands in front of the terminal to perform face authentication at the entrance of the facility. The terminal performs face authentication in conjunction with the facility access management system.

[0291] Step 9:

[0292] When the face authentication is successful, the terminal notifies, "The authentication has been successful. Please enter."

[0293] Step 10:<00009​​​​​​​​​ The terminal transmits the captured face data to the server in real time. A status message "Transmitting face data" is displayed.

[0297] Step 12:

[0298] The server compares the received face data with the face authentication database and displays a message "Comparing...".

[0299] Step 13:

[0300] The server retrieves the personal information associated with the matching face data from the business card database. A notification "Personal information retrieved" is received.

[0301] Step 14:

[0302] The terminal transmits the captured face data to the emotion engine. A message "Analyzing emotion data" is displayed.

[0303] Step 15:

[0304] The emotion engine analyzes the received face data to identify the user's emotional state. A notification "Emotion data analyzed" is received.

[0305] Step 16:

[0306] The emotion engine transmits the analyzed emotion data to the server. A status message "Transmitting emotion data" is displayed.

[0307] Step 17:

[0308] The server transmits the retrieved personal information and emotion data to the terminal.

[0309] Step 18:

[0310] The device (smart glasses) displays the transmitted person information and emotion data on the lenses. In addition to "Taro Tanaka - Sales Department," information such as "Current emotion: Joy" is displayed.

[0311] Step 19:

[0312] The device (smart glasses) uses its voice output function to announce, "This is Taro Tanaka from the Sales Department. Your current emotion is joy."

[0313] Thus, in addition to integrating facial recognition and a business card database, this system utilizes an emotion engine to provide even deeper information. This allows users to understand not only the information but also the emotional state of others within a specific facility, thereby improving work efficiency and the quality of communication.

[0314] (Example 2)

[0315] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0316] Conventional facial recognition systems can obtain basic information about a person, but they struggle to understand that person's emotional state. Therefore, especially in environments where person-to-person communication is crucial, there is a need for a means to quickly grasp a person's emotional state. Furthermore, there is a need for a method to effectively perform facial recognition and emotion analysis within specific facilities while minimizing the risk of information leakage.

[0317] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0318] In this invention, the server includes means for transmitting facial data to the server, means for comparing it with a facial recognition database, and means for analyzing the facial data to identify emotional states. This makes it possible to understand not only facial recognition but also the emotional states of individuals in real time within a specific facility.

[0319] "Facial data" refers to image data of a person's face captured by a camera.

[0320] A "server" refers to a computer system that stores, analyzes, and manages facial and emotional data.

[0321] A "facial recognition database" is a database stored on a server that uses facial data to authenticate individuals.

[0322] "Personal information" refers to basic information and attribute data about a person, obtained from sources such as business card databases.

[0323] A "terminal" is a hardware device used by a user to perform input and output, including smart glasses and sensor-equipped devices.

[0324] "Smart glasses" are wearable devices that can capture facial data and display information and output audio.

[0325] An "emotion engine" is a combination of software and hardware that analyzes captured facial data to identify emotional states.

[0326] "Emotional data" refers to data that indicates a person's emotional state, as identified by the emotion engine.

[0327] "Specific facilities" refers to physical locations where the use of the system is restricted to minimize the risk of information leakage.

[0328] This invention relates to a system that links facial recognition data and business card data to display or output information and emotional state of a person using smart glasses and an emotion engine within a specific facility. In this system, "facial data" is captured, sent to a server for analysis, and then person information and emotional state are identified.

[0329] Hardware and software to be used

[0330] The following hardware will be used as the terminal:

[0331] Cameras with sensors (installed at facility entrances and on smart glasses)

[0332] Smart glasses (display information on the lenses, with audio output capability)

[0333] Access control system (manages entry and exit to facilities)

[0334] Servers and related software:

[0335] Facial recognition database (a database for storing and matching facial data)

[0336] Business card database (a database for storing personal information)

[0337] Emotion engine (software that analyzes facial data to identify emotional states)

[0338] Information display and voice notification engine (software for displaying information on smart glasses and providing voice notifications)

[0339] Explanation of natural language processing

[0340] 1. Capture and register facial data

[0341] The user stands in front of a sensor-equipped camera at the facility entrance and their face data is captured. The terminal encodes this face data and sends it to the server. The server decodes the face data, compares it with existing data, and stores it in a face recognition database. Next, the server prompts the user to upload business card data, and once the user uploads the business card data, it is stored on the server.

[0342] 2. Authentication within the facility

[0343] When a user enters a facility, the device performs facial recognition and sends the data to the server. The server compares the data with the facial recognition database, and if authentication is successful, the device authorizes the use of the smart glasses. Once the setup is complete, the smart glasses become available for use.

[0344] 3. Facial recognition and data linkage

[0345] As a user moves around the facility wearing smart glasses, the smart glasses capture the faces of people around them in real time. The smart glasses encode the captured face data and send it to a server. The server compares it with a facial recognition database and retrieves matching person information from a business card database, then sends it to the smart glasses.

[0346] 4. Analysis of the Emotional Engine

[0347] The emotion engine analyzes facial data captured by the smart glasses and identifies the emotional state. The identified emotion data is encoded and sent to a server. The server integrates this emotion data with the person's information and sends it back to the smart glasses.

[0348] 5. Display of Information and Emotion

[0349] The smart glasses receive person information and emotion data transmitted from the server and display it on the lenses. They also have a voice notification function.

[0350] Specific example

[0351] 1. User registration and facial data acquisition:

[0352] The user stands still at the entrance of the facility, facing the camera.

[0353] The device will notify you that "face data has been captured" and send the face data to the server.

[0354] The server displays a confirmation message stating, "Face data saving complete," and prompts the user to upload business card data.

[0355] The user uploads business card data, and the device notifies them that "business card data has been sent to the server."

[0356] 2. Authentication within the facility:

[0357] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[0358] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[0359] 3. Facial recognition and data linkage:

[0360] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[0361] The captured facial data is sent to the server.

[0362] The server displays "Matching against facial recognition database" and retrieves information on the matching person.

[0363] 4. Analysis of the Emotion Engine:

[0364] The emotion engine analyzes the facial data and sends a message to the server saying, "Emotion data has been analyzed."

[0365] The server processes the emotional data as additional information and integrates it with the person's information.

[0366] 5. Displaying information and emotions:

[0367] The server sends the acquired person information and emotional data to the terminal.

[0368] The smart glasses display this information on the lenses. In addition to "Employee A - Sales Department," information such as "Current emotion: Joy" is displayed. A voice notification also says, "Employee A, Sales Department. Your current emotion is joy."

[0369] Example of a prompt

[0370] "Please capture facial data at the facility entrance and upload your business card data."

[0371] "Perform real-time facial recognition and emotion analysis while moving around the facility, and display the results on smart glasses."

[0372] "Link facial recognition data with business card data, use an emotion engine to analyze emotional states, and provide notifications via display and audio."

[0373] This system allows users to quickly obtain information about others within a specific facility, as well as understand their emotional state. This enables more appropriate communication and is expected to improve work efficiency. Furthermore, by limiting the use of the system to specific facilities, the risk of information leakage can be minimized.

[0374] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0375] Step 1:

[0376] The user stands in front of a sensor-equipped camera at the facility entrance. The device captures facial data, encodes it, and then sends it to the server.

[0377] [Input] User's face image

[0378] [Data Processing] The device encodes the facial image.

[0379] [Output] Encoded face data is sent to the server.

[0380] Specifically, the camera recognizes the contours of the face, and face capture is performed when the camera remains still for a certain period of time.

[0381] Step 2:

[0382] The server decodes the received facial data and compares it with the facial recognition database. After confirming a match with existing data, it registers it as new data.

[0383] [Input] Encoded facial data sent from the device

[0384] [Data Processing] The server decodes the facial data and performs matching calculations.

[0385] [Output] The new face data is saved to the facial recognition database, and a registration completion message is sent to the device.

[0386] Specifically, the server first decomposes the facial data into multiple feature points and compares them with existing data in the database using a high-speed matching algorithm.

[0387] Step 3:

[0388] The terminal displays a registration completion message from the server and prompts the user to upload business card data. The user uploads the business card data to the terminal, and the terminal sends it to the server.

[0389] [Input] Registration completion message from the server, user's business card data

[0390] [Data Processing] The terminal encodes the business card data.

[0391] [Output] Business card data is sent to the server.

[0392] Specifically, the device displays "Please upload your business card data," and the user uploads a business card image to the device.

[0393] Step 4:

[0394] The server receives business card data, decodes it, and saves it to the business card database. It also notifies the terminal that the business card data has been successfully received.

[0395] [Input] Encoded business card data sent from the terminal

[0396] [Data Processing] The server decodes and saves the business card data.

[0397] [Output] The business card data is saved to the business card database, and a message confirming receipt is sent to the terminal.

[0398] Specifically, the server analyzes business card data using OCR (optical character recognition) technology, extracts important information, and stores it in a database.

[0399] Step 5:

[0400] When a user enters the facility, the terminal performs facial recognition again and sends the data to the server. The server compares the data with the facial recognition database and sends the authentication result back to the terminal.

[0401] [Input] User's face image

[0402] [Data Processing] The terminal encodes the facial image and sends it to the server. The server decodes the facial data and performs matching calculations.

[0403] [Output] Authentication result is sent to the terminal.

[0404] Specifically, the device will display "Authentication in progress," and upon successful authentication, it will notify the user with "Authentication successful."

[0405] Step 6:

[0406] After authentication, the device will authorize the use of the smart glasses and automatically perform pairing and necessary settings.

[0407] [Input] Authentication result from the server

[0408] [Data Processing] The device resets the smart glasses settings and performs pairing.

[0409] [Output] Smart glasses become ready for use.

[0410] Specifically, the device connects to the smart glasses using Bluetooth and performs the initial setup.

[0411] Step 7:

[0412] As the user moves around the facility, the smart glasses continuously capture the faces of people around them in real time and send the facial data to a server.

[0413] [Input] Captured facial data

[0414] [Data Processing] The smart glasses encode the facial data and send it to the server.

[0415] [Output] Encoded face data is sent to the server.

[0416] Specifically, the smart glasses' camera tracks the face in real time and automatically sends the data to a server.

[0417] Step 8:

[0418] The server matches the received facial data, retrieves matching person information from the business card database, and sends it to the smart glasses.

[0419] [Input] Face data transmitted from smart glasses

[0420] [Data Processing] The server matches facial data and retrieves person information.

[0421] [Output] Person information is sent to smart glasses.

[0422] Specifically, the server cross-references the facial recognition database and the business card database to quickly retrieve data.

[0423] Step 9:

[0424] The emotion engine analyzes facial data captured by the smart glasses to identify the emotional state. The identified emotion data is then sent to the server.

[0425] [Input] Captured facial data

[0426] [Data Processing] The emotion engine analyzes facial data and identifies emotional states.

[0427] [Output] Emotional state data is sent to the server.

[0428] Specifically, the emotion engine analyzes facial movements and changes in expression to identify emotional states in real time.

[0429] Step 10:

[0430] The server integrates emotional data with person information and sends it to the smart glasses. The smart glasses display the information on the lenses and provide audio notifications as needed.

[0431] [Input] Emotional state data, person information

[0432] [Data Processing] The server integrates the data and formats it for smart glasses.

[0433] [Output] Display and audio notifications are provided via smart glasses.

[0434] In terms of specific actions, the smart glasses display "Employee A - Sales Department" and "Current Emotion: Joyful" on the lens, and announce via voice, "Employee A, Sales Department. Your current emotion is joyful."

[0435] (Application Example 2)

[0436] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0437] Many existing facial recognition systems only perform person recognition and are insufficient for providing advanced customer service. Especially in physical stores, it is crucial to quickly understand customers' emotional states and provide individualized support, but current systems primarily rely on abstract data and manual information acquisition, lacking practicality. Furthermore, without linking customer information with emotional data, appropriate service recommendations and highly accurate customer service are difficult. This project aims to solve these problems and improve the customer experience and provide efficient service delivery.

[0438] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0439] In this invention, the server includes means for transmitting facial data to the server, means for comparing it with a facial recognition database stored on the server, means for acquiring person information, means for transmitting the acquired person information and emotion data to a terminal, means for displaying or outputting audio to the terminal, means for analyzing facial data with an emotion engine and identifying the emotional state, and means for performing the above operations only within a specific facility. This makes it possible to automatically link customer facial recognition data and business card information to understand the customer's emotional state. As a result, store employees can acquire customer information in real time and provide individually personalized customer service.

[0440] "Facial data" refers to image data of a person's face captured by a camera or other imaging device.

[0441] A "server" is a computer system that manages, stores, and processes data via a network.

[0442] A "facial recognition database" is a database that stores multiple pre-registered facial data and compares them with newly acquired facial data.

[0443] "Personal information" refers to information about individual individuals (e.g., name and affiliation) obtained from sources such as business card databases.

[0444] A "terminal" is a device that a user directly operates and that displays or notifies information via sound. In this invention, smart glasses are the primary example of this type of device.

[0445] An "emotion engine" is software or hardware that analyzes facial data to identify a person's emotional state (e.g., joy, anger, sadness, etc.).

[0446] "Within a specific facility" refers to a location to which the present invention is applied in a limited manner, and means the area to which facial recognition and emotion analysis are to be performed.

[0447] The system for carrying out this invention includes functions for capturing facial data, matching it with a database, analyzing emotions using an emotion engine, displaying the acquired information in real time, and providing voice notifications. The specific configuration and operation details of this system are described below.

[0448] System Configuration

[0449] The user puts on smart glasses.

[0450] The device captures facial data of people in its surroundings using a camera mounted on smart glasses and sends this data to a server.

[0451] The server compares the facial data with a facial recognition database to obtain the person's information.

[0452] The server uses an emotion engine to analyze the captured facial data and identify emotional data.

[0453] The server sends the acquired person information and emotion data to the terminal.

[0454] The device displays this information on the smart glasses' screen and provides voice notifications as needed.

[0455] Examples of specific hardware and software to be used

[0456] Hardware:

[0457] Smart glasses: Capture facial data and display information.

[0458] Server: Performs data processing and manages databases.

[0459] software:

[0460] OpenCV: Image processing library for face recognition

[0461] Requests module: A library for communicating with the server.

[0462] Emotion engine: Software that analyzes emotional states from facial data; a specific example is Microsoft Azure's Emotion API.

[0463] System operation example

[0464] Facial data capture and registration

[0465] The user captures their face with a camera through smart glasses at the facility entrance, and the device notifies them that "face data has been captured." This face data is sent to a server and stored in a face recognition database. Next, the user uploads business card data, and the device notifies them that "business card data has been sent to the server." This business card data is stored in a business card database.

[0466] Authentication and information display within the facility

[0467] As a user moves around a facility wearing smart glasses, the cameras on the smart glasses continuously capture the faces of people in their surroundings. The captured facial data is sent to a server in real time and compared with a facial recognition database. Matching person information is retrieved from a business card database and sent to the device. Furthermore, an emotion engine analyzes the emotional state, and the results are also sent from the server to the device.

[0468] For example, when a user enters a meeting room, the smart glasses display shows "Taro Yamada - Gold Member, Current Emotion: Joyful," along with an audio notification. This information allows the user to respond quickly and appropriately.

[0469] Example of a prompt

[0470] If you want to ask the AI ​​model that generates a specific and immediately usable action plan for this application, use the following prompt:

[0471] "You are developing a system that uses smart glasses to support customer service in physical stores by utilizing facial recognition data and emotion engine analysis results. Please provide recommendations for actions that will allow for more effective customer service based on real-time customer information."

[0472] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0473] Step 1:

[0474] The user puts on smart glasses and faces the camera at the entrance of the facility.

[0475] Input: Capture a face at the facility entrance.

[0476] Data processing: The smart glasses' camera acquires image data of the face.

[0477] Output: Captured facial data is sent to the terminal.

[0478] Step 2:

[0479] The device sends the captured facial data to the server.

[0480] Input: Facial image data acquired by the smart glasses' camera.

[0481] Data processing: Convert the encoded facial image data into a transmission format.

[0482] Output: Face data is sent to the server.

[0483] Step 3:

[0484] The server compares the facial data with a facial recognition database.

[0485] Input: Image data of a face sent from the device.

[0486] Data processing: Matching is performed using a facial recognition algorithm (e.g., a deep learning model).

[0487] Output: Matching person information is retrieved.

[0488] Step 4:

[0489] The server retrieves matching person information from the business card database.

[0490] Input: Person information matching the facial recognition database.

[0491] Data processing: Extract necessary personal information (name, affiliation, etc.) from the business card database.

[0492] Output: Person information is saved on the server.

[0493] Step 5:

[0494] The server uses an emotion engine to analyze facial data and identify emotional data.

[0495] Input: Image data of a face sent from the device.

[0496] Data processing: Using emotion analysis algorithms, we analyze emotional states such as joy, anger, and sadness.

[0497] Output: Emotional data is saved to the server.

[0498] Step 6:

[0499] The server sends the acquired person information and emotional data to the terminal.

[0500] Input: Person information obtained from a business card database and emotional data analyzed by an emotion engine.

[0501] Data processing: Integrate this data and convert it into a transmission format.

[0502] Output: Person information and emotion data are sent to the terminal.

[0503] Step 7:

[0504] The device displays the person information and emotion data it receives on the smart glasses' display.

[0505] Input: Person information and emotion data sent from the server.

[0506] Data processing: Converts received data into a display format.

[0507] Output: The smart glasses display shows "○○ ○○ - Affiliation: △△, Current emotion: Joy." Voice notifications are also provided as needed.

[0508] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0509] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0510] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0511] [Second Embodiment]

[0512] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0513] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0514] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0515] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0516] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0517] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0518] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0519] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0520] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0521] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0522] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0523] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0524] This invention is a system that links facial recognition data and business card data to display or output information about a person using smart glasses within a specific facility.

[0525] Facial data capture and registration

[0526] The user stands in front of a sensor-equipped terminal at the facility entrance, and their face is captured by the camera. The terminal sends the captured facial data to a server. The server stores this facial data in a facial recognition database. The user also uploads business card data and affiliation lists to the terminal and sends them to the server. The server stores this information in a business card database.

[0527] Authentication within the facility

[0528] When a user enters the facility, the terminal works in conjunction with the facility's access control system to perform facial recognition. Once authentication is complete, the terminal authorizes the user to use smart glasses and performs the necessary settings.

[0529] Facial recognition and data linkage

[0530] When a user wears smart glasses and moves around the facility, the smart glasses (device) capture the faces of people around them using its camera. The captured face data is sent to a server in real time. The server compares the data with a facial recognition database and retrieves matching person information from a business card database.

[0531] Displaying information

[0532] The server sends the acquired person information to the device. The device (smart glasses) displays this person information on the lens or notifies the user via voice through the speaker.

[0533] Specific example

[0534] 1. User registration and facial data acquisition:

[0535] The user stands still at the entrance of the facility, facing the camera.

[0536] The device will notify you that "face data has been captured" and send the face data to the server.

[0537] The server displays a confirmation message stating, "Face data saving complete."

[0538] 2. Authentication within the facility:

[0539] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[0540] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[0541] 3. Facial recognition and data linkage:

[0542] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[0543] The captured facial data is sent to the server.

[0544] The server displays "Matching against facial recognition database" and retrieves matching person information.

[0545] 4. Display of information:

[0546] The server sends the retrieved person information (for example, "Taro Tanaka - Sales Department") to the terminal.

[0547] Smart glasses display this information on the lenses. For example, "Taro Tanaka - Sales Department" is visually shown. It can also provide voice notifications such as "Taro Tanaka, Sales Department."

[0548] This system allows users to quickly obtain and verify information about others within a specific facility. Furthermore, by limiting the system's use to specific facilities, the risk of information leakage can be minimized.

[0549] The following describes the processing flow.

[0550] Step 1:

[0551] The user stands in front of a sensor-equipped terminal at the facility entrance, facing the camera directly, and remains still for approximately 5 seconds.

[0552] Step 2:

[0553] The device captures the user's face. The captured facial data is saved to the device in real time.

[0554] Step 3:

[0555] The device sends the captured facial data to the server. Once the transmission is complete, the device notifies the user that "facial data has been sent to the server."

[0556] Step 4:

[0557] The server saves the received facial data to the facial recognition database. Once saving is complete, the server sends a message stating, "Facial data saving complete."

[0558] Step 5:

[0559] Users upload business card data and affiliation lists to their devices. This is done through a specified input format.

[0560] Step 6:

[0561] The device sends the uploaded business card data to the server. Once the transmission is complete, a notification will appear stating, "Business card data has been sent to the server."

[0562] Step 7:

[0563] The server saves the submitted business card data to the business card database. Once saving is complete, the server notifies the user that "Business card data has been saved."

[0564] Step 8:

[0565] The user stands in front of a terminal for facial recognition at the facility entrance. The terminal is linked to the facility's access control system and authenticates the user's face.

[0566] Step 9:

[0567] The device displays a notification saying, "Authentication successful. Please enter," once facial recognition is complete.

[0568] Step 10:

[0569] The user puts on smart glasses and begins moving around the facility. The smart glasses (device) continuously capture the faces of people in the surrounding area using its camera.

[0570] Step 11:

[0571] The device sends the captured facial data to the server in real time. The status "Sending facial data" is displayed.

[0572] Step 12:

[0573] The server compares the received face data with the face recognition database and finds a matching face. It then outputs the message "Matching..." as the processing status.

[0574] Step 13:

[0575] The server retrieves person information linked to matching facial data from the business card database. After retrieval, a notification is sent stating, "Person information has been retrieved."

[0576] Step 14:

[0577] The server sends the acquired person information to the terminal. The status "Sending person information" is displayed.

[0578] Step 15:

[0579] The device (smart glasses) displays the transmitted person information on the lens. It also provides voice notifications via the speaker, such as "This is person AA," if necessary.

[0580] (Example 1)

[0581] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0582] Conventional facial recognition systems require manual recognition and data acquisition, making it difficult to quickly and efficiently obtain information about individuals within a specific facility. Furthermore, the lack of restrictions on system usage increases the risk of data leaks. Additionally, insufficient real-time information sharing and the incomplete integration of current facial recognition data with business card data make the system impractical.

[0583] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0584] In this invention, the server includes means for capturing facial data, means for transmitting facial data to the server, means for comparing it with a facial recognition database stored on the server, means for obtaining person information from a linked database, means for transmitting the obtained person information to a terminal, means for displaying or outputting audio on the terminal, and means for performing the above operations using smart glasses within a specific facility. This makes it possible to quickly and efficiently obtain and verify the information of a person within a specific facility, minimizing the risk of information leakage. Furthermore, real-time information sharing is realized, the linkage between facial recognition data and business card data is enhanced, and practicality is improved.

[0585] "Facial data" refers to digital data that includes images of an individual's face and its characteristics, captured by a camera.

[0586] A "server" is a computer system that stores, manages, and processes data on a network.

[0587] A "facial recognition database" is a dedicated database used to store captured facial data and perform facial recognition based on that data.

[0588] A "terminal" refers to a hardware device operated by the user, and this includes smart glasses and computing devices.

[0589] A "business card database" is a database that stores business card information and related data, and uses this information to provide personal information.

[0590] "Smart glasses" are wearable devices worn on the head that are equipped with cameras and displays and have the function of displaying and capturing information.

[0591] "Within a specific facility" refers to designated areas or the interior of buildings where the use of the system is restricted.

[0592] This invention relates to a system that links facial recognition data with personal information and displays or outputs information about another person using smart glasses within a specific facility. This system allows users to quickly obtain and verify information about others within a designated facility. This minimizes the risk of information leakage and improves practicality.

[0593] Hardware to use

[0594] 1. Facial data capture device: A camera installed in smart glasses (e.g., Microsoft HoloLens, Google Glass).

[0595] 2. Terminal: Smart glasses or a computer device operated by the user.

[0596] 3. Server: A computer system for managing and processing databases (e.g., Linux server, AWS server).

[0597] Software to use

[0598] 1. Facial recognition algorithms: Facial recognition APIs such as OpenCV and Amazon Rekognition.

[0599] 2. Database management systems: such as MySQL and PostgreSQL.

[0600] 3. Communication Protocols: Protocols for real-time communication such as HTTP, WebSocket, and MQTT.

[0601] Specific details of data processing and data calculations

[0602] 1. User registration and facial data acquisition:

[0603] The user stands still at the entrance of the facility, facing the camera.

[0604] The device captures facial data from the camera image and notifies the user that "facial data has been captured."

[0605] The device transmits facial data to the server in real time.

[0606] The server saves the facial data to a MySQL database and notifies the terminal that "Facial data saving is complete."

[0607] 2. Authentication within the facility:

[0608] As the user passes through the facility entrance, they turn their face towards the camera again as they walk.

[0609] The device activates the facial recognition system, sends facial data to the server, and performs authentication.

[0610] The server checks whether the face data matches the data stored in the database and displays "Authentication successful" on the screen.

[0611] Once authentication is successful, the device communicates with the smart glasses and automatically performs the necessary settings (for example, Wi-Fi connection and user profile synchronization).

[0612] 3. Facial recognition and data linkage:

[0613] As the user wears the smart glasses and moves around the facility, the smart glasses' camera captures the faces of people in the surrounding area.

[0614] The captured facial data is transmitted from the smart glasses to the server via a high-speed network.

[0615] The server performs real-time matching against the facial recognition database and retrieves matching person information from the business card database.

[0616] During the matching process, the server displays the current processing progress on the smart glasses.

[0617] 4. Display of information:

[0618] The server sends the acquired person information to the smart glasses using the WebSocket protocol.

[0619] Smart glasses visually display information such as "Taro Tanaka - Sales Department" on the lenses.

[0620] The smart glasses' speaker notifies the user via voice, "Taro Tanaka, Sales Department."

[0621] Users receive visual and audible notifications, allowing them to instantly understand information about people around them.

[0622] Specific example

[0623] 1. User registration and facial data acquisition:

[0624] The user stands still at the entrance of the facility, facing the camera.

[0625] The device will notify you that "face data has been captured" and send the face data to the server.

[0626] The server displays a confirmation message stating, "Face data saving complete."

[0627] 2. Authentication within the facility:

[0628] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[0629] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[0630] 3. Facial recognition and data linkage:

[0631] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[0632] The captured facial data is sent to the server.

[0633] The server displays "Matching against facial recognition database" and retrieves matching person information.

[0634] 4. Display of information:

[0635] The server sends the retrieved person information (for example, "Taro Tanaka - Sales Department") to the terminal.

[0636] Smart glasses display this information on the lenses. For example, "Taro Tanaka - Sales Department" is visually shown.

[0637] It is also possible to notify users via voice message saying, "Taro Tanaka, Sales Department."

[0638] Example of a prompt

[0639] "Please describe a system that captures a user's face at the facility entrance, registers it in a database, and uses smart glasses to display or output information about other users within the facility in real time."

[0640] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0641] Step 1:

[0642] Capture and transmit facial data

[0643] The user stands still at the entrance of the facility, facing the camera. The device captures the face and sends the data to the server. Specifically, the device detects the face from the camera image, performs image processing, and generates face data. This face data is sent to the server in real time via the network. The server stores the received face data in a face recognition database.

[0644] Input: User's facial image (input from camera)

[0645] Data processing: Generating face data through image processing.

[0646] Output: Captured facial data (sent from the device to the server)

[0647] Specific actions:

[0648] The user stands in the designated location, faces the camera, and remains still for a few seconds.

[0649] The device's camera captures facial images, and internal processing generates facial data.

[0650] The device sends the generated facial data to the server.

[0651] The server stores the facial data in a database.

[0652] Step 2:

[0653] Authentication within the facility

[0654] When a user enters a facility, the terminal performs facial recognition. The terminal sends facial data to a server for authentication. The server compares the facial data with the data stored in the database to confirm a match. If authentication is successful, the terminal authorizes the use of smart glasses and performs the necessary settings.

[0655] Input: Entered facial data (captured from the device's camera)

[0656] Data processing: Matching using facial recognition algorithms

[0657] Output: Authentication result (success / failure)

[0658] Specific actions:

[0659] The user turns their face towards the camera as they pass through the facility entrance.

[0660] The device captures the facial data again and sends it to the server.

[0661] The server compares the face data stored in the database and returns the authentication result to the terminal.

[0662] If authentication is successful, the device will display permission to use the smart glasses on the screen and automatically configure the settings.

[0663] Step 3:

[0664] Facial recognition and data linkage

[0665] When a user wears smart glasses and moves around the facility, the smart glasses' camera captures the faces of people in the surrounding area. The captured facial data is sent to a server in real time. The server compares it with a facial recognition database and retrieves matching person information from a business card database.

[0666] Input: Images of people's faces in the surrounding area (input from the smart glasses' camera)

[0667] Data processing: Facial recognition and database matching

[0668] Output: Matching person information

[0669] Specific actions:

[0670] Users wear smart glasses and walk around the facility.

[0671] The smart glasses' camera continuously captures the faces of people in the surrounding area.

[0672] The captured facial data is transmitted from the smart glasses to the server via a high-speed network.

[0673] The server performs real-time matching against the facial recognition database and retrieves matching person information from the business card database.

[0674] Step 4:

[0675] Displaying information

[0676] The server transmits the acquired person information to the device (smart glasses). The device (smart glasses) either displays this information on the lenses or notifies the user via audio through the speaker.

[0677] Input: Matching results (person information)

[0678] Data processing: Information transmission

[0679] Output: Visual display or audio notification

[0680] Specific actions:

[0681] The server sends the acquired person information to the smart glasses using the WebSocket protocol.

[0682] Smart glasses visually display information such as "Taro Tanaka - Sales Department" on the lenses.

[0683] The smart glasses' speaker notifies the user via voice, "Taro Tanaka, Sales Department."

[0684] Users receive visual and audible notifications, allowing them to instantly understand information about people around them.

[0685] (Application Example 1)

[0686] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0687] In conventional systems, when acquiring personal information through facial recognition within a specific facility, the information obtained was limited, making it difficult to instantly retrieve and display detailed customer purchase history or special requests in commercial facilities. Furthermore, centralized information management was difficult, and data collection from various databases was complex. Additionally, delays in providing individual services could occur, posing a risk of lowering customer satisfaction.

[0688] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0689] In this invention, the server includes means for capturing facial data, means for transmitting the facial data to the server, means for matching and retrieving person information stored in a facial recognition database, and means for transmitting customer information, including purchase history and special requests, to a terminal. This enables store employees in commercial facilities to instantly obtain detailed information about visiting customers using smart devices and provide personalized services.

[0690] "Facial data" refers to digital image data that captures a person's face as a key feature.

[0691] A "server" is a computer system that sends, receives, and stores data over a network.

[0692] A "facial recognition database" is a database system that stores captured facial data for later comparison.

[0693] "Personal information" refers to information related to an individual obtained by matching it with a facial recognition database.

[0694] A "terminal" is a device that receives personal information and displays or outputs it to the user as audio.

[0695] "Within a specific commercial facility" refers to a specific physical area where commercial activities take place.

[0696] A "smart device" is an electronic device with advanced functions, such as smart glasses or a smartphone.

[0697] "Purchase history" refers to a record of products and services that a customer has purchased in the past.

[0698] A "special request" refers to a customer's specific request regarding a particular product or service.

[0699] A "purchase history database" is a database system used to store and manage a customer's purchase history.

[0700] This invention relates to a system that recognizes customer facial data within a specific commercial facility, allowing store staff to instantly retrieve and display associated purchase history and special requests. To implement this invention, a server, smart devices, a facial recognition database, a purchase history database, and the like are used.

[0701] System Configuration

[0702] The server stores facial data and matches it against a facial recognition database. It also incorporates a purchase history database that stores and manages purchase history and special requests. This database is used to understand and provide detailed customer information immediately.

[0703] A smart device (e.g., smart glasses) is equipped with a camera that captures the customer's face and a display that shows the information. The smart device has the function of capturing facial data and transmitting that data to a server in real time. It also displays or notifies the customer of the information transmitted from the server via sound.

[0704] Face data capture and transmission

[0705] The user (in this case, a store employee) wears a smart device while working in a commercial facility. When a customer enters the store, the camera on the smart device captures facial data. This facial data is transmitted to a server via the network.

[0706] Facial recognition and data matching

[0707] The server compares the transmitted facial data with a facial recognition database to retrieve matching person information. Furthermore, it extracts the customer's purchase history and special request information from a purchase history database. The retrieved information is immediately sent to the smart device.

[0708] Display and notification of customer information

[0709] The smart device displays personal information transmitted from the server. For example, it might show information such as "Hanako Yamada, Purchase History: Item A, Item B, Special Request: Express Delivery." This allows store staff to provide personalized service quickly. Voice notifications are also available depending on the settings.

[0710] Specific example

[0711] When creating an example prompt for a system in a meeting facility where a store employee wearing smart glasses takes a picture of a newly entering customer's face with a camera and sends the data to a server. The server then compares it with a facial recognition database and displays the person's information on a screen, the input should be: "Prompt: Write Python code to capture customer face data and display that information."

[0712] This invention makes it possible to improve customer service within specific commercial facilities and increase customer satisfaction. This allows for the provision of efficient and prompt service, contributing to the promotion of commercial activities.

[0713] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0714] Step 1:

[0715] A user wears a smart device and waits in a commercial facility. The smart device's camera continuously captures images of people in the surrounding area. The input is the smart device's camera feed, and the output is captured facial data. This facial data is temporarily stored on the smart device.

[0716] Step 2:

[0717] A smart device sends captured facial data to a server over the network. The input is the captured facial data, and the output is a message indicating that the facial data was successfully sent to the server. This data transmission is performed using an HTTP POST request.

[0718] Step 3:

[0719] The server compares the received facial data with a facial recognition database. The input is the newly received facial data, and the output is the matching result. A facial recognition algorithm is used in the matching process. The matching result is either matching person information (if any) or an error message if there is no match.

[0720] Step 4:

[0721] If the server finds a match in the facial recognition database, it retrieves the relevant purchase history and special request information from the purchase history database. The input is the ID of the matched person, and the output is the purchase history and request information of that person. This data is retrieved using SQL queries or similar methods.

[0722] Step 5:

[0723] The server sends customer information (purchase history and special requests) to the smart device. The input is the customer's personal information, and the output is a message indicating successful data transmission to the smart device. This transmission is performed using an HTTP GET request.

[0724] Step 6:

[0725] A smart device receives customer information transmitted from a server and displays it on its screen or provides an audio notification. The input is customer information received from the server, and the output is the information displayed on the screen or an audio notification. This process uses both a display function and a text-to-speech (TTS) function.

[0726] These steps enable the present invention to quickly and efficiently acquire customer information within a specific commercial facility and to provide personalized service by store staff.

[0727] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0728] This invention is a system that links facial recognition data and business card data, and uses smart glasses and an emotion engine within a specific facility to display or output information and emotional state of the person concerned.

[0729] Facial data capture and registration

[0730] The user stands in front of a sensor-equipped terminal at the facility entrance, and their face is captured by the camera. The terminal sends the captured facial data to a server. The server stores this facial data in a facial recognition database. The user also uploads business card data and affiliation lists to the terminal and sends them to the server. The server stores this information in a business card database.

[0731] Authentication within the facility

[0732] When a user enters the facility, the terminal works in conjunction with the facility's access control system to perform facial recognition. Once authentication is complete, the terminal authorizes the user to use smart glasses and performs the necessary settings.

[0733] Facial recognition and data linkage

[0734] As a user moves around the facility wearing smart glasses, the smart glasses (device) continuously capture the faces of people around them using its camera. The captured face data is transmitted to a server in real time. The server compares the data with a facial recognition database and retrieves matching person information from a business card database.

[0735] Analysis of the Emotion Engine

[0736] The emotion engine analyzes facial data captured by the device to identify the user's emotional state. This emotional data is sent to a server and processed along with the person's information.

[0737] Display of Information and Emotion

[0738] The server sends the acquired person information and emotion data from the emotion engine to the device. The device (smart glasses) displays this information on the lenses. It can also provide voice notifications through the speaker.

[0739] Specific example

[0740] 1. User registration and facial data acquisition:

[0741] The user stands still at the entrance of the facility, facing the camera.

[0742] The device will notify you that "face data has been captured" and send the face data to the server.

[0743] The server displays a confirmation message stating, "Face data saving complete," and prompts the user to upload business card data.

[0744] The user uploads business card data, and the device notifies them that "business card data has been sent to the server."

[0745] 2. Authentication within the facility:

[0746] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[0747] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[0748] 3. Facial recognition and data linkage:

[0749] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[0750] The captured facial data is sent to the server.

[0751] The server displays "Matching against facial recognition database" and retrieves matching person information.

[0752] 4. Analysis of the Emotion Engine:

[0753] The emotion engine analyzes the facial data and sends a message to the server saying, "Emotion data has been analyzed."

[0754] The server processes the emotional data as additional information and integrates it with the person's information.

[0755] 5. Displaying information and emotions:

[0756] The server sends the acquired person information and emotional data to the terminal.

[0757] The smart glasses display this information on the lenses. In addition to "Taro Tanaka - Sales Department," information such as "Current emotion: Joy" is displayed. A voice notification also says, "Taro Tanaka, Sales Department. Your current emotion is joy."

[0758] This system allows users to quickly obtain information about others within a specific facility, as well as understand their emotional state. This enables more appropriate communication and is expected to improve work efficiency. Furthermore, by limiting the use of the system to specific facilities, the risk of information leakage can be minimized.

[0759] The following describes the processing flow.

[0760] Step 1:

[0761] The user stands in front of a sensor-equipped terminal at the facility entrance, facing the camera directly, and remains still for approximately 5 seconds.

[0762] Step 2:

[0763] The device captures the user's face. The captured facial data is saved in real time.

[0764] Step 3:

[0765] The device sends the captured facial data to the server. Once the transmission is complete, the device notifies the user that "facial data has been sent to the server."

[0766] Step 4:

[0767] The server saves the received facial data to the facial recognition database. It then sends a message stating, "Facial data saving complete."

[0768] Step 5:

[0769] Users upload business card data and affiliation lists to their devices. The upload is performed through a specified input format.

[0770] Step 6:

[0771] The device sends the uploaded business card data to the server. Once the transmission is complete, a notification will appear stating, "Business card data has been sent to the server."

[0772] Step 7:

[0773] The server saves the submitted business card data to the business card database. It then sends a notification stating, "Business card data saving complete."

[0774] Step 8:

[0775] The user stands in front of a terminal for facial recognition at the facility entrance. The terminal works in conjunction with the facility's access control system to perform facial recognition.

[0776] Step 9:

[0777] The device will notify you with the message, "Authentication successful. Please enter," once facial recognition is complete.

[0778] Step 10:

[0779] The user puts on smart glasses and begins moving around the facility. The smart glasses (device) continuously capture the faces of people in the surrounding area using its camera.

[0780] Step 11:

[0781] The device sends the captured facial data to the server in real time. The status "Sending facial data" is displayed.

[0782] Step 12:

[0783] The server compares the received facial data with the facial recognition database and displays the message "Matching...".

[0784] Step 13:

[0785] The server retrieves the person's information linked to the matched facial data from the business card database. A notification is sent stating, "Person's information has been retrieved."

[0786] Step 14:

[0787] The device sends the captured facial data to the emotion engine. A message appears stating, "Analyzing emotion data."

[0788] Step 15:

[0789] The emotion engine analyzes the received facial data to identify the user's emotional state. A notification is then displayed stating, "Emotional data has been analyzed."

[0790] Step 16:

[0791] The emotion engine sends the analyzed emotion data to the server. The status "Sending emotion data" is displayed.

[0792] Step 17:

[0793] The server sends the acquired person information and emotion data to the terminal.

[0794] Step 18:

[0795] The device (smart glasses) displays the transmitted person information and emotion data on the lenses. In addition to "Taro Tanaka - Sales Department," information such as "Current emotion: Joy" is displayed.

[0796] Step 19:

[0797] The device (smart glasses) uses its voice output function to announce, "This is Taro Tanaka from the Sales Department. Your current emotion is joy."

[0798] Thus, in addition to integrating facial recognition and a business card database, this system utilizes an emotion engine to provide even deeper information. This allows users to understand not only the information but also the emotional state of others within a specific facility, thereby improving work efficiency and the quality of communication.

[0799] (Example 2)

[0800] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0801] Conventional facial recognition systems can obtain basic information about a person, but they struggle to understand that person's emotional state. Therefore, especially in environments where person-to-person communication is crucial, there is a need for a means to quickly grasp a person's emotional state. Furthermore, there is a need for a method to effectively perform facial recognition and emotion analysis within specific facilities while minimizing the risk of information leakage.

[0802] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0803] In this invention, the server includes means for transmitting facial data to the server, means for comparing it with a facial recognition database, and means for analyzing the facial data to identify emotional states. This makes it possible to understand not only facial recognition but also the emotional states of individuals in real time within a specific facility.

[0804] "Facial data" refers to image data of a person's face captured by a camera.

[0805] A "server" refers to a computer system that stores, analyzes, and manages facial and emotional data.

[0806] A "facial recognition database" is a database stored on a server that uses facial data to authenticate individuals.

[0807] "Personal information" refers to basic information and attribute data about a person, obtained from sources such as business card databases.

[0808] A "terminal" is a hardware device used by a user to perform input and output, including smart glasses and sensor-equipped devices.

[0809] "Smart glasses" are wearable devices that can capture facial data and display information and output audio.

[0810] An "emotion engine" is a combination of software and hardware that analyzes captured facial data to identify emotional states.

[0811] "Emotional data" refers to data that indicates a person's emotional state, as identified by the emotion engine.

[0812] "Specific facilities" refers to physical locations where the use of the system is restricted to minimize the risk of information leakage.

[0813] This invention relates to a system that links facial recognition data and business card data to display or output information and emotional state of a person using smart glasses and an emotion engine within a specific facility. In this system, "facial data" is captured, sent to a server for analysis, and then person information and emotional state are identified.

[0814] Hardware and software to be used

[0815] The following hardware will be used as the terminal:

[0816] Cameras with sensors (installed at facility entrances and on smart glasses)

[0817] Smart glasses (display information on the lenses, with audio output capability)

[0818] Access control system (manages entry and exit to facilities)

[0819] Servers and related software:

[0820] Facial recognition database (a database for storing and matching facial data)

[0821] Business card database (a database for storing personal information)

[0822] Emotion engine (software that analyzes facial data to identify emotional states)

[0823] Information display and voice notification engine (software for displaying information on smart glasses and providing voice notifications)

[0824] Explanation of natural language processing

[0825] 1. Capture and register facial data

[0826] The user stands in front of a sensor-equipped camera at the facility entrance and their face data is captured. The terminal encodes this face data and sends it to the server. The server decodes the face data, compares it with existing data, and stores it in a face recognition database. Next, the server prompts the user to upload business card data, and once the user uploads the business card data, it is stored on the server.

[0827] 2. Authentication within the facility

[0828] When a user enters a facility, the device performs facial recognition and sends the data to the server. The server compares the data with the facial recognition database, and if authentication is successful, the device authorizes the use of the smart glasses. Once the setup is complete, the smart glasses become available for use.

[0829] 3. Facial recognition and data linkage

[0830] As a user moves around the facility wearing smart glasses, the smart glasses capture the faces of people around them in real time. The smart glasses encode the captured face data and send it to a server. The server compares it with a facial recognition database and retrieves matching person information from a business card database, then sends it to the smart glasses.

[0831] 4. Analysis of the Emotional Engine

[0832] The emotion engine analyzes facial data captured by the smart glasses and identifies the emotional state. The identified emotion data is encoded and sent to a server. The server integrates this emotion data with the person's information and sends it back to the smart glasses.

[0833] 5. Display of Information and Emotion

[0834] The smart glasses receive person information and emotion data transmitted from the server and display it on the lenses. They also have a voice notification function.

[0835] Specific example

[0836] 1. User registration and facial data acquisition:

[0837] The user stands still at the entrance of the facility, facing the camera.

[0838] The device will notify you that "face data has been captured" and send the face data to the server.

[0839] The server displays a confirmation message stating, "Face data saving complete," and prompts the user to upload business card data.

[0840] The user uploads business card data, and the device notifies them that "business card data has been sent to the server."

[0841] 2. Authentication within the facility:

[0842] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[0843] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[0844] 3. Facial recognition and data linkage:

[0845] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[0846] The captured facial data is sent to the server.

[0847] The server displays "Matching against facial recognition database" and retrieves information on the matching person.

[0848] 4. Analysis of the Emotion Engine:

[0849] The emotion engine analyzes the facial data and sends a message to the server saying, "Emotion data has been analyzed."

[0850] The server processes the emotional data as additional information and integrates it with the person's information.

[0851] 5. Displaying information and emotions:

[0852] The server sends the acquired person information and emotional data to the terminal.

[0853] The smart glasses display this information on the lenses. In addition to "Employee A - Sales Department," information such as "Current emotion: Joy" is displayed. A voice notification also says, "Employee A, Sales Department. Your current emotion is joy."

[0854] Example of a prompt

[0855] "Please capture facial data at the facility entrance and upload your business card data."

[0856] "Perform real-time facial recognition and emotion analysis while moving around the facility, and display the results on smart glasses."

[0857] "Link facial recognition data with business card data, use an emotion engine to analyze emotional states, and provide notifications via display and audio."

[0858] This system allows users to quickly obtain information about others within a specific facility, as well as understand their emotional state. This enables more appropriate communication and is expected to improve work efficiency. Furthermore, by limiting the use of the system to specific facilities, the risk of information leakage can be minimized.

[0859] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0860] Step 1:

[0861] The user stands in front of a sensor-equipped camera at the facility entrance. The device captures facial data, encodes it, and then sends it to the server.

[0862] [Input] User's face image

[0863] [Data Processing] The device encodes the facial image.

[0864] [Output] Encoded face data is sent to the server.

[0865] Specifically, the camera recognizes the contours of the face, and face capture is performed when the camera remains still for a certain period of time.

[0866] Step 2:

[0867] The server decodes the received facial data and compares it with the facial recognition database. After confirming a match with existing data, it registers it as new data.

[0868] [Input] Encoded facial data sent from the device

[0869] [Data Processing] The server decodes the facial data and performs matching calculations.

[0870] [Output] The new face data is saved to the facial recognition database, and a registration completion message is sent to the device.

[0871] Specifically, the server first decomposes the facial data into multiple feature points and compares them with existing data in the database using a high-speed matching algorithm.

[0872] Step 3:

[0873] The terminal displays a registration completion message from the server and prompts the user to upload business card data. The user uploads the business card data to the terminal, and the terminal sends it to the server.

[0874] [Input] Registration completion message from the server, user's business card data

[0875] [Data Processing] The terminal encodes the business card data.

[0876] [Output] Business card data is sent to the server.

[0877] Specifically, the device displays "Please upload your business card data," and the user uploads a business card image to the device.

[0878] Step 4:

[0879] The server receives business card data, decodes it, and saves it to the business card database. It also notifies the terminal that the business card data has been successfully received.

[0880] [Input] Encoded business card data sent from the terminal

[0881] [Data Processing] The server decodes and saves the business card data.

[0882] [Output] The business card data is saved to the business card database, and a message confirming receipt is sent to the terminal.

[0883] Specifically, the server analyzes business card data using OCR (optical character recognition) technology, extracts important information, and stores it in a database.

[0884] Step 5:

[0885] When a user enters the facility, the terminal performs facial recognition again and sends the data to the server. The server compares the data with the facial recognition database and sends the authentication result back to the terminal.

[0886] [Input] User's face image

[0887] [Data Processing] The terminal encodes the facial image and sends it to the server. The server decodes the facial data and performs matching calculations.

[0888] [Output] Authentication result is sent to the terminal.

[0889] Specifically, the device will display "Authentication in progress," and upon successful authentication, it will notify the user with "Authentication successful."

[0890] Step 6:

[0891] After authentication, the device will authorize the use of the smart glasses and automatically perform pairing and necessary settings.

[0892] [Input] Authentication result from the server

[0893] [Data Processing] The device resets the smart glasses settings and performs pairing.

[0894] [Output] Smart glasses become ready for use.

[0895] Specifically, the device connects to the smart glasses using Bluetooth and performs the initial setup.

[0896] Step 7:

[0897] As the user moves around the facility, the smart glasses continuously capture the faces of people around them in real time and send the facial data to a server.

[0898] [Input] Captured facial data

[0899] [Data Processing] The smart glasses encode the facial data and send it to the server.

[0900] [Output] Encoded face data is sent to the server.

[0901] Specifically, the smart glasses' camera tracks the face in real time and automatically sends the data to a server.

[0902] Step 8:

[0903] The server matches the received facial data, retrieves matching person information from the business card database, and sends it to the smart glasses.

[0904] [Input] Face data transmitted from smart glasses

[0905] [Data Processing] The server matches facial data and retrieves person information.

[0906] [Output] Person information is sent to smart glasses.

[0907] Specifically, the server cross-references the facial recognition database and the business card database to quickly retrieve data.

[0908] Step 9:

[0909] The emotion engine analyzes facial data captured by the smart glasses to identify the emotional state. The identified emotion data is then sent to the server.

[0910] [Input] Captured facial data

[0911] [Data Processing] The emotion engine analyzes facial data and identifies emotional states.

[0912] [Output] Emotional state data is sent to the server.

[0913] Specifically, the emotion engine analyzes facial movements and changes in expression to identify emotional states in real time.

[0914] Step 10:

[0915] The server integrates emotional data with person information and sends it to the smart glasses. The smart glasses display the information on the lenses and provide audio notifications as needed.

[0916] [Input] Emotional state data, person information

[0917] [Data Processing] The server integrates the data and formats it for smart glasses.

[0918] [Output] Display and audio notifications are provided via smart glasses.

[0919] In terms of specific actions, the smart glasses display "Employee A - Sales Department" and "Current Emotion: Joyful" on the lens, and announce via voice, "Employee A, Sales Department. Your current emotion is joyful."

[0920] (Application Example 2)

[0921] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0922] Many existing facial recognition systems only perform person recognition and are insufficient for providing advanced customer service. Especially in physical stores, it is crucial to quickly understand customers' emotional states and provide individualized support, but current systems primarily rely on abstract data and manual information acquisition, lacking practicality. Furthermore, without linking customer information with emotional data, appropriate service recommendations and highly accurate customer service are difficult. This project aims to solve these problems and improve the customer experience and provide efficient service delivery.

[0923] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0924] In this invention, the server includes means for transmitting facial data to the server, means for comparing it with a facial recognition database stored on the server, means for acquiring person information, means for transmitting the acquired person information and emotion data to a terminal, means for displaying or outputting audio to the terminal, means for analyzing facial data with an emotion engine and identifying the emotional state, and means for performing the above operations only within a specific facility. This makes it possible to automatically link customer facial recognition data and business card information to understand the customer's emotional state. As a result, store employees can acquire customer information in real time and provide individually personalized customer service.

[0925] "Facial data" refers to image data of a person's face captured by a camera or other imaging device.

[0926] A "server" is a computer system that manages, stores, and processes data via a network.

[0927] A "facial recognition database" is a database that stores multiple pre-registered facial data and compares them with newly acquired facial data.

[0928] "Personal information" refers to information about individual individuals (e.g., name and affiliation) obtained from sources such as business card databases.

[0929] A "terminal" is a device that a user directly operates and that displays or notifies information via sound. In this invention, smart glasses are the primary example of this type of device.

[0930] An "emotion engine" is software or hardware that analyzes facial data to identify a person's emotional state (e.g., joy, anger, sadness, etc.).

[0931] "Within a specific facility" refers to a location to which the present invention is applied in a limited manner, and means the area to which facial recognition and emotion analysis are to be performed.

[0932] The system for carrying out this invention includes functions for capturing facial data, matching it with a database, analyzing emotions using an emotion engine, displaying the acquired information in real time, and providing voice notifications. The specific configuration and operation details of this system are described below.

[0933] System Configuration

[0934] The user puts on smart glasses.

[0935] The device captures facial data of people in its surroundings using a camera mounted on smart glasses and sends this data to a server.

[0936] The server compares the facial data with a facial recognition database to obtain the person's information.

[0937] The server uses an emotion engine to analyze the captured facial data and identify emotional data.

[0938] The server sends the acquired person information and emotion data to the terminal.

[0939] The device displays this information on the smart glasses' screen and provides voice notifications as needed.

[0940] Examples of specific hardware and software to be used

[0941] Hardware:

[0942] Smart glasses: Capture facial data and display information.

[0943] Server: Performs data processing and manages databases.

[0944] software:

[0945] OpenCV: Image processing library for face recognition

[0946] Requests module: A library for communicating with the server.

[0947] Emotion engine: Software that analyzes emotional states from facial data; a specific example is Microsoft Azure's Emotion API.

[0948] System operation example

[0949] Facial data capture and registration

[0950] The user captures their face with a camera through smart glasses at the facility entrance, and the device notifies them that "face data has been captured." This face data is sent to a server and stored in a face recognition database. Next, the user uploads business card data, and the device notifies them that "business card data has been sent to the server." This business card data is stored in a business card database.

[0951] Authentication and information display within the facility

[0952] As a user moves around a facility wearing smart glasses, the cameras on the smart glasses continuously capture the faces of people in their surroundings. The captured facial data is sent to a server in real time and compared with a facial recognition database. Matching person information is retrieved from a business card database and sent to the device. Furthermore, an emotion engine analyzes the emotional state, and the results are also sent from the server to the device.

[0953] For example, when a user enters a meeting room, the smart glasses display shows "Taro Yamada - Gold Member, Current Emotion: Joyful," along with an audio notification. This information allows the user to respond quickly and appropriately.

[0954] Example of a prompt

[0955] If you want to ask the AI ​​model that generates a specific and immediately usable action plan for this application, use the following prompt:

[0956] "You are developing a system that uses smart glasses to support customer service in physical stores by utilizing facial recognition data and emotion engine analysis results. Please provide recommendations for actions that will allow for more effective customer service based on real-time customer information."

[0957] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0958] Step 1:

[0959] The user puts on smart glasses and faces the camera at the entrance of the facility.

[0960] Input: Capture a face at the facility entrance.

[0961] Data processing: The smart glasses' camera acquires image data of the face.

[0962] Output: Captured facial data is sent to the terminal.

[0963] Step 2:

[0964] The device sends the captured facial data to the server.

[0965] Input: Facial image data acquired by the smart glasses' camera.

[0966] Data processing: Convert the encoded facial image data into a transmission format.

[0967] Output: Face data is sent to the server.

[0968] Step 3:

[0969] The server compares the facial data with a facial recognition database.

[0970] Input: Image data of a face sent from the device.

[0971] Data processing: Matching is performed using a facial recognition algorithm (e.g., a deep learning model).

[0972] Output: Matching person information is retrieved.

[0973] Step 4:

[0974] The server retrieves matching person information from the business card database.

[0975] Input: Person information matching the facial recognition database.

[0976] Data processing: Extract necessary personal information (name, affiliation, etc.) from the business card database.

[0977] Output: Person information is saved on the server.

[0978] Step 5:

[0979] The server uses an emotion engine to analyze facial data and identify emotional data.

[0980] Input: Image data of a face sent from the device.

[0981] Data processing: Using emotion analysis algorithms, we analyze emotional states such as joy, anger, and sadness.

[0982] Output: Emotional data is saved to the server.

[0983] Step 6:

[0984] The server sends the acquired person information and emotional data to the terminal.

[0985] Input: Person information obtained from a business card database and emotional data analyzed by an emotion engine.

[0986] Data processing: Integrate this data and convert it into a transmission format.

[0987] Output: Person information and emotion data are sent to the terminal.

[0988] Step 7:

[0989] The device displays the person information and emotion data it receives on the smart glasses' display.

[0990] Input: Person information and emotion data sent from the server.

[0991] Data processing: Converts received data into a display format.

[0992] Output: The smart glasses display shows "○○ ○○ - Affiliation: △△, Current emotion: Joy." Voice notifications are also provided as needed.

[0993] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0994] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0995] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0996] [Third Embodiment]

[0997] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0998] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0999] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1000] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1001] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1002] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1003] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1004] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1005] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1006] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1007] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1008] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1009] This invention is a system that links facial recognition data and business card data to display or output information about a person using smart glasses within a specific facility.

[1010] Facial data capture and registration

[1011] The user stands in front of a sensor-equipped terminal at the facility entrance, and their face is captured by the camera. The terminal sends the captured facial data to a server. The server stores this facial data in a facial recognition database. The user also uploads business card data and affiliation lists to the terminal and sends them to the server. The server stores this information in a business card database.

[1012] Authentication within the facility

[1013] When a user enters the facility, the terminal works in conjunction with the facility's access control system to perform facial recognition. Once authentication is complete, the terminal authorizes the user to use smart glasses and performs the necessary settings.

[1014] Facial recognition and data linkage

[1015] When a user wears smart glasses and moves around the facility, the smart glasses (device) capture the faces of people around them using its camera. The captured face data is sent to a server in real time. The server compares the data with a facial recognition database and retrieves matching person information from a business card database.

[1016] Displaying information

[1017] The server sends the acquired person information to the device. The device (smart glasses) displays this person information on the lens or notifies the user via voice through the speaker.

[1018] Specific example

[1019] 1. User registration and facial data acquisition:

[1020] The user stands still at the entrance of the facility, facing the camera.

[1021] The device will notify you that "face data has been captured" and send the face data to the server.

[1022] The server displays a confirmation message stating, "Face data saving complete."

[1023] 2. Authentication within the facility:

[1024] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[1025] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[1026] 3. Facial recognition and data linkage:

[1027] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[1028] The captured facial data is sent to the server.

[1029] The server displays "Matching against facial recognition database" and retrieves matching person information.

[1030] 4. Display of information:

[1031] The server sends the retrieved person information (for example, "Taro Tanaka - Sales Department") to the terminal.

[1032] Smart glasses display this information on the lenses. For example, "Taro Tanaka - Sales Department" is visually shown. It can also provide voice notifications such as "Taro Tanaka, Sales Department."

[1033] This system allows users to quickly obtain and verify information about others within a specific facility. Furthermore, by limiting the system's use to specific facilities, the risk of information leakage can be minimized.

[1034] The following describes the processing flow.

[1035] Step 1:

[1036] The user stands in front of a sensor-equipped terminal at the facility entrance, facing the camera directly, and remains still for approximately 5 seconds.

[1037] Step 2:

[1038] The device captures the user's face. The captured facial data is saved to the device in real time.

[1039] Step 3:

[1040] The device sends the captured facial data to the server. Once the transmission is complete, the device notifies the user that "facial data has been sent to the server."

[1041] Step 4:

[1042] The server saves the received facial data to the facial recognition database. Once saving is complete, the server sends a message stating, "Facial data saving complete."

[1043] Step 5:

[1044] Users upload business card data and affiliation lists to their devices. This is done through a specified input format.

[1045] Step 6:

[1046] The device sends the uploaded business card data to the server. Once the transmission is complete, a notification will appear stating, "Business card data has been sent to the server."

[1047] Step 7:

[1048] The server saves the submitted business card data to the business card database. Once saving is complete, the server notifies the user that "Business card data has been saved."

[1049] Step 8:

[1050] The user stands in front of a terminal for facial recognition at the facility entrance. The terminal is linked to the facility's access control system and authenticates the user's face.

[1051] Step 9:

[1052] The device displays a notification saying, "Authentication successful. Please enter," once facial recognition is complete.

[1053] Step 10:

[1054] The user puts on smart glasses and begins moving around the facility. The smart glasses (device) continuously capture the faces of people in the surrounding area using its camera.

[1055] Step 11:

[1056] The device sends the captured facial data to the server in real time. The status "Sending facial data" is displayed.

[1057] Step 12:

[1058] The server compares the received face data with the face recognition database and finds a matching face. It then outputs the message "Matching..." as the processing status.

[1059] Step 13:

[1060] The server retrieves person information linked to matching facial data from the business card database. After retrieval, a notification is sent stating, "Person information has been retrieved."

[1061] Step 14:

[1062] The server sends the acquired person information to the terminal. The status "Sending person information" is displayed.

[1063] Step 15:

[1064] The device (smart glasses) displays the transmitted person information on the lens. It also provides voice notifications via the speaker, such as "This is person AA," if necessary.

[1065] (Example 1)

[1066] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1067] Conventional facial recognition systems require manual recognition and data acquisition, making it difficult to quickly and efficiently obtain information about individuals within a specific facility. Furthermore, the lack of restrictions on system usage increases the risk of data leaks. Additionally, insufficient real-time information sharing and the incomplete integration of current facial recognition data with business card data make the system impractical.

[1068] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1069] In this invention, the server includes means for capturing facial data, means for transmitting facial data to the server, means for comparing it with a facial recognition database stored on the server, means for obtaining person information from a linked database, means for transmitting the obtained person information to a terminal, means for displaying or outputting audio on the terminal, and means for performing the above operations using smart glasses within a specific facility. This makes it possible to quickly and efficiently obtain and verify the information of a person within a specific facility, minimizing the risk of information leakage. Furthermore, real-time information sharing is realized, the linkage between facial recognition data and business card data is enhanced, and practicality is improved.

[1070] "Facial data" refers to digital data that includes images of an individual's face and its characteristics, captured by a camera.

[1071] A "server" is a computer system that stores, manages, and processes data on a network.

[1072] A "facial recognition database" is a dedicated database used to store captured facial data and perform facial recognition based on that data.

[1073] A "terminal" refers to a hardware device operated by the user, and this includes smart glasses and computing devices.

[1074] A "business card database" is a database that stores business card information and related data, and uses this information to provide personal information.

[1075] "Smart glasses" are wearable devices worn on the head that are equipped with cameras and displays and have the function of displaying and capturing information.

[1076] "Within a specific facility" refers to designated areas or the interior of buildings where the use of the system is restricted.

[1077] This invention relates to a system that links facial recognition data with personal information and displays or outputs information about another person using smart glasses within a specific facility. This system allows users to quickly obtain and verify information about others within a designated facility. This minimizes the risk of information leakage and improves practicality.

[1078] Hardware to use

[1079] 1. Facial data capture device: A camera installed in smart glasses (e.g., Microsoft HoloLens, Google Glass).

[1080] 2. Terminal: Smart glasses or a computer device operated by the user.

[1081] 3. Server: A computer system for managing and processing databases (e.g., Linux server, AWS server).

[1082] Software to use

[1083] 1. Facial recognition algorithms: Facial recognition APIs such as OpenCV and Amazon Rekognition.

[1084] 2. Database management systems: such as MySQL and PostgreSQL.

[1085] 3. Communication Protocols: Protocols for real-time communication such as HTTP, WebSocket, and MQTT.

[1086] Specific details of data processing and data calculations

[1087] 1. User registration and facial data acquisition:

[1088] The user stands still at the entrance of the facility, facing the camera.

[1089] The device captures facial data from the camera image and notifies the user that "facial data has been captured."

[1090] The device transmits facial data to the server in real time.

[1091] The server saves the facial data to a MySQL database and notifies the terminal that "Facial data saving is complete."

[1092] 2. Authentication within the facility:

[1093] As the user passes through the facility entrance, they turn their face towards the camera again as they walk.

[1094] The device activates the facial recognition system, sends facial data to the server, and performs authentication.

[1095] The server checks whether the face data matches the data stored in the database and displays "Authentication successful" on the screen.

[1096] Once authentication is successful, the device communicates with the smart glasses and automatically performs the necessary settings (for example, Wi-Fi connection and user profile synchronization).

[1097] 3. Facial recognition and data linkage:

[1098] As the user wears the smart glasses and moves around the facility, the smart glasses' camera captures the faces of people in the surrounding area.

[1099] The captured facial data is transmitted from the smart glasses to the server via a high-speed network.

[1100] The server performs real-time matching against the facial recognition database and retrieves matching person information from the business card database.

[1101] During the matching process, the server displays the current processing progress on the smart glasses.

[1102] 4. Display of information:

[1103] The server sends the acquired person information to the smart glasses using the WebSocket protocol.

[1104] Smart glasses visually display information such as "Taro Tanaka - Sales Department" on the lenses.

[1105] The smart glasses' speaker notifies the user via voice, "Taro Tanaka, Sales Department."

[1106] Users receive visual and audible notifications, allowing them to instantly understand information about people around them.

[1107] Specific example

[1108] 1. User registration and facial data acquisition:

[1109] The user stands still at the entrance of the facility, facing the camera.

[1110] The device will notify you that "face data has been captured" and send the face data to the server.

[1111] The server displays a confirmation message stating, "Face data saving complete."

[1112] 2. Authentication within the facility:

[1113] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[1114] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[1115] 3. Facial recognition and data linkage:

[1116] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[1117] The captured facial data is sent to the server.

[1118] The server displays "Matching against facial recognition database" and retrieves matching person information.

[1119] 4. Display of information:

[1120] The server sends the retrieved person information (for example, "Taro Tanaka - Sales Department") to the terminal.

[1121] Smart glasses display this information on the lenses. For example, "Taro Tanaka - Sales Department" is visually shown.

[1122] It is also possible to notify users via voice message saying, "Taro Tanaka, Sales Department."

[1123] Example of a prompt

[1124] "Please describe a system that captures a user's face at the facility entrance, registers it in a database, and uses smart glasses to display or output information about other users within the facility in real time."

[1125] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1126] Step 1:

[1127] Capture and transmit facial data

[1128] The user stands still at the entrance of the facility, facing the camera. The device captures the face and sends the data to the server. Specifically, the device detects the face from the camera image, performs image processing, and generates face data. This face data is sent to the server in real time via the network. The server stores the received face data in a face recognition database.

[1129] Input: User's facial image (input from camera)

[1130] Data processing: Generating face data through image processing.

[1131] Output: Captured facial data (sent from the device to the server)

[1132] Specific actions:

[1133] The user stands in the designated location, faces the camera, and remains still for a few seconds.

[1134] The device's camera captures facial images, and internal processing generates facial data.

[1135] The device sends the generated facial data to the server.

[1136] The server stores the facial data in a database.

[1137] Step 2:

[1138] Authentication within the facility

[1139] When a user enters a facility, the terminal performs facial recognition. The terminal sends facial data to a server for authentication. The server compares the facial data with the data stored in the database to confirm a match. If authentication is successful, the terminal authorizes the use of smart glasses and performs the necessary settings.

[1140] Input: Entered facial data (captured from the device's camera)

[1141] Data processing: Matching using facial recognition algorithms

[1142] Output: Authentication result (success / failure)

[1143] Specific actions:

[1144] The user turns their face towards the camera as they pass through the facility entrance.

[1145] The device captures the facial data again and sends it to the server.

[1146] The server compares the face data stored in the database and returns the authentication result to the terminal.

[1147] If authentication is successful, the device will display permission to use the smart glasses on the screen and automatically configure the settings.

[1148] Step 3:

[1149] Facial recognition and data linkage

[1150] When a user wears smart glasses and moves around the facility, the smart glasses' camera captures the faces of people in the surrounding area. The captured facial data is sent to a server in real time. The server compares it with a facial recognition database and retrieves matching person information from a business card database.

[1151] Input: Images of people's faces in the surrounding area (input from the smart glasses' camera)

[1152] Data processing: Facial recognition and database matching

[1153] Output: Matching person information

[1154] Specific actions:

[1155] Users wear smart glasses and walk around the facility.

[1156] The smart glasses' camera continuously captures the faces of people in the surrounding area.

[1157] The captured facial data is transmitted from the smart glasses to the server via a high-speed network.

[1158] The server performs real-time matching against the facial recognition database and retrieves matching person information from the business card database.

[1159] Step 4:

[1160] Displaying information

[1161] The server transmits the acquired person information to the device (smart glasses). The device (smart glasses) either displays this information on the lenses or notifies the user via audio through the speaker.

[1162] Input: Matching results (person information)

[1163] Data processing: Information transmission

[1164] Output: Visual display or audio notification

[1165] Specific actions:

[1166] The server sends the acquired person information to the smart glasses using the WebSocket protocol.

[1167] Smart glasses visually display information such as "Taro Tanaka - Sales Department" on the lenses.

[1168] The smart glasses' speaker notifies the user via voice, "Taro Tanaka, Sales Department."

[1169] Users receive visual and audible notifications, allowing them to instantly understand information about people around them.

[1170] (Application Example 1)

[1171] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1172] In conventional systems, when acquiring personal information through facial recognition within a specific facility, the information obtained was limited, making it difficult to instantly retrieve and display detailed customer purchase history or special requests in commercial facilities. Furthermore, centralized information management was difficult, and data collection from various databases was complex. Additionally, delays in providing individual services could occur, posing a risk of lowering customer satisfaction.

[1173] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1174] In this invention, the server includes means for capturing facial data, means for transmitting the facial data to the server, means for matching and retrieving person information stored in a facial recognition database, and means for transmitting customer information, including purchase history and special requests, to a terminal. This enables store employees in commercial facilities to instantly obtain detailed information about visiting customers using smart devices and provide personalized services.

[1175] "Facial data" refers to digital image data that captures a person's face as a key feature.

[1176] A "server" is a computer system that sends, receives, and stores data over a network.

[1177] A "facial recognition database" is a database system that stores captured facial data for later comparison.

[1178] "Personal information" refers to information related to an individual obtained by matching it with a facial recognition database.

[1179] A "terminal" is a device that receives personal information and displays or outputs it to the user as audio.

[1180] "Within a specific commercial facility" refers to a specific physical area where commercial activities take place.

[1181] A "smart device" is an electronic device with advanced functions, such as smart glasses or a smartphone.

[1182] "Purchase history" refers to a record of products and services that a customer has purchased in the past.

[1183] A "special request" refers to a customer's specific request regarding a particular product or service.

[1184] A "purchase history database" is a database system used to store and manage a customer's purchase history.

[1185] This invention relates to a system that recognizes customer facial data within a specific commercial facility, allowing store staff to instantly retrieve and display associated purchase history and special requests. To implement this invention, a server, smart devices, a facial recognition database, a purchase history database, and the like are used.

[1186] System Configuration

[1187] The server stores facial data and matches it against a facial recognition database. It also incorporates a purchase history database that stores and manages purchase history and special requests. This database is used to understand and provide detailed customer information immediately.

[1188] A smart device (e.g., smart glasses) is equipped with a camera that captures the customer's face and a display that shows the information. The smart device has the function of capturing facial data and transmitting that data to a server in real time. It also displays or notifies the customer of the information transmitted from the server via sound.

[1189] Face data capture and transmission

[1190] The user (in this case, a store employee) wears a smart device while working in a commercial facility. When a customer enters the store, the camera on the smart device captures facial data. This facial data is transmitted to a server via the network.

[1191] Facial recognition and data matching

[1192] The server compares the transmitted facial data with a facial recognition database to retrieve matching person information. Furthermore, it extracts the customer's purchase history and special request information from a purchase history database. The retrieved information is immediately sent to the smart device.

[1193] Display and notification of customer information

[1194] The smart device displays personal information transmitted from the server. For example, it might show information such as "Hanako Yamada, Purchase History: Item A, Item B, Special Request: Express Delivery." This allows store staff to provide personalized service quickly. Voice notifications are also available depending on the settings.

[1195] Specific example

[1196] When creating an example prompt for a system in a meeting facility where a store employee wearing smart glasses takes a picture of a newly entering customer's face with a camera and sends the data to a server. The server then compares it with a facial recognition database and displays the person's information on a screen, the input should be: "Prompt: Write Python code to capture customer face data and display that information."

[1197] This invention makes it possible to improve customer service within specific commercial facilities and increase customer satisfaction. This allows for the provision of efficient and prompt service, contributing to the promotion of commercial activities.

[1198] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1199] Step 1:

[1200] A user wears a smart device and waits in a commercial facility. The smart device's camera continuously captures images of people in the surrounding area. The input is the smart device's camera feed, and the output is captured facial data. This facial data is temporarily stored on the smart device.

[1201] Step 2:

[1202] A smart device sends captured facial data to a server over the network. The input is the captured facial data, and the output is a message indicating that the facial data was successfully sent to the server. This data transmission is performed using an HTTP POST request.

[1203] Step 3:

[1204] The server compares the received facial data with a facial recognition database. The input is the newly received facial data, and the output is the matching result. A facial recognition algorithm is used in the matching process. The matching result is either matching person information (if any) or an error message if there is no match.

[1205] Step 4:

[1206] If the server finds a match in the facial recognition database, it retrieves the relevant purchase history and special request information from the purchase history database. The input is the ID of the matched person, and the output is the purchase history and request information of that person. This data is retrieved using SQL queries or similar methods.

[1207] Step 5:

[1208] The server sends customer information (purchase history and special requests) to the smart device. The input is the customer's personal information, and the output is a message indicating successful data transmission to the smart device. This transmission is performed using an HTTP GET request.

[1209] Step 6:

[1210] A smart device receives customer information transmitted from a server and displays it on its screen or provides an audio notification. The input is customer information received from the server, and the output is the information displayed on the screen or an audio notification. This process uses both a display function and a text-to-speech (TTS) function.

[1211] These steps enable the present invention to quickly and efficiently acquire customer information within a specific commercial facility and to provide personalized service by store staff.

[1212] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1213] This invention is a system that links facial recognition data and business card data, and uses smart glasses and an emotion engine within a specific facility to display or output information and emotional state of the person concerned.

[1214] Facial data capture and registration

[1215] The user stands in front of a sensor-equipped terminal at the facility entrance, and their face is captured by the camera. The terminal sends the captured facial data to a server. The server stores this facial data in a facial recognition database. The user also uploads business card data and affiliation lists to the terminal and sends them to the server. The server stores this information in a business card database.

[1216] Authentication within the facility

[1217] When a user enters the facility, the terminal works in conjunction with the facility's access control system to perform facial recognition. Once authentication is complete, the terminal authorizes the user to use smart glasses and performs the necessary settings.

[1218] Facial recognition and data linkage

[1219] As a user moves around the facility wearing smart glasses, the smart glasses (device) continuously capture the faces of people around them using its camera. The captured face data is transmitted to a server in real time. The server compares the data with a facial recognition database and retrieves matching person information from a business card database.

[1220] Analysis of the Emotion Engine

[1221] The emotion engine analyzes facial data captured by the device to identify the user's emotional state. This emotional data is sent to a server and processed along with the person's information.

[1222] Display of Information and Emotion

[1223] The server sends the acquired person information and emotion data from the emotion engine to the device. The device (smart glasses) displays this information on the lenses. It can also provide voice notifications through the speaker.

[1224] Specific example

[1225] 1. User registration and facial data acquisition:

[1226] The user stands still at the entrance of the facility, facing the camera.

[1227] The device will notify you that "face data has been captured" and send the face data to the server.

[1228] The server displays a confirmation message stating, "Face data saving complete," and prompts the user to upload business card data.

[1229] The user uploads business card data, and the device notifies them that "business card data has been sent to the server."

[1230] 2. Authentication within the facility:

[1231] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[1232] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[1233] 3. Facial recognition and data linkage:

[1234] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[1235] The captured facial data is sent to the server.

[1236] The server displays "Matching against facial recognition database" and retrieves matching person information.

[1237] 4. Analysis of the Emotion Engine:

[1238] The emotion engine analyzes the facial data and sends a message to the server saying, "Emotion data has been analyzed."

[1239] The server processes the emotional data as additional information and integrates it with the person's information.

[1240] 5. Displaying information and emotions:

[1241] The server sends the acquired person information and emotional data to the terminal.

[1242] The smart glasses display this information on the lenses. In addition to "Taro Tanaka - Sales Department," information such as "Current emotion: Joy" is displayed. A voice notification also says, "Taro Tanaka, Sales Department. Your current emotion is joy."

[1243] This system allows users to quickly obtain information about others within a specific facility, as well as understand their emotional state. This enables more appropriate communication and is expected to improve work efficiency. Furthermore, by limiting the use of the system to specific facilities, the risk of information leakage can be minimized.

[1244] The following describes the processing flow.

[1245] Step 1:

[1246] The user stands in front of a sensor-equipped terminal at the facility entrance, facing the camera directly, and remains still for approximately 5 seconds.

[1247] Step 2:

[1248] The device captures the user's face. The captured facial data is saved in real time.

[1249] Step 3:

[1250] The device sends the captured facial data to the server. Once the transmission is complete, the device notifies the user that "facial data has been sent to the server."

[1251] Step 4:

[1252] The server saves the received facial data to the facial recognition database. It then sends a message stating, "Facial data saving complete."

[1253] Step 5:

[1254] Users upload business card data and affiliation lists to their devices. The upload is performed through a specified input format.

[1255] Step 6:

[1256] The device sends the uploaded business card data to the server. Once the transmission is complete, a notification will appear stating, "Business card data has been sent to the server."

[1257] Step 7:

[1258] The server saves the submitted business card data to the business card database. It then sends a notification stating, "Business card data saving complete."

[1259] Step 8:

[1260] The user stands in front of a terminal for facial recognition at the facility entrance. The terminal works in conjunction with the facility's access control system to perform facial recognition.

[1261] Step 9:

[1262] The device will notify you with the message, "Authentication successful. Please enter," once facial recognition is complete.

[1263] Step 10:

[1264] The user puts on smart glasses and begins moving around the facility. The smart glasses (device) continuously capture the faces of people in the surrounding area using its camera.

[1265] Step 11:

[1266] The device sends the captured facial data to the server in real time. The status "Sending facial data" is displayed.

[1267] Step 12:

[1268] The server compares the received facial data with the facial recognition database and displays the message "Matching...".

[1269] Step 13:

[1270] The server retrieves the person's information linked to the matched facial data from the business card database. A notification is sent stating, "Person's information has been retrieved."

[1271] Step 14:

[1272] The device sends the captured facial data to the emotion engine. A message appears stating, "Analyzing emotion data."

[1273] Step 15:

[1274] The emotion engine analyzes the received facial data to identify the user's emotional state. A notification is then displayed stating, "Emotional data has been analyzed."

[1275] Step 16:

[1276] The emotion engine sends the analyzed emotion data to the server. The status "Sending emotion data" is displayed.

[1277] Step 17:

[1278] The server sends the acquired person information and emotion data to the terminal.

[1279] Step 18:

[1280] The device (smart glasses) displays the transmitted person information and emotion data on the lenses. In addition to "Taro Tanaka - Sales Department," information such as "Current emotion: Joy" is displayed.

[1281] Step 19:

[1282] The device (smart glasses) uses its voice output function to announce, "This is Taro Tanaka from the Sales Department. Your current emotion is joy."

[1283] Thus, in addition to integrating facial recognition and a business card database, this system utilizes an emotion engine to provide even deeper information. This allows users to understand not only the information but also the emotional state of others within a specific facility, thereby improving work efficiency and the quality of communication.

[1284] (Example 2)

[1285] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1286] Conventional facial recognition systems can obtain basic information about a person, but they struggle to understand that person's emotional state. Therefore, especially in environments where person-to-person communication is crucial, there is a need for a means to quickly grasp a person's emotional state. Furthermore, there is a need for a method to effectively perform facial recognition and emotion analysis within specific facilities while minimizing the risk of information leakage.

[1287] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1288] In this invention, the server includes means for transmitting facial data to the server, means for comparing it with a facial recognition database, and means for analyzing the facial data to identify emotional states. This makes it possible to understand not only facial recognition but also the emotional states of individuals in real time within a specific facility.

[1289] "Facial data" refers to image data of a person's face captured by a camera.

[1290] A "server" refers to a computer system that stores, analyzes, and manages facial and emotional data.

[1291] A "facial recognition database" is a database stored on a server that uses facial data to authenticate individuals.

[1292] "Personal information" refers to basic information and attribute data about a person, obtained from sources such as business card databases.

[1293] A "terminal" is a hardware device used by a user to perform input and output, including smart glasses and sensor-equipped devices.

[1294] "Smart glasses" are wearable devices that can capture facial data and display information and output audio.

[1295] An "emotion engine" is a combination of software and hardware that analyzes captured facial data to identify emotional states.

[1296] "Emotional data" refers to data that indicates a person's emotional state, as identified by the emotion engine.

[1297] "Specific facilities" refers to physical locations where the use of the system is restricted to minimize the risk of information leakage.

[1298] This invention relates to a system that links facial recognition data and business card data to display or output information and emotional state of a person using smart glasses and an emotion engine within a specific facility. In this system, "facial data" is captured, sent to a server for analysis, and then person information and emotional state are identified.

[1299] Hardware and software to be used

[1300] The following hardware will be used as the terminal:

[1301] Cameras with sensors (installed at facility entrances and on smart glasses)

[1302] Smart glasses (display information on the lenses, with audio output capability)

[1303] Access control system (manages entry and exit to facilities)

[1304] Servers and related software:

[1305] Facial recognition database (a database for storing and matching facial data)

[1306] Business card database (a database for storing personal information)

[1307] Emotion engine (software that analyzes facial data to identify emotional states)

[1308] Information display and voice notification engine (software for displaying information on smart glasses and providing voice notifications)

[1309] Explanation of natural language processing

[1310] 1. Capture and register facial data

[1311] The user stands in front of a sensor-equipped camera at the facility entrance and their face data is captured. The terminal encodes this face data and sends it to the server. The server decodes the face data, compares it with existing data, and stores it in a face recognition database. Next, the server prompts the user to upload business card data, and once the user uploads the business card data, it is stored on the server.

[1312] 2. Authentication within the facility

[1313] When a user enters a facility, the device performs facial recognition and sends the data to the server. The server compares the data with the facial recognition database, and if authentication is successful, the device authorizes the use of the smart glasses. Once the setup is complete, the smart glasses become available for use.

[1314] 3. Facial recognition and data linkage

[1315] As a user moves around the facility wearing smart glasses, the smart glasses capture the faces of people around them in real time. The smart glasses encode the captured face data and send it to a server. The server compares it with a facial recognition database and retrieves matching person information from a business card database, then sends it to the smart glasses.

[1316] 4. Analysis of the Emotional Engine

[1317] The emotion engine analyzes facial data captured by the smart glasses and identifies the emotional state. The identified emotion data is encoded and sent to a server. The server integrates this emotion data with the person's information and sends it back to the smart glasses.

[1318] 5. Display of Information and Emotion

[1319] The smart glasses receive person information and emotion data transmitted from the server and display it on the lenses. They also have a voice notification function.

[1320] Specific example

[1321] 1. User registration and facial data acquisition:

[1322] The user stands still at the entrance of the facility, facing the camera.

[1323] The device will notify you that "face data has been captured" and send the face data to the server.

[1324] The server displays a confirmation message stating, "Face data saving complete," and prompts the user to upload business card data.

[1325] The user uploads business card data, and the device notifies them that "business card data has been sent to the server."

[1326] 2. Authentication within the facility:

[1327] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[1328] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[1329] 3. Facial recognition and data linkage:

[1330] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[1331] The captured facial data is sent to the server.

[1332] The server displays "Matching against facial recognition database" and retrieves information on the matching person.

[1333] 4. Analysis of the Emotion Engine:

[1334] The emotion engine analyzes the facial data and sends a message to the server saying, "Emotion data has been analyzed."

[1335] The server processes the emotional data as additional information and integrates it with the person's information.

[1336] 5. Displaying information and emotions:

[1337] The server sends the acquired person information and emotional data to the terminal.

[1338] The smart glasses display this information on the lenses. In addition to "Employee A - Sales Department," information such as "Current emotion: Joy" is displayed. A voice notification also says, "Employee A, Sales Department. Your current emotion is joy."

[1339] Example of a prompt

[1340] "Please capture facial data at the facility entrance and upload your business card data."

[1341] "Perform real-time facial recognition and emotion analysis while moving around the facility, and display the results on smart glasses."

[1342] "Link facial recognition data with business card data, use an emotion engine to analyze emotional states, and provide notifications via display and audio."

[1343] This system allows users to quickly obtain information about others within a specific facility, as well as understand their emotional state. This enables more appropriate communication and is expected to improve work efficiency. Furthermore, by limiting the use of the system to specific facilities, the risk of information leakage can be minimized.

[1344] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1345] Step 1:

[1346] The user stands in front of a sensor-equipped camera at the facility entrance. The device captures facial data, encodes it, and then sends it to the server.

[1347] [Input] User's face image

[1348] [Data Processing] The device encodes the facial image.

[1349] [Output] Encoded face data is sent to the server.

[1350] Specifically, the camera recognizes the contours of the face, and face capture is performed when the camera remains still for a certain period of time.

[1351] Step 2:

[1352] The server decodes the received facial data and compares it with the facial recognition database. After confirming a match with existing data, it registers it as new data.

[1353] [Input] Encoded facial data sent from the device

[1354] [Data Processing] The server decodes the facial data and performs matching calculations.

[1355] [Output] The new face data is saved to the facial recognition database, and a registration completion message is sent to the device.

[1356] Specifically, the server first decomposes the facial data into multiple feature points and compares them with existing data in the database using a high-speed matching algorithm.

[1357] Step 3:

[1358] The terminal displays a registration completion message from the server and prompts the user to upload business card data. The user uploads the business card data to the terminal, and the terminal sends it to the server.

[1359] [Input] Registration completion message from the server, user's business card data

[1360] [Data Processing] The terminal encodes the business card data.

[1361] [Output] Business card data is sent to the server.

[1362] Specifically, the device displays "Please upload your business card data," and the user uploads a business card image to the device.

[1363] Step 4:

[1364] The server receives business card data, decodes it, and saves it to the business card database. It also notifies the terminal that the business card data has been successfully received.

[1365] [Input] Encoded business card data sent from the terminal

[1366] [Data Processing] The server decodes and saves the business card data.

[1367] [Output] The business card data is saved to the business card database, and a message confirming receipt is sent to the terminal.

[1368] Specifically, the server analyzes business card data using OCR (optical character recognition) technology, extracts important information, and stores it in a database.

[1369] Step 5:

[1370] When a user enters the facility, the terminal performs facial recognition again and sends the data to the server. The server compares the data with the facial recognition database and sends the authentication result back to the terminal.

[1371] [Input] User's face image

[1372] [Data Processing] The terminal encodes the facial image and sends it to the server. The server decodes the facial data and performs matching calculations.

[1373] [Output] Authentication result is sent to the terminal.

[1374] Specifically, the device will display "Authentication in progress," and upon successful authentication, it will notify the user with "Authentication successful."

[1375] Step 6:

[1376] After authentication, the device will authorize the use of the smart glasses and automatically perform pairing and necessary settings.

[1377] [Input] Authentication result from the server

[1378] [Data Processing] The device resets the smart glasses settings and performs pairing.

[1379] [Output] Smart glasses become ready for use.

[1380] Specifically, the device connects to the smart glasses using Bluetooth and performs the initial setup.

[1381] Step 7:

[1382] As the user moves around the facility, the smart glasses continuously capture the faces of people around them in real time and send the facial data to a server.

[1383] [Input] Captured facial data

[1384] [Data Processing] The smart glasses encode the facial data and send it to the server.

[1385] [Output] Encoded face data is sent to the server.

[1386] Specifically, the smart glasses' camera tracks the face in real time and automatically sends the data to a server.

[1387] Step 8:

[1388] The server matches the received facial data, retrieves matching person information from the business card database, and sends it to the smart glasses.

[1389] [Input] Face data transmitted from smart glasses

[1390] [Data Processing] The server matches facial data and retrieves person information.

[1391] [Output] Person information is sent to smart glasses.

[1392] Specifically, the server cross-references the facial recognition database and the business card database to quickly retrieve data.

[1393] Step 9:

[1394] The emotion engine analyzes facial data captured by the smart glasses to identify the emotional state. The identified emotion data is then sent to the server.

[1395] [Input] Captured facial data

[1396] [Data Processing] The emotion engine analyzes facial data and identifies emotional states.

[1397] [Output] Emotional state data is sent to the server.

[1398] Specifically, the emotion engine analyzes facial movements and changes in expression to identify emotional states in real time.

[1399] Step 10:

[1400] The server integrates emotional data with person information and sends it to the smart glasses. The smart glasses display the information on the lenses and provide audio notifications as needed.

[1401] [Input] Emotional state data, person information

[1402] [Data Processing] The server integrates the data and formats it for smart glasses.

[1403] [Output] Display and audio notifications are provided via smart glasses.

[1404] In terms of specific actions, the smart glasses display "Employee A - Sales Department" and "Current Emotion: Joyful" on the lens, and announce via voice, "Employee A, Sales Department. Your current emotion is joyful."

[1405] (Application Example 2)

[1406] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1407] Many existing facial recognition systems only perform person recognition and are insufficient for providing advanced customer service. Especially in physical stores, it is crucial to quickly understand customers' emotional states and provide individualized support, but current systems primarily rely on abstract data and manual information acquisition, lacking practicality. Furthermore, without linking customer information with emotional data, appropriate service recommendations and highly accurate customer service are difficult. This project aims to solve these problems and improve the customer experience and provide efficient service delivery.

[1408] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1409] In this invention, the server includes means for transmitting facial data to the server, means for comparing it with a facial recognition database stored on the server, means for acquiring person information, means for transmitting the acquired person information and emotion data to a terminal, means for displaying or outputting audio to the terminal, means for analyzing facial data with an emotion engine and identifying the emotional state, and means for performing the above operations only within a specific facility. This makes it possible to automatically link customer facial recognition data and business card information to understand the customer's emotional state. As a result, store employees can acquire customer information in real time and provide individually personalized customer service.

[1410] "Facial data" refers to image data of a person's face captured by a camera or other imaging device.

[1411] A "server" is a computer system that manages, stores, and processes data via a network.

[1412] A "facial recognition database" is a database that stores multiple pre-registered facial data and compares them with newly acquired facial data.

[1413] "Personal information" refers to information about individual individuals (e.g., name and affiliation) obtained from sources such as business card databases.

[1414] A "terminal" is a device that a user directly operates and that displays or notifies information via sound. In this invention, smart glasses are the primary example of this type of device.

[1415] An "emotion engine" is software or hardware that analyzes facial data to identify a person's emotional state (e.g., joy, anger, sadness, etc.).

[1416] "Within a specific facility" refers to a location to which the present invention is applied in a limited manner, and means the area to which facial recognition and emotion analysis are to be performed.

[1417] The system for carrying out this invention includes functions for capturing facial data, matching it with a database, analyzing emotions using an emotion engine, displaying the acquired information in real time, and providing voice notifications. The specific configuration and operation details of this system are described below.

[1418] System Configuration

[1419] The user puts on smart glasses.

[1420] The device captures facial data of people in its surroundings using a camera mounted on smart glasses and sends this data to a server.

[1421] The server compares the facial data with a facial recognition database to obtain the person's information.

[1422] The server uses an emotion engine to analyze the captured facial data and identify emotional data.

[1423] The server sends the acquired person information and emotion data to the terminal.

[1424] The device displays this information on the smart glasses' screen and provides voice notifications as needed.

[1425] Examples of specific hardware and software to be used

[1426] Hardware:

[1427] Smart glasses: Capture facial data and display information.

[1428] Server: Performs data processing and manages databases.

[1429] software:

[1430] OpenCV: Image processing library for face recognition

[1431] Requests module: A library for communicating with the server.

[1432] Emotion engine: Software that analyzes emotional states from facial data; a specific example is Microsoft Azure's Emotion API.

[1433] System operation example

[1434] Facial data capture and registration

[1435] The user captures their face with a camera through smart glasses at the facility entrance, and the device notifies them that "face data has been captured." This face data is sent to a server and stored in a face recognition database. Next, the user uploads business card data, and the device notifies them that "business card data has been sent to the server." This business card data is stored in a business card database.

[1436] Authentication and information display within the facility

[1437] As a user moves around a facility wearing smart glasses, the cameras on the smart glasses continuously capture the faces of people in their surroundings. The captured facial data is sent to a server in real time and compared with a facial recognition database. Matching person information is retrieved from a business card database and sent to the device. Furthermore, an emotion engine analyzes the emotional state, and the results are also sent from the server to the device.

[1438] For example, when a user enters a meeting room, the smart glasses display shows "Taro Yamada - Gold Member, Current Emotion: Joyful," along with an audio notification. This information allows the user to respond quickly and appropriately.

[1439] Example of a prompt

[1440] If you want to ask the AI ​​model that generates a specific and immediately usable action plan for this application, use the following prompt:

[1441] "You are developing a system that uses smart glasses to support customer service in physical stores by utilizing facial recognition data and emotion engine analysis results. Please provide recommendations for actions that will allow for more effective customer service based on real-time customer information."

[1442] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1443] Step 1:

[1444] The user puts on smart glasses and faces the camera at the entrance of the facility.

[1445] Input: Capture a face at the facility entrance.

[1446] Data processing: The smart glasses' camera acquires image data of the face.

[1447] Output: Captured facial data is sent to the terminal.

[1448] Step 2:

[1449] The device sends the captured facial data to the server.

[1450] Input: Facial image data acquired by the smart glasses' camera.

[1451] Data processing: Convert the encoded facial image data into a transmission format.

[1452] Output: Face data is sent to the server.

[1453] Step 3:

[1454] The server compares the facial data with a facial recognition database.

[1455] Input: Image data of a face sent from the device.

[1456] Data processing: Matching is performed using a facial recognition algorithm (e.g., a deep learning model).

[1457] Output: Matching person information is retrieved.

[1458] Step 4:

[1459] The server retrieves matching person information from the business card database.

[1460] Input: Person information matching the facial recognition database.

[1461] Data processing: Extract necessary personal information (name, affiliation, etc.) from the business card database.

[1462] Output: Person information is saved on the server.

[1463] Step 5:

[1464] The server uses an emotion engine to analyze facial data and identify emotional data.

[1465] Input: Image data of a face sent from the device.

[1466] Data processing: Using emotion analysis algorithms, we analyze emotional states such as joy, anger, and sadness.

[1467] Output: Emotional data is saved to the server.

[1468] Step 6:

[1469] The server sends the acquired person information and emotional data to the terminal.

[1470] Input: Person information obtained from a business card database and emotional data analyzed by an emotion engine.

[1471] Data processing: Integrate this data and convert it into a transmission format.

[1472] Output: Person information and emotion data are sent to the terminal.

[1473] Step 7:

[1474] The device displays the person information and emotion data it receives on the smart glasses' display.

[1475] Input: Person information and emotion data sent from the server.

[1476] Data processing: Converts received data into a display format.

[1477] Output: The smart glasses display shows "○○ ○○ - Affiliation: △△, Current emotion: Joy." Voice notifications are also provided as needed.

[1478] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1479] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1480] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1481] [Fourth Embodiment]

[1482] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1483] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1484] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1485] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1486] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1487] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1488] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1489] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1490] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1491] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1492] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1493] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1494] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1495] This invention is a system that links facial recognition data and business card data to display or output information about a person using smart glasses within a specific facility.

[1496] Facial data capture and registration

[1497] The user stands in front of a sensor-equipped terminal at the facility entrance, and their face is captured by the camera. The terminal sends the captured facial data to a server. The server stores this facial data in a facial recognition database. The user also uploads business card data and affiliation lists to the terminal and sends them to the server. The server stores this information in a business card database.

[1498] Authentication within the facility

[1499] When a user enters the facility, the terminal works in conjunction with the facility's access control system to perform facial recognition. Once authentication is complete, the terminal authorizes the user to use smart glasses and performs the necessary settings.

[1500] Facial recognition and data linkage

[1501] When a user wears smart glasses and moves around the facility, the smart glasses (device) capture the faces of people around them using its camera. The captured face data is sent to a server in real time. The server compares the data with a facial recognition database and retrieves matching person information from a business card database.

[1502] Displaying information

[1503] The server sends the acquired person information to the device. The device (smart glasses) displays this person information on the lens or notifies the user via voice through the speaker.

[1504] Specific example

[1505] 1. User registration and facial data acquisition:

[1506] The user stands still at the entrance of the facility, facing the camera.

[1507] The device will notify you that "face data has been captured" and send the face data to the server.

[1508] The server displays a confirmation message stating, "Face data saving complete."

[1509] 2. Authentication within the facility:

[1510] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[1511] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[1512] 3. Facial recognition and data linkage:

[1513] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[1514] The captured facial data is sent to the server.

[1515] The server displays "Matching against facial recognition database" and retrieves matching person information.

[1516] 4. Display of information:

[1517] The server sends the retrieved person information (for example, "Taro Tanaka - Sales Department") to the terminal.

[1518] Smart glasses display this information on the lenses. For example, "Taro Tanaka - Sales Department" is visually shown. It can also provide voice notifications such as "Taro Tanaka, Sales Department."

[1519] This system allows users to quickly obtain and verify information about others within a specific facility. Furthermore, by limiting the system's use to specific facilities, the risk of information leakage can be minimized.

[1520] The following describes the processing flow.

[1521] Step 1:

[1522] The user stands in front of a sensor-equipped terminal at the facility entrance, facing the camera directly, and remains still for approximately 5 seconds.

[1523] Step 2:

[1524] The device captures the user's face. The captured facial data is saved to the device in real time.

[1525] Step 3:

[1526] The device sends the captured facial data to the server. Once the transmission is complete, the device notifies the user that "facial data has been sent to the server."

[1527] Step 4:

[1528] The server saves the received facial data to the facial recognition database. Once saving is complete, the server sends a message stating, "Facial data saving complete."

[1529] Step 5:

[1530] Users upload business card data and affiliation lists to their devices. This is done through a specified input format.

[1531] Step 6:

[1532] The device sends the uploaded business card data to the server. Once the transmission is complete, a notification will appear stating, "Business card data has been sent to the server."

[1533] Step 7:

[1534] The server saves the submitted business card data to the business card database. Once saving is complete, the server notifies the user that "Business card data has been saved."

[1535] Step 8:

[1536] The user stands in front of a terminal for facial recognition at the facility entrance. The terminal is linked to the facility's access control system and authenticates the user's face.

[1537] Step 9:

[1538] The device displays a notification saying, "Authentication successful. Please enter," once facial recognition is complete.

[1539] Step 10:

[1540] The user puts on smart glasses and begins moving around the facility. The smart glasses (device) continuously capture the faces of people in the surrounding area using its camera.

[1541] Step 11:

[1542] The device sends the captured facial data to the server in real time. The status "Sending facial data" is displayed.

[1543] Step 12:

[1544] The server compares the received face data with the face recognition database and finds a matching face. It then outputs the message "Matching..." as the processing status.

[1545] Step 13:

[1546] The server retrieves person information linked to matching facial data from the business card database. After retrieval, a notification is sent stating, "Person information has been retrieved."

[1547] Step 14:

[1548] The server sends the acquired person information to the terminal. The status "Sending person information" is displayed.

[1549] Step 15:

[1550] The device (smart glasses) displays the transmitted person information on the lens. It also provides voice notifications via the speaker, such as "This is person AA," if necessary.

[1551] (Example 1)

[1552] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1553] Conventional facial recognition systems require manual recognition and data acquisition, making it difficult to quickly and efficiently obtain information about individuals within a specific facility. Furthermore, the lack of restrictions on system usage increases the risk of data leaks. Additionally, insufficient real-time information sharing and the incomplete integration of current facial recognition data with business card data make the system impractical.

[1554] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1555] In this invention, the server includes means for capturing facial data, means for transmitting facial data to the server, means for comparing it with a facial recognition database stored on the server, means for obtaining person information from a linked database, means for transmitting the obtained person information to a terminal, means for displaying or outputting audio on the terminal, and means for performing the above operations using smart glasses within a specific facility. This makes it possible to quickly and efficiently obtain and verify the information of a person within a specific facility, minimizing the risk of information leakage. Furthermore, real-time information sharing is realized, the linkage between facial recognition data and business card data is enhanced, and practicality is improved.

[1556] "Facial data" refers to digital data that includes images of an individual's face and its characteristics, captured by a camera.

[1557] A "server" is a computer system that stores, manages, and processes data on a network.

[1558] A "facial recognition database" is a dedicated database used to store captured facial data and perform facial recognition based on that data.

[1559] A "terminal" refers to a hardware device operated by the user, and this includes smart glasses and computing devices.

[1560] A "business card database" is a database that stores business card information and related data, and uses this information to provide personal information.

[1561] "Smart glasses" are wearable devices worn on the head that are equipped with cameras and displays and have the function of displaying and capturing information.

[1562] "Within a specific facility" refers to designated areas or the interior of buildings where the use of the system is restricted.

[1563] This invention relates to a system that links facial recognition data with personal information and displays or outputs information about another person using smart glasses within a specific facility. This system allows users to quickly obtain and verify information about others within a designated facility. This minimizes the risk of information leakage and improves practicality.

[1564] Hardware to use

[1565] 1. Facial data capture device: A camera installed in smart glasses (e.g., Microsoft HoloLens, Google Glass).

[1566] 2. Terminal: Smart glasses or a computer device operated by the user.

[1567] 3. Server: A computer system for managing and processing databases (e.g., Linux server, AWS server).

[1568] Software to use

[1569] 1. Facial recognition algorithms: Facial recognition APIs such as OpenCV and Amazon Rekognition.

[1570] 2. Database management systems: such as MySQL and PostgreSQL.

[1571] 3. Communication Protocols: Protocols for real-time communication such as HTTP, WebSocket, and MQTT.

[1572] Specific details of data processing and data calculations

[1573] 1. User registration and facial data acquisition:

[1574] The user stands still at the entrance of the facility, facing the camera.

[1575] The device captures facial data from the camera image and notifies the user that "facial data has been captured."

[1576] The device transmits facial data to the server in real time.

[1577] The server saves the facial data to a MySQL database and notifies the terminal that "Facial data saving is complete."

[1578] 2. Authentication within the facility:

[1579] As the user passes through the facility entrance, they turn their face towards the camera again as they walk.

[1580] The device activates the facial recognition system, sends facial data to the server, and performs authentication.

[1581] The server checks whether the face data matches the data stored in the database and displays "Authentication successful" on the screen.

[1582] Once authentication is successful, the device communicates with the smart glasses and automatically performs the necessary settings (for example, Wi-Fi connection and user profile synchronization).

[1583] 3. Facial recognition and data linkage:

[1584] As the user wears the smart glasses and moves around the facility, the smart glasses' camera captures the faces of people in the surrounding area.

[1585] The captured facial data is transmitted from the smart glasses to the server via a high-speed network.

[1586] The server performs real-time matching against the facial recognition database and retrieves matching person information from the business card database.

[1587] During the matching process, the server displays the current processing progress on the smart glasses.

[1588] 4. Display of information:

[1589] The server sends the acquired person information to the smart glasses using the WebSocket protocol.

[1590] Smart glasses visually display information such as "Taro Tanaka - Sales Department" on the lenses.

[1591] The smart glasses' speaker notifies the user via voice, "Taro Tanaka, Sales Department."

[1592] Users receive visual and audible notifications, allowing them to instantly understand information about people around them.

[1593] Specific example

[1594] 1. User registration and facial data acquisition:

[1595] The user stands still at the entrance of the facility, facing the camera.

[1596] The device will notify you that "face data has been captured" and send the face data to the server.

[1597] The server displays a confirmation message stating, "Face data saving complete."

[1598] 2. Authentication within the facility:

[1599] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[1600] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[1601] 3. Facial recognition and data linkage:

[1602] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[1603] The captured facial data is sent to the server.

[1604] The server displays "Matching against facial recognition database" and retrieves matching person information.

[1605] 4. Display of information:

[1606] The server sends the retrieved person information (for example, "Taro Tanaka - Sales Department") to the terminal.

[1607] Smart glasses display this information on the lenses. For example, "Taro Tanaka - Sales Department" is visually shown.

[1608] It is also possible to notify users via voice message saying, "Taro Tanaka, Sales Department."

[1609] Example of a prompt

[1610] "Please describe a system that captures a user's face at the facility entrance, registers it in a database, and uses smart glasses to display or output information about other users within the facility in real time."

[1611] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1612] Step 1:

[1613] Capture and transmit facial data

[1614] The user stands still at the entrance of the facility, facing the camera. The device captures the face and sends the data to the server. Specifically, the device detects the face from the camera image, performs image processing, and generates face data. This face data is sent to the server in real time via the network. The server stores the received face data in a face recognition database.

[1615] Input: User's facial image (input from camera)

[1616] Data processing: Generating face data through image processing.

[1617] Output: Captured facial data (sent from the device to the server)

[1618] Specific actions:

[1619] The user stands in the designated location, faces the camera, and remains still for a few seconds.

[1620] The device's camera captures facial images, and internal processing generates facial data.

[1621] The device sends the generated facial data to the server.

[1622] The server stores the facial data in a database.

[1623] Step 2:

[1624] Authentication within the facility

[1625] When a user enters a facility, the terminal performs facial recognition. The terminal sends facial data to a server for authentication. The server compares the facial data with the data stored in the database to confirm a match. If authentication is successful, the terminal authorizes the use of smart glasses and performs the necessary settings.

[1626] Input: Entered facial data (captured from the device's camera)

[1627] Data processing: Matching using facial recognition algorithms

[1628] Output: Authentication result (success / failure)

[1629] Specific actions:

[1630] The user turns their face towards the camera as they pass through the facility entrance.

[1631] The device captures the facial data again and sends it to the server.

[1632] The server compares the face data stored in the database and returns the authentication result to the terminal.

[1633] If authentication is successful, the device will display permission to use the smart glasses on the screen and automatically configure the settings.

[1634] Step 3:

[1635] Facial recognition and data linkage

[1636] When a user wears smart glasses and moves around the facility, the smart glasses' camera captures the faces of people in the surrounding area. The captured facial data is sent to a server in real time. The server compares it with a facial recognition database and retrieves matching person information from a business card database.

[1637] Input: Images of people's faces in the surrounding area (input from the smart glasses' camera)

[1638] Data processing: Facial recognition and database matching

[1639] Output: Matching person information

[1640] Specific actions:

[1641] Users wear smart glasses and walk around the facility.

[1642] The smart glasses' camera continuously captures the faces of people in the surrounding area.

[1643] The captured facial data is transmitted from the smart glasses to the server via a high-speed network.

[1644] The server performs real-time matching against the facial recognition database and retrieves matching person information from the business card database.

[1645] Step 4:

[1646] Displaying information

[1647] The server transmits the acquired person information to the device (smart glasses). The device (smart glasses) either displays this information on the lenses or notifies the user via audio through the speaker.

[1648] Input: Matching result (person information)

[1649] Data processing: Information transmission

[1650] Output: Visual display or audio notification

[1651] Specific actions:

[1652] The server sends the acquired person information to the smart glasses using the WebSocket protocol.

[1653] Smart glasses visually display information such as "Taro Tanaka - Sales Department" on the lenses.

[1654] The smart glasses' speaker notifies the user via voice, "Taro Tanaka, Sales Department."

[1655] Users receive visual and audible notifications, allowing them to instantly understand information about people around them.

[1656] (Application Example 1)

[1657] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1658] In conventional systems, when acquiring personal information through facial recognition within a specific facility, the information obtained was limited, making it difficult to instantly retrieve and display detailed customer purchase history or special requests in commercial facilities. Furthermore, centralized information management was difficult, and data collection from various databases was complex. Additionally, delays in providing individual services could occur, posing a risk of lowering customer satisfaction.

[1659] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1660] In this invention, the server includes means for capturing facial data, means for transmitting the facial data to the server, means for matching and retrieving person information stored in a facial recognition database, and means for transmitting customer information, including purchase history and special requests, to a terminal. This enables store employees in commercial facilities to instantly obtain detailed information about visiting customers using smart devices and provide personalized services.

[1661] "Facial data" refers to digital image data that captures a person's face as a key feature.

[1662] A "server" is a computer system that sends, receives, and stores data over a network.

[1663] A "facial recognition database" is a database system that stores captured facial data for later comparison.

[1664] "Personal information" refers to information related to an individual obtained by matching it with a facial recognition database.

[1665] A "terminal" is a device that receives personal information and displays or outputs it to the user as audio.

[1666] "Within a specific commercial facility" refers to a specific physical area where commercial activities take place.

[1667] A "smart device" is an electronic device with advanced functions, such as smart glasses or a smartphone.

[1668] "Purchase history" refers to a record of products and services that a customer has purchased in the past.

[1669] A "special request" refers to a customer's specific request regarding a particular product or service.

[1670] A "purchase history database" is a database system used to store and manage a customer's purchase history.

[1671] This invention relates to a system that recognizes customer facial data within a specific commercial facility, allowing store staff to instantly retrieve and display associated purchase history and special requests. To implement this invention, a server, smart devices, a facial recognition database, a purchase history database, and the like are used.

[1672] System Configuration

[1673] The server stores facial data and matches it against a facial recognition database. It also incorporates a purchase history database that stores and manages purchase history and special requests. This database is used to understand and provide detailed customer information immediately.

[1674] A smart device (e.g., smart glasses) is equipped with a camera that captures the customer's face and a display that shows the information. The smart device has the function of capturing facial data and transmitting that data to a server in real time. It also displays or notifies the customer of the information transmitted from the server via sound.

[1675] Face data capture and transmission

[1676] The user (in this case, a store employee) wears a smart device while working in a commercial facility. When a customer enters the store, the camera on the smart device captures facial data. This facial data is transmitted to a server via the network.

[1677] Facial recognition and data matching

[1678] The server compares the transmitted facial data with a facial recognition database to retrieve matching person information. Furthermore, it extracts the customer's purchase history and special request information from a purchase history database. The retrieved information is immediately sent to the smart device.

[1679] Display and notification of customer information

[1680] The smart device displays personal information transmitted from the server. For example, it might show information such as "Hanako Yamada, Purchase History: Item A, Item B, Special Request: Express Delivery." This allows store staff to provide personalized service quickly. Voice notifications are also available depending on the settings.

[1681] Specific example

[1682] When creating an example prompt for a system in a meeting facility where a store employee wearing smart glasses takes a picture of a newly entering customer's face with a camera and sends the data to a server. The server then compares it with a facial recognition database and displays the person's information on a screen, the input should be: "Prompt: Write Python code to capture customer face data and display that information."

[1683] This invention makes it possible to improve customer service within specific commercial facilities and increase customer satisfaction. This allows for the provision of efficient and prompt service, contributing to the promotion of commercial activities.

[1684] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1685] Step 1:

[1686] A user wears a smart device and waits in a commercial facility. The smart device's camera continuously captures images of people in the surrounding area. The input is the smart device's camera feed, and the output is captured facial data. This facial data is temporarily stored on the smart device.

[1687] Step 2:

[1688] A smart device sends captured facial data to a server over the network. The input is the captured facial data, and the output is a message indicating that the facial data was successfully sent to the server. This data transmission is performed using an HTTP POST request.

[1689] Step 3:

[1690] The server compares the received facial data with a facial recognition database. The input is the newly received facial data, and the output is the matching result. A facial recognition algorithm is used in the matching process. The matching result is either matching person information (if any) or an error message if there is no match.

[1691] Step 4:

[1692] If the server finds a match in the facial recognition database, it retrieves the relevant purchase history and special request information from the purchase history database. The input is the ID of the matched person, and the output is the purchase history and request information of that person. This data is retrieved using SQL queries or similar methods.

[1693] Step 5:

[1694] The server sends customer information (purchase history and special requests) to the smart device. The input is the customer's personal information, and the output is a message indicating successful data transmission to the smart device. This transmission is performed using an HTTP GET request.

[1695] Step 6:

[1696] A smart device receives customer information transmitted from a server and displays it on its screen or provides an audio notification. The input is customer information received from the server, and the output is the information displayed on the screen or an audio notification. This process uses both a display function and a text-to-speech (TTS) function.

[1697] These steps enable the present invention to quickly and efficiently acquire customer information within a specific commercial facility and to provide personalized service by store staff.

[1698] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1699] This invention is a system that links facial recognition data and business card data, and uses smart glasses and an emotion engine within a specific facility to display or output information and emotional state of the person concerned.

[1700] Facial data capture and registration

[1701] The user stands in front of a sensor-equipped terminal at the facility entrance, and their face is captured by the camera. The terminal sends the captured facial data to a server. The server stores this facial data in a facial recognition database. The user also uploads business card data and affiliation lists to the terminal and sends them to the server. The server stores this information in a business card database.

[1702] Authentication within the facility

[1703] When a user enters the facility, the terminal works in conjunction with the facility's access control system to perform facial recognition. Once authentication is complete, the terminal authorizes the user to use smart glasses and performs the necessary settings.

[1704] Facial recognition and data linkage

[1705] As a user moves around the facility wearing smart glasses, the smart glasses (device) continuously capture the faces of people around them using its camera. The captured face data is transmitted to a server in real time. The server compares the data with a facial recognition database and retrieves matching person information from a business card database.

[1706] Analysis of the Emotion Engine

[1707] The emotion engine analyzes facial data captured by the device to identify the user's emotional state. This emotional data is sent to a server and processed along with the person's information.

[1708] Display of Information and Emotion

[1709] The server sends the acquired person information and emotion data from the emotion engine to the device. The device (smart glasses) displays this information on the lenses. It can also provide voice notifications through the speaker.

[1710] Specific example

[1711] 1. User registration and facial data acquisition:

[1712] The user stands still at the entrance of the facility, facing the camera.

[1713] The device will notify you that "face data has been captured" and send the face data to the server.

[1714] The server displays a confirmation message stating, "Face data saving complete," and prompts the user to upload business card data.

[1715] The user uploads business card data, and the device notifies them that "business card data has been sent to the server."

[1716] 2. Authentication within the facility:

[1717] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[1718] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[1719] 3. Facial recognition and data linkage:

[1720] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[1721] The captured facial data is sent to the server.

[1722] The server displays "Matching against facial recognition database" and retrieves matching person information.

[1723] 4. Analysis of the Emotion Engine:

[1724] The emotion engine analyzes the facial data and sends a message to the server saying, "Emotion data has been analyzed."

[1725] The server processes the emotional data as additional information and integrates it with the person's information.

[1726] 5. Displaying information and emotions:

[1727] The server sends the acquired person information and emotional data to the terminal.

[1728] The smart glasses display this information on the lenses. In addition to "Taro Tanaka - Sales Department," information such as "Current emotion: Joy" is displayed. A voice notification also says, "Taro Tanaka, Sales Department. Your current emotion is joy."

[1729] This system allows users to quickly obtain information about others within a specific facility, as well as understand their emotional state. This enables more appropriate communication and is expected to improve work efficiency. Furthermore, by limiting the use of the system to specific facilities, the risk of information leakage can be minimized.

[1730] The following describes the processing flow.

[1731] Step 1:

[1732] The user stands in front of a sensor-equipped terminal at the facility entrance, facing the camera directly, and remains still for approximately 5 seconds.

[1733] Step 2:

[1734] The device captures the user's face. The captured facial data is saved in real time.

[1735] Step 3:

[1736] The device sends the captured facial data to the server. Once the transmission is complete, the device notifies the user that "facial data has been sent to the server."

[1737] Step 4:

[1738] The server saves the received facial data to the facial recognition database. It then sends a message stating, "Facial data saving complete."

[1739] Step 5:

[1740] Users upload business card data and affiliation lists to their devices. The upload is performed through a specified input format.

[1741] Step 6:

[1742] The device sends the uploaded business card data to the server. Once the transmission is complete, a notification will appear stating, "Business card data has been sent to the server."

[1743] Step 7:

[1744] The server saves the submitted business card data to the business card database. It then sends a notification stating, "Business card data saving complete."

[1745] Step 8:

[1746] The user stands in front of a terminal for facial recognition at the facility entrance. The terminal works in conjunction with the facility's access control system to perform facial recognition.

[1747] Step 9:

[1748] The device will notify you with the message, "Authentication successful. Please enter," once facial recognition is complete.

[1749] Step 10:

[1750] The user puts on smart glasses and begins moving around the facility. The smart glasses (device) continuously capture the faces of people in the surrounding area using its camera.

[1751] Step 11:

[1752] The device sends the captured facial data to the server in real time. The status "Sending facial data" is displayed.

[1753] Step 12:

[1754] The server compares the received facial data with the facial recognition database and displays the message "Matching...".

[1755] Step 13:

[1756] The server retrieves the person's information linked to the matched facial data from the business card database. A notification is sent stating, "Person's information has been retrieved."

[1757] Step 14:

[1758] The device sends the captured facial data to the emotion engine. A message appears stating, "Analyzing emotion data."

[1759] Step 15:

[1760] The emotion engine analyzes the received facial data to identify the user's emotional state. A notification is then displayed stating, "Emotional data has been analyzed."

[1761] Step 16:

[1762] The emotion engine sends the analyzed emotion data to the server. The status "Sending emotion data" is displayed.

[1763] Step 17:

[1764] The server sends the acquired person information and emotion data to the terminal.

[1765] Step 18:

[1766] The device (smart glasses) displays the transmitted person information and emotion data on the lenses. In addition to "Taro Tanaka - Sales Department," information such as "Current emotion: Joy" is displayed.

[1767] Step 19:

[1768] The device (smart glasses) uses its voice output function to announce, "This is Taro Tanaka from the Sales Department. Your current emotion is joy."

[1769] Thus, in addition to integrating facial recognition and a business card database, this system utilizes an emotion engine to provide even deeper information. This allows users to understand not only the information but also the emotional state of others within a specific facility, thereby improving work efficiency and the quality of communication.

[1770] (Example 2)

[1771] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1772] Conventional facial recognition systems can obtain basic information about a person, but they struggle to understand that person's emotional state. Therefore, especially in environments where person-to-person communication is crucial, there is a need for a means to quickly grasp a person's emotional state. Furthermore, there is a need for a method to effectively perform facial recognition and emotion analysis within specific facilities while minimizing the risk of information leakage.

[1773] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1774] In this invention, the server includes means for transmitting facial data to the server, means for comparing it with a facial recognition database, and means for analyzing the facial data to identify emotional states. This makes it possible to understand not only facial recognition but also the emotional states of individuals in real time within a specific facility.

[1775] "Facial data" refers to image data of a person's face captured by a camera.

[1776] A "server" refers to a computer system that stores, analyzes, and manages facial and emotional data.

[1777] A "facial recognition database" is a database stored on a server that uses facial data to authenticate individuals.

[1778] "Personal information" refers to basic information and attribute data about a person, obtained from sources such as business card databases.

[1779] A "terminal" is a hardware device used by a user to perform input and output, including smart glasses and sensor-equipped devices.

[1780] "Smart glasses" are wearable devices that can capture facial data and display information and output audio.

[1781] An "emotion engine" is a combination of software and hardware that analyzes captured facial data to identify emotional states.

[1782] "Emotional data" refers to data that indicates a person's emotional state, as identified by the emotion engine.

[1783] "Specific facilities" refers to physical locations where the use of the system is restricted to minimize the risk of information leakage.

[1784] This invention relates to a system that links facial recognition data and business card data to display or output information and emotional state of a person using smart glasses and an emotion engine within a specific facility. In this system, "facial data" is captured, sent to a server for analysis, and then person information and emotional state are identified.

[1785] Hardware and software to be used

[1786] The following hardware will be used as the terminal:

[1787] Cameras with sensors (installed at facility entrances and on smart glasses)

[1788] Smart glasses (display information on the lenses, with audio output capability)

[1789] Access control system (manages entry and exit to facilities)

[1790] Servers and related software:

[1791] Facial recognition database (a database for storing and matching facial data)

[1792] Business card database (a database for storing personal information)

[1793] Emotion engine (software that analyzes facial data to identify emotional states)

[1794] Information display and voice notification engine (software for displaying information on smart glasses and providing voice notifications)

[1795] Explanation of natural language processing

[1796] 1. Capture and register facial data

[1797] The user stands in front of a sensor-equipped camera at the facility entrance and their face data is captured. The terminal encodes this face data and sends it to the server. The server decodes the face data, compares it with existing data, and stores it in a face recognition database. Next, the server prompts the user to upload business card data, and once the user uploads the business card data, it is stored on the server.

[1798] 2. Authentication within the facility

[1799] When a user enters a facility, the device performs facial recognition and sends the data to the server. The server compares the data with the facial recognition database, and if authentication is successful, the device authorizes the use of the smart glasses. Once the setup is complete, the smart glasses become available for use.

[1800] 3. Facial recognition and data linkage

[1801] As a user moves around the facility wearing smart glasses, the smart glasses capture the faces of people around them in real time. The smart glasses encode the captured face data and send it to a server. The server compares it with a facial recognition database and retrieves matching person information from a business card database, then sends it to the smart glasses.

[1802] 4. Analysis of the Emotional Engine

[1803] The emotion engine analyzes facial data captured by the smart glasses and identifies the emotional state. The identified emotion data is encoded and sent to a server. The server integrates this emotion data with the person's information and sends it back to the smart glasses.

[1804] 5. Display of Information and Emotion

[1805] The smart glasses receive person information and emotion data transmitted from the server and display it on the lenses. They also have a voice notification function.

[1806] Specific example

[1807] 1. User registration and facial data acquisition:

[1808] The user stands still at the entrance of the facility, facing the camera.

[1809] The device will notify you that "face data has been captured" and send the face data to the server.

[1810] The server displays a confirmation message stating, "Face data saving complete," and prompts the user to upload business card data.

[1811] The user uploads business card data, and the device notifies them that "business card data has been sent to the server."

[1812] 2. Authentication within the facility:

[1813] When a user enters the facility, the terminal performs facial recognition and displays "Authentication successful."

[1814] After authentication, you will be permitted to use the smart glasses, and the necessary settings will be configured.

[1815] 3. Facial recognition and data linkage:

[1816] When the user enters the meeting room, the smart glasses' camera captures the faces of people around them.

[1817] The captured facial data is sent to the server.

[1818] The server displays "Matching against facial recognition database" and retrieves information on the matching person.

[1819] 4. Analysis of the Emotion Engine:

[1820] The emotion engine analyzes the facial data and sends a message to the server saying, "Emotion data has been analyzed."

[1821] The server processes the emotional data as additional information and integrates it with the person's information.

[1822] 5. Displaying information and emotions:

[1823] The server sends the acquired person information and emotional data to the terminal.

[1824] The smart glasses display this information on the lenses. In addition to "Employee A - Sales Department," information such as "Current emotion: Joy" is displayed. A voice notification also says, "Employee A, Sales Department. Your current emotion is joy."

[1825] Example of a prompt

[1826] "Please capture facial data at the facility entrance and upload your business card data."

[1827] "Perform real-time facial recognition and emotion analysis while moving around the facility, and display the results on smart glasses."

[1828] "Link facial recognition data with business card data, use an emotion engine to analyze emotional states, and provide notifications via display and audio."

[1829] This system allows users to quickly obtain information about others within a specific facility, as well as understand their emotional state. This enables more appropriate communication and is expected to improve work efficiency. Furthermore, by limiting the use of the system to specific facilities, the risk of information leakage can be minimized.

[1830] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1831] Step 1:

[1832] The user stands in front of a sensor-equipped camera at the facility entrance. The device captures facial data, encodes it, and then sends it to the server.

[1833] [Input] User's face image

[1834] [Data Processing] The device encodes the facial image.

[1835] [Output] Encoded face data is sent to the server.

[1836] Specifically, the camera recognizes the contours of the face, and face capture is performed when the camera remains still for a certain period of time.

[1837] Step 2:

[1838] The server decodes the received facial data and compares it with the facial recognition database. After confirming a match with existing data, it registers it as new data.

[1839] [Input] Encoded facial data sent from the device

[1840] [Data Processing] The server decodes the facial data and performs matching calculations.

[1841] [Output] The new face data is saved to the facial recognition database, and a registration completion message is sent to the device.

[1842] Specifically, the server first decomposes the facial data into multiple feature points and compares them with existing data in the database using a high-speed matching algorithm.

[1843] Step 3:

[1844] The terminal displays a registration completion message from the server and prompts the user to upload business card data. The user uploads the business card data to the terminal, and the terminal sends it to the server.

[1845] [Input] Registration completion message from the server, user's business card data

[1846] [Data Processing] The terminal encodes the business card data.

[1847] [Output] Business card data is sent to the server.

[1848] Specifically, the device displays "Please upload your business card data," and the user uploads a business card image to the device.

[1849] Step 4:

[1850] The server receives business card data, decodes it, and saves it to the business card database. It also notifies the terminal that the business card data has been successfully received.

[1851] [Input] Encoded business card data sent from the terminal

[1852] [Data Processing] The server decodes and saves the business card data.

[1853] [Output] The business card data is saved to the business card database, and a message confirming receipt is sent to the terminal.

[1854] Specifically, the server analyzes business card data using OCR (optical character recognition) technology, extracts important information, and stores it in a database.

[1855] Step 5:

[1856] When a user enters the facility, the terminal performs facial recognition again and sends the data to the server. The server compares the data with the facial recognition database and sends the authentication result back to the terminal.

[1857] [Input] User's face image

[1858] [Data Processing] The terminal encodes the facial image and sends it to the server. The server decodes the facial data and performs matching calculations.

[1859] [Output] Authentication result is sent to the terminal.

[1860] Specifically, the device will display "Authentication in progress," and upon successful authentication, it will notify the user with "Authentication successful."

[1861] Step 6:

[1862] After authentication, the device will authorize the use of the smart glasses and automatically perform pairing and necessary settings.

[1863] [Input] Authentication result from the server

[1864] [Data Processing] The device resets the smart glasses settings and performs pairing.

[1865] [Output] Smart glasses become ready for use.

[1866] Specifically, the device connects to the smart glasses using Bluetooth and performs the initial setup.

[1867] Step 7:

[1868] As the user moves around the facility, the smart glasses continuously capture the faces of people around them in real time and send the facial data to a server.

[1869] [Input] Captured facial data

[1870] [Data Processing] The smart glasses encode the facial data and send it to the server.

[1871] [Output] Encoded face data is sent to the server.

[1872] Specifically, the smart glasses' camera tracks the face in real time and automatically sends the data to a server.

[1873] Step 8:

[1874] The server matches the received facial data, retrieves matching person information from the business card database, and sends it to the smart glasses.

[1875] [Input] Face data transmitted from smart glasses

[1876] [Data Processing] The server matches facial data and retrieves person information.

[1877] [Output] Person information is sent to smart glasses.

[1878] Specifically, the server cross-references the facial recognition database and the business card database to quickly retrieve data.

[1879] Step 9:

[1880] The emotion engine analyzes facial data captured by the smart glasses to identify the emotional state. The identified emotion data is then sent to the server.

[1881] [Input] Captured facial data

[1882] [Data Processing] The emotion engine analyzes facial data and identifies emotional states.

[1883] [Output] Emotional state data is sent to the server.

[1884] Specifically, the emotion engine analyzes facial movements and changes in expression to identify emotional states in real time.

[1885] Step 10:

[1886] The server integrates emotional data with person information and sends it to the smart glasses. The smart glasses display the information on the lenses and provide audio notifications as needed.

[1887] [Input] Emotional state data, person information

[1888] [Data Processing] The server integrates the data and formats it for smart glasses.

[1889] [Output] Display and audio notifications are provided via smart glasses.

[1890] In terms of specific actions, the smart glasses display "Employee A - Sales Department" and "Current Emotion: Joyful" on the lens, and announce via voice, "Employee A, Sales Department. Your current emotion is joyful."

[1891] (Application Example 2)

[1892] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1893] Many existing facial recognition systems only perform person recognition and are insufficient for providing advanced customer service. Especially in physical stores, it is crucial to quickly understand customers' emotional states and provide individualized support, but current systems primarily rely on abstract data and manual information acquisition, lacking practicality. Furthermore, without linking customer information with emotional data, appropriate service recommendations and highly accurate customer service are difficult. This project aims to solve these problems and improve the customer experience and provide efficient service delivery.

[1894] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1895] In this invention, the server includes means for transmitting facial data to the server, means for comparing it with a facial recognition database stored on the server, means for acquiring person information, means for transmitting the acquired person information and emotion data to a terminal, means for displaying or outputting audio to the terminal, means for analyzing facial data with an emotion engine and identifying the emotional state, and means for performing the above operations only within a specific facility. This makes it possible to automatically link customer facial recognition data and business card information to understand the customer's emotional state. As a result, store employees can acquire customer information in real time and provide individually personalized customer service.

[1896] "Facial data" refers to image data of a person's face captured by a camera or other imaging device.

[1897] A "server" is a computer system that manages, stores, and processes data via a network.

[1898] A "facial recognition database" is a database that stores multiple pre-registered facial data and compares them with newly acquired facial data.

[1899] "Personal information" refers to information about individual individuals (e.g., name and affiliation) obtained from sources such as business card databases.

[1900] A "terminal" is a device that a user directly operates and that displays or notifies information via sound. In this invention, smart glasses are the primary example of this type of device.

[1901] An "emotion engine" is software or hardware that analyzes facial data to identify a person's emotional state (e.g., joy, anger, sadness, etc.).

[1902] "Within a specific facility" refers to a location to which the present invention is applied in a limited manner, and means the area to which facial recognition and emotion analysis are to be performed.

[1903] The system for carrying out this invention includes functions for capturing facial data, matching it with a database, analyzing emotions using an emotion engine, displaying the acquired information in real time, and providing voice notifications. The specific configuration and operation details of this system are described below.

[1904] System Configuration

[1905] The user puts on smart glasses.

[1906] The device captures facial data of people in its surroundings using a camera mounted on smart glasses and sends this data to a server.

[1907] The server compares the facial data with a facial recognition database to obtain the person's information.

[1908] The server uses an emotion engine to analyze the captured facial data and identify emotional data.

[1909] The server sends the acquired person information and emotion data to the terminal.

[1910] The device displays this information on the smart glasses' screen and provides voice notifications as needed.

[1911] Examples of specific hardware and software to be used

[1912] Hardware:

[1913] Smart glasses: Capture facial data and display information.

[1914] Server: Performs data processing and manages databases.

[1915] software:

[1916] OpenCV: Image processing library for face recognition

[1917] Requests module: A library for communicating with the server.

[1918] Emotion engine: Software that analyzes emotional states from facial data; a specific example is Microsoft Azure's Emotion API.

[1919] System operation example

[1920] Facial data capture and registration

[1921] The user captures their face with a camera through smart glasses at the facility entrance, and the device notifies them that "face data has been captured." This face data is sent to a server and stored in a face recognition database. Next, the user uploads business card data, and the device notifies them that "business card data has been sent to the server." This business card data is stored in a business card database.

[1922] Authentication and information display within the facility

[1923] As a user moves around a facility wearing smart glasses, the cameras on the smart glasses continuously capture the faces of people in their surroundings. The captured facial data is sent to a server in real time and compared with a facial recognition database. Matching person information is retrieved from a business card database and sent to the device. Furthermore, an emotion engine analyzes the emotional state, and the results are also sent from the server to the device.

[1924] For example, when a user enters a meeting room, the smart glasses display shows "Taro Yamada - Gold Member, Current Emotion: Joyful," along with an audio notification. This information allows the user to respond quickly and appropriately.

[1925] Example of a prompt

[1926] If you want to ask the AI ​​model that generates a specific and immediately usable action plan for this application, use the following prompt:

[1927] "You are developing a system that uses smart glasses to support customer service in physical stores by utilizing facial recognition data and emotion engine analysis results. Please provide recommendations for actions that will allow for more effective customer service based on real-time customer information."

[1928] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1929] Step 1:

[1930] The user puts on smart glasses and faces the camera at the entrance of the facility.

[1931] Input: Capture a face at the facility entrance.

[1932] Data processing: The smart glasses' camera acquires image data of the face.

[1933] Output: Captured facial data is sent to the terminal.

[1934] Step 2:

[1935] The device sends the captured facial data to the server.

[1936] Input: Facial image data acquired by the smart glasses' camera.

[1937] Data processing: Convert the encoded facial image data into a transmission format.

[1938] Output: Face data is sent to the server.

[1939] Step 3:

[1940] The server compares the facial data with a facial recognition database.

[1941] Input: Image data of a face sent from the device.

[1942] Data processing: Matching is performed using a facial recognition algorithm (e.g., a deep learning model).

[1943] Output: Matching person information is retrieved.

[1944] Step 4:

[1945] The server retrieves matching person information from the business card database.

[1946] Input: Person information matching the facial recognition database.

[1947] Data processing: Extract necessary personal information (name, affiliation, etc.) from the business card database.

[1948] Output: Person information is saved on the server.

[1949] Step 5:

[1950] The server uses an emotion engine to analyze facial data and identify emotional data.

[1951] Input: Image data of a face sent from the device.

[1952] Data processing: Using emotion analysis algorithms, we analyze emotional states such as joy, anger, and sadness.

[1953] Output: Emotional data is saved to the server.

[1954] Step 6:

[1955] The server sends the acquired person information and emotional data to the terminal.

[1956] Input: Person information obtained from a business card database and emotional data analyzed by an emotion engine.

[1957] Data processing: Integrate this data and convert it into a transmission format.

[1958] Output: Person information and emotion data are sent to the terminal.

[1959] Step 7:

[1960] The device displays the person information and emotion data it receives on the smart glasses' display.

[1961] Input: Person information and emotion data sent from the server.

[1962] Data processing: Converts received data into a display format.

[1963] Output: The smart glasses display shows "○○ ○○ - Affiliation: △△, Current emotion: Joy." Voice notifications are also provided as needed.

[1964] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1965] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1966] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1967] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1968] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1969] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1970] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1971] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1972] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1973] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1974] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1975] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1976] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1977] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1978] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1979] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1980] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1981] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1982] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1983] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1984] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1985] The following is further disclosed regarding the embodiments described above.

[1986] (Claim 1)

[1987] A means of capturing facial data,

[1988] A means of sending facial data to a server,

[1989] A means of matching against a facial recognition database stored on a server,

[1990] Means of obtaining personal information,

[1991] A means of transmitting acquired person information to a terminal,

[1992] A means of displaying or outputting audio on the terminal,

[1993] A system that includes means for performing the above operations only within a specific facility.

[1994] (Claim 2)

[1995] The system according to claim 1, wherein the means for capturing facial data is a camera provided on smart glasses.

[1996] (Claim 3)

[1997] The system according to claim 1, wherein the means for obtaining person information is to obtain information from a business card database.

[1998] "Example 1"

[1999] (Claim 1)

[2000] A means of capturing facial data,

[2001] A means of sending facial data to a server,

[2002] A means of matching against a facial recognition database stored on a server,

[2003] A means of obtaining person information from a linked database,

[2004] A means of transmitting acquired person information to a terminal,

[2005] A means of displaying or outputting audio on the terminal,

[2006] A system that includes means for performing the above actions using smart glasses within a specific facility.

[2007] (Claim 2)

[2008] The system according to claim 1, wherein the means for capturing facial data is a camera provided on smart glasses.

[2009] (Claim 3)

[2010] The system according to claim 1, wherein the means for obtaining person information is to obtain information from a business card database.

[2011] "Application Example 1"

[2012] (Claim 1)

[2013] A means of capturing facial data,

[2014] A means of sending facial data to a server,

[2015] A means of matching against a facial recognition database stored on a server,

[2016] Means of obtaining personal information,

[2017] A means of transmitting acquired person information to a terminal,

[2018] A means of displaying or outputting audio on the terminal,

[2019] The acquired personal information is obtained through means including purchase history or special requests,

[2020] A system that includes means for performing the above operations only within a specific commercial facility.

[2021] (Claim 2)

[2022] The system according to claim 1, wherein the means for capturing facial data is a camera provided on a smart device.

[2023] (Claim 3)

[2024] The system according to claim 1, wherein the means for obtaining person information is to obtain information from a purchase history database.

[2025] "Example 2 of combining an emotion engine"

[2026] (Claim 1)

[2027] A means of capturing facial data,

[2028] A means of sending facial data to a server,

[2029] A means of matching against a facial recognition database stored on a server,

[2030] Means of obtaining personal information,

[2031] A means of transmitting acquired person information to a terminal,

[2032] A means of displaying or outputting audio on the terminal,

[2033] A means of identifying emotional states by analyzing facial data,

[2034] A means of sending emotional data to a server and integrating it with person information,

[2035] A system that includes means for performing the above operations only within a specific facility.

[2036] (Claim 2)

[2037] The system according to claim 1, wherein the means for capturing facial data is a camera provided on smart glasses.

[2038] (Claim 3)

[2039] The system according to claim 1, wherein the means for obtaining person information is to obtain information from a business card database.

[2040] "Application example 2 when combining with an emotional engine"

[2041] (Claim 1)

[2042] A means of capturing facial data,

[2043] A means of sending facial data to a server,

[2044] A means of matching against a facial recognition database stored on a server,

[2045] Means of obtaining personal information,

[2046] A means of transmitting acquired person information and emotional data to a terminal,

[2047] A means of displaying or outputting audio on the terminal,

[2048] A means of analyzing facial data with an emotion engine to identify emotional states,

[2049] A system that includes means for performing the above operations only within a specific facility.

[2050] (Claim 2)

[2051] The system according to claim 1, wherein the means for capturing facial data is a camera provided on smart glasses.

[2052] (Claim 3)

[2053] The system according to claim 1, wherein the means for obtaining person information is to obtain information from a business card database. [Explanation of Symbols]

[2054] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of capturing facial data, A means of sending facial data to a server, A means of matching against a facial recognition database stored on a server, Means of obtaining personal information, A means of transmitting acquired person information to a terminal, A means of displaying or outputting audio on the terminal, A system that includes means for performing the above operations only within a specific facility.

2. The system according to claim 1, wherein the means for capturing facial data is a camera provided on smart glasses.

3. The system according to claim 1, wherein the means for obtaining person information is to obtain information from a business card database.

Images